Open palisadoes opened 10 months ago
@Manik2708, great idea, but we need a better rate limiter for our API right now. I have thought about implementing it with Redis. Currently, I am working on two different issues. Maybe you can first implement the rate limiting using Redis. After that, we can test Redis caching, as the rate limiter is of higher priority and will be beneficial to the project a lot.
Is the issue opened for this? Like can I get more context about rate limiting?
express-rate-limiter
to rate limit our API, but it is not an ideal solution for the rate-limiting problem, especially considering that we are soon having a cloud instance for our API.Feel free to mention me in the comments on that issue if you need my help.
@Manik2708, great idea, but we need a better rate limiter for our API right now. I have thought about implementing it with Redis. Currently, I am working on two different issues. Maybe you can first implement the rate limiting using Redis. After that, we can test Redis caching, as the rate limiter is of higher priority and will be beneficial to the project a lot.
Is the issue opened for this? Like can I get more context about rate limiting?
- You need to create a new issue. I will be happy to assist you in creating one if you wish.
- Currently, we use
express-rate-limiter
to rate limit our API, but it is not an ideal solution for the rate-limiting problem, especially considering that we are soon having a cloud instance for our API.- A better approach would be to implement rate limiting by storing IPs as key-value pairs in Redis and limiting access based on that method.
- Here is the resource that you can refer to: https://youtu.be/_DqfiW08HkA?si=mZ6wblV3SNPiN0Ru
- After creating the issue, please comment "Can I work on this issue?" so that maintainers can assign it.
Feel free to mention me in the comments on that issue if you need my help.
Thanks for this, surely going to open the issue!
@Manik2708 So I think we should first let tsyringe
fix that issue and then proceed with this here
It's failing. Please resubmit
@palisadoes
@vasujain275 @Manik2708
Waiting seems like the preferred solution
@palisadoes
I also think we should wait for the rate limiter issue to complete then we can find an approach to test redis caching.
@palisadoes I have been analyzing the Deploy Workflow for the past couple of days and here are the issues I found -
Possible Solution for 2nd Issue -
Possible Solution for 1st Issue -
@kb-0311 and @noman2002, your insights on this approach would be appreciated
@vasujain275 Why does the workflow break when the two commits are made in a very short interval of time? Is it because of the remote execution of shell commands in short intervals that is causing the issue or something else entirely?
Agreed on creating two workflows to deploy and pull from the docker hub.
I am still not convinced about the solution to submit only the latest commit at regular time intervals. If there are urgent bug fixes required in production then the amount of time for the bug fix PRs to reflect in prod env would be considerable.
I think I want to understand the root of the problem of 1st issue so that we can analyze better solutions.
@kb-0311
@vasujain275 sorry for the late reply. You can proceed with this approach. I would even say after the workflow to push the api to docker hub is executed, delay the execution of the workflow for pulling the API in VPS by some reasonable amount of time. This should solve our problem if SSH execution is the bottleneck.
There's fly.io and render for web server hosting. Anyway, nowadays most companies have stopped providing good free tiers for stateful continously running servers, most free options are available only in serverless. But talawa-api isn't build around the idea of serverless.
Mongodb atlas is not the only cloud database with free tier. Infact sql database providers have more generous free tiers compared to mongodb. For postgresql there's supabase and neondb, for mysql there's planetscale. I have said this before, mongodb is only good for dumping unorganized, non relational data, which isn't true for 99.99% applications in the world, all data is relational/graphs. Mongodb has a very niche use case, and talawa-api doesn't have that use case.
@xoldyckk The api has already been deployed on a VPS : https://api-demo.talawa.io/graphql
@vasujain275 sorry for the late reply. You can proceed with this approach. I would even say after the workflow to push the api to docker hub is executed, delay the execution of the workflow for pulling the API in VPS by some reasonable amount of time. This should solve our problem if SSH execution is the bottleneck.
Ok, will complete this new workflow and add a PR for it
@palisadoes @kb-0311 @xoldyckk Here are the updates on this -
@vasujain275 That sounds great.
@varshith257 Do keep me updated on the PR when you submit to implement multi stage Dockerfile.
@kb-0311 I have done implementing multi-stage Dockerfile build and awaiting to make a PR but issue is not assigned yet to me. Waiting for approval from @palisadoes or @Cioppolo14 .
@varshith257 We are working on a solution right now for Docker Hub. We will follow up. Thanks for your patience.
Update on this - The work on this issue is paused due to this issue being dependent on - #1885 being merged. Once this pr is merged, I will start work on it according to this approach - https://github.com/PalisadoesFoundation/talawa-api/issues/1428#issuecomment-1957799874
This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.
This issue is still stalled due to #1885 not being merged. The PR is waiting for review
@vasujain275 If anything making you halt to make progress because of #1885. Please take work from my PR and test it and go ahead to progress.
@palisadoes is really worried about to see this ASAP to make available to all of us
@varshith257 Okay, I will take docker files from your pr and paste it in my fork to test the ci cd pipeline
👍
On Sun, 24 Mar, 2024, 9:36 am Vasu Jain, @.***> wrote:
@varshith257 https://github.com/varshith257 Okay, I will take docker files from your pr and paste it in my fork to test the ci cd pipeline
— Reply to this email directly, view it on GitHub https://github.com/PalisadoesFoundation/talawa-api/issues/1428#issuecomment-2016685819, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4BF3HCL7T6Y6QMMPR6ZPY3YZZGK7AVCNFSM6AAAAAA7O5GJSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJWGY4DKOBRHE . You are receiving this because you were mentioned.Message ID: @.***>
@varshith257 Okay, I will take docker files from your pr and paste it in my fork to test the ci cd pipeline
Sorry for no progress last week on this issue, I was not well. Resuming the work again
@vasujain275 Are you fine now?
@vasujain275 Are you fine now?
Better now, thanks for asking
Any progress on this?
Any progress on this?
Yeah, I am researching a solution that will be effective than current one. Currently, we are using npm run dev
on the demo api server containers. Does the build process issue you were working on is fixed? If it is resolved, we can then serve pre built html css js files directly.
Yes the build issue is fixed
Yes the build issue is fixed
Great, is it merged?
This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.
PR for #1751 by @varshith257 merged yesterday only, will work on this from today
I haven’t been paying much attention to this, however I noticed that Talawa-API Docker instances do not import sample data. The CLI commands in the INSTALLATION.md file execute without error but don’t populate the Docker Mongo instance database. The setup script doesn’t prompt to load data either.
This is covered in this issue:
My understanding was that the GoDaddy API deployment was via Docker, and that there would be a job running to periodically overwrite the database with sample data to reduce the risk of abuse.
The discovery of this issue means that the instance is not functional in that the data will always be blank.
I’ve also noticed that the cloud deployments have been failing too. Why is that?
We should have this working and eventually well documented in INSTALLATION.md in summary form and docs.talawa.io in more detail.
Please give a status update on this. It’s not clear to me.
@palisadoes Sorry for late reply, I am currently busy at school due to end sem exams till 30 May. I can continue the work after that If anybody do not solve this issue till them. Sorry for the inconvenience
@vasujain275
Thanks for understanding
This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.
@varshith257 Did you fix the build issues as I am not able to get the server up and running in production mode, here is the list of commands I run -
npm install
npm setup
npm run prebuild
npm run build
npm run start
Please take a look into it
PS: I am successfully able to run the server in dev mode using - npm run dev
@noman2002 @Cioppolo14 This is a cloud instance issue to follow.
I have reviewed the Issue that was raised to fix the build errors - https://github.com/PalisadoesFoundation/talawa-api/issues/1916 but saw that was not able to successfully fix all errors in the build process due to stability issues.
@palisadoes shall we continue with dev images only as we are doing currently? They will 3x the size of production docker images (600MB -> 1.8GB) but this is the only viable option imo right now
Why 1.8GB image size is required? The multi-stage build has been done to reduce image size, it is now ~500MB of dev and prod (~200MB).
@varshith257 Did you fix the build issues as I am not able to get the server up and running in production mode, here is the list of commands I run -
It won't because we are still pointing it to a stale file. There needs a refactoring of the codebase to support the production environment. I have closed the issue by fixing some of them, But it needs to be fixed with tsconfig and fixing of the errors emitted. The compiler from ts-> js is accompanied by incompatibility and it breaks down due to some libraries being built of ECMA.
@vasujain275 You can go with testing it with dev image. As we are developing this for developers dev image could be right one to go with IMO
@palisadoes
Do we need a docker account at this stage?
@palisadoes If we need a paid account for docker hub then I can implement the pipeline without docker hub. Docker hub would make this a little easier but we can implement the pipeline without it too.
OK. I'll shutdown the docker account over the weekend.
@palisadoes The ci/cd is ready but it is failing often when there are several commits in small time intervals. How about if we update the cloud instance every 6 or 12 hours? It will improve the stability and uptime of the instance significantly.
We need to have a cloud based API instance for developers to test against. It must:
Please coordinate these efforts with @noman2002 and @kb-0311. Ask to be assigned this task by them.