search

PalisadoesFoundation / talawa-api

API Backend for the Talawa Mobile App. Click on the link below to see our documentation
https://docs.talawa.io/
GNU General Public License v3.0
209 stars 709 forks source link

Cloud Based API Instance for Developers #1428

Open palisadoes opened 10 months ago

palisadoes commented 10 months ago

We need to have a cloud based API instance for developers to test against. It must:

  1. Be integrated into our CI/CD pipeline so that the app is rebuilt with every newly merged PR.
  2. Initialize the database to a known working state with appropriate services will be restarted when this happens: (there are existing API calls for this):
    1. with each CI/CD update
    2. every 24 hours
  3. Use a talawa.io sub-domain as the endpoint
  4. The API instance will have a rate limiting mechanism to reduce the risk of abuse
  5. Preferably use a free service
  6. Use a palisadoes role based account for management
  7. The demo cloud API must be easily switched between branches as the code source. We will be migrating to master as the default branch soon.

Please coordinate these efforts with @noman2002 and @kb-0311. Ask to be assigned this task by them.

vasujain275 commented 7 months ago

@Manik2708, great idea, but we need a better rate limiter for our API right now. I have thought about implementing it with Redis. Currently, I am working on two different issues. Maybe you can first implement the rate limiting using Redis. After that, we can test Redis caching, as the rate limiter is of higher priority and will be beneficial to the project a lot.

Is the issue opened for this? Like can I get more context about rate limiting?

  1. You need to create a new issue. I will be happy to assist you in creating one if you wish.
  2. Currently, we use express-rate-limiter to rate limit our API, but it is not an ideal solution for the rate-limiting problem, especially considering that we are soon having a cloud instance for our API.
  3. A better approach would be to implement rate limiting by storing IPs as key-value pairs in Redis and limiting access based on that method.
  4. Here is the resource that you can refer to: https://youtu.be/_DqfiW08HkA?si=mZ6wblV3SNPiN0Ru
  5. After creating the issue, please comment "Can I work on this issue?" so that maintainers can assign it.

Feel free to mention me in the comments on that issue if you need my help.

Manik2708 commented 7 months ago

@Manik2708, great idea, but we need a better rate limiter for our API right now. I have thought about implementing it with Redis. Currently, I am working on two different issues. Maybe you can first implement the rate limiting using Redis. After that, we can test Redis caching, as the rate limiter is of higher priority and will be beneficial to the project a lot.

Is the issue opened for this? Like can I get more context about rate limiting?

  1. You need to create a new issue. I will be happy to assist you in creating one if you wish.
  2. Currently, we use express-rate-limiter to rate limit our API, but it is not an ideal solution for the rate-limiting problem, especially considering that we are soon having a cloud instance for our API.
  3. A better approach would be to implement rate limiting by storing IPs as key-value pairs in Redis and limiting access based on that method.
  4. Here is the resource that you can refer to: https://youtu.be/_DqfiW08HkA?si=mZ6wblV3SNPiN0Ru
  5. After creating the issue, please comment "Can I work on this issue?" so that maintainers can assign it.

Feel free to mention me in the comments on that issue if you need my help.

Thanks for this, surely going to open the issue!

vasujain275 commented 7 months ago

@Manik2708 So I think we should first let tsyringe fix that issue and then proceed with this here

palisadoes commented 7 months ago

It's failing. Please resubmit

vasujain275 commented 7 months ago

@palisadoes

  1. It is failing because two commits were added in a very short time gap.
  2. That failed the workflow for latest commit as the previous commit workflow was still running.
  3. The api instance is working here correctly - https://api-demo.talawa.io/
  4. The failed workflow will not affect the deployment of the previous commit to our cloud instance.
  5. This edge case only happens when commits are pushed at a small time gap.
palisadoes commented 7 months ago

@vasujain275 @Manik2708

  1. Can we let people know this exists and close?
  2. Or should we wait for this to be completed?
    1. https://github.com/PalisadoesFoundation/talawa-api/issues/1782

Waiting seems like the preferred solution

vasujain275 commented 7 months ago

@palisadoes

I also think we should wait for the rate limiter issue to complete then we can find an approach to test redis caching.

vasujain275 commented 7 months ago

@palisadoes I have been analyzing the Deploy Workflow for the past couple of days and here are the issues I found -

  1. The workflow will sort of break if there are 2 commits in a very short interval of time. However, it does not affect the functioning of the cloud instance, i.e., no downtime because of this issue.
  2. It takes 10-20 min on every new commit to create new docker containers and delete the previous ones. This is Downtime for API Instance.

Possible Solution for 2nd Issue -

  1. There will be two workflows -
    • Firstly, a workflow will create Docker Image of latest state of Talawa API and push it to Docker Hub.
    • Second Workflow will shh into the VPS and simply pull the Docker Image of Talawa API along with other required images like Redis and mongo.
  2. This will significantly decrease the time to create new containers, as we only need to pull the prebuilt docker images from the docker hub.

Possible Solution for 1st Issue -

  1. As mentioned in previous solution, if push Docker Image of Talawa API to Docker Hub on every commit. We can only deploy the latest Docker Image every 1hr or so.
  2. This way, for example, If there are 5 commits in that 1hr, only the latest commit's image will be Deployed to the Cloud Instance.

@kb-0311 and @noman2002, your insights on this approach would be appreciated

kb-0311 commented 7 months ago

@vasujain275 Why does the workflow break when the two commits are made in a very short interval of time? Is it because of the remote execution of shell commands in short intervals that is causing the issue or something else entirely?

Agreed on creating two workflows to deploy and pull from the docker hub.

I am still not convinced about the solution to submit only the latest commit at regular time intervals. If there are urgent bug fixes required in production then the amount of time for the bug fix PRs to reflect in prod env would be considerable.

I think I want to understand the root of the problem of 1st issue so that we can analyze better solutions.

vasujain275 commented 7 months ago

@kb-0311

  1. Yes, the workflow is breaking because of the remote execution of shell commands in short intervals.
  2. The root cause of the 1st Issue is, in my opinion, the time it takes to execute remote commands is too long. That's Why I think breaking the workflow into two can be a valid solution to both problems.
kb-0311 commented 7 months ago

@vasujain275 sorry for the late reply. You can proceed with this approach. I would even say after the workflow to push the api to docker hub is executed, delay the execution of the workflow for pulling the API in VPS by some reasonable amount of time. This should solve our problem if SSH execution is the bottleneck.

xoldd commented 6 months ago

There's fly.io and render for web server hosting. Anyway, nowadays most companies have stopped providing good free tiers for stateful continously running servers, most free options are available only in serverless. But talawa-api isn't build around the idea of serverless.

Mongodb atlas is not the only cloud database with free tier. Infact sql database providers have more generous free tiers compared to mongodb. For postgresql there's supabase and neondb, for mysql there's planetscale. I have said this before, mongodb is only good for dumping unorganized, non relational data, which isn't true for 99.99% applications in the world, all data is relational/graphs. Mongodb has a very niche use case, and talawa-api doesn't have that use case.

kb-0311 commented 6 months ago

@xoldyckk The api has already been deployed on a VPS : https://api-demo.talawa.io/graphql

vasujain275 commented 6 months ago

@vasujain275 sorry for the late reply. You can proceed with this approach. I would even say after the workflow to push the api to docker hub is executed, delay the execution of the workflow for pulling the API in VPS by some reasonable amount of time. This should solve our problem if SSH execution is the bottleneck.

Ok, will complete this new workflow and add a PR for it

vasujain275 commented 6 months ago

@palisadoes @kb-0311 @xoldyckk Here are the updates on this -

  1. Waiting for @varshith257 to complete the multi-stage build Dockerfile here at - #1751.
  2. Once the new multi-stage Dockerfile is merged, I will use my personal Docker Hub account to create and test the Talawa API GitHub Action for uploading images to Docker Hub on every push.
  3. Once the pipeline is done and tested, we can add the official Palisadoes Foundation Docker Hub account credentials as GitHub Secrets and test the pipeline and cloud instance at - https://api-demo.talawa.io/
kb-0311 commented 6 months ago

@vasujain275 That sounds great.

@varshith257 Do keep me updated on the PR when you submit to implement multi stage Dockerfile.

varshith257 commented 6 months ago

@kb-0311 I have done implementing multi-stage Dockerfile build and awaiting to make a PR but issue is not assigned yet to me. Waiting for approval from @palisadoes or @Cioppolo14 .

Cioppolo14 commented 6 months ago

@varshith257 We are working on a solution right now for Docker Hub. We will follow up. Thanks for your patience.

vasujain275 commented 6 months ago

Update on this - The work on this issue is paused due to this issue being dependent on - #1885 being merged. Once this pr is merged, I will start work on it according to this approach - https://github.com/PalisadoesFoundation/talawa-api/issues/1428#issuecomment-1957799874

github-actions[bot] commented 5 months ago

This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.

vasujain275 commented 5 months ago

This issue is still stalled due to #1885 not being merged. The PR is waiting for review

varshith257 commented 5 months ago

@vasujain275 If anything making you halt to make progress because of #1885. Please take work from my PR and test it and go ahead to progress.

@palisadoes is really worried about to see this ASAP to make available to all of us

vasujain275 commented 5 months ago

@varshith257 Okay, I will take docker files from your pr and paste it in my fork to test the ci cd pipeline

varshith257 commented 5 months ago

👍

On Sun, 24 Mar, 2024, 9:36 am Vasu Jain, @.***> wrote:

@varshith257 https://github.com/varshith257 Okay, I will take docker files from your pr and paste it in my fork to test the ci cd pipeline

— Reply to this email directly, view it on GitHub https://github.com/PalisadoesFoundation/talawa-api/issues/1428#issuecomment-2016685819, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4BF3HCL7T6Y6QMMPR6ZPY3YZZGK7AVCNFSM6AAAAAA7O5GJSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJWGY4DKOBRHE . You are receiving this because you were mentioned.Message ID: @.***>

vasujain275 commented 5 months ago

@varshith257 Okay, I will take docker files from your pr and paste it in my fork to test the ci cd pipeline

Sorry for no progress last week on this issue, I was not well. Resuming the work again

varshith257 commented 5 months ago

@vasujain275 Are you fine now?

vasujain275 commented 5 months ago

@vasujain275 Are you fine now?

Better now, thanks for asking

varshith257 commented 5 months ago

Any progress on this?

vasujain275 commented 5 months ago

Any progress on this?

Yeah, I am researching a solution that will be effective than current one. Currently, we are using npm run dev on the demo api server containers. Does the build process issue you were working on is fixed? If it is resolved, we can then serve pre built html css js files directly.

varshith257 commented 5 months ago

Yes the build issue is fixed

vasujain275 commented 5 months ago

Yes the build issue is fixed

Great, is it merged?

github-actions[bot] commented 4 months ago

This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.

vasujain275 commented 4 months ago

PR for #1751 by @varshith257 merged yesterday only, will work on this from today

palisadoes commented 4 months ago

I haven’t been paying much attention to this, however I noticed that Talawa-API Docker instances do not import sample data. The CLI commands in the INSTALLATION.md file execute without error but don’t populate the Docker Mongo instance database. The setup script doesn’t prompt to load data either.

This is covered in this issue:

My understanding was that the GoDaddy API deployment was via Docker, and that there would be a job running to periodically overwrite the database with sample data to reduce the risk of abuse.

The discovery of this issue means that the instance is not functional in that the data will always be blank.

I’ve also noticed that the cloud deployments have been failing too. Why is that?

We should have this working and eventually well documented in INSTALLATION.md in summary form and docs.talawa.io in more detail.

Please give a status update on this. It’s not clear to me.

vasujain275 commented 4 months ago

@palisadoes Sorry for late reply, I am currently busy at school due to end sem exams till 30 May. I can continue the work after that If anybody do not solve this issue till them. Sorry for the inconvenience

palisadoes commented 4 months ago

@vasujain275

  1. Thanks. This can wait.
  2. We have an issue created to fix the Docker data importation which should help when you return.
vasujain275 commented 4 months ago

Thanks for understanding

github-actions[bot] commented 3 months ago

This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.

vasujain275 commented 3 months ago

@varshith257 Did you fix the build issues as I am not able to get the server up and running in production mode, here is the list of commands I run - 

npm install
npm setup
npm run prebuild
npm run build
npm run start

Please take a look into it

PS: I am successfully able to run the server in dev mode using - npm run dev

palisadoes commented 3 months ago

@noman2002 @Cioppolo14 This is a cloud instance issue to follow.

vasujain275 commented 3 months ago

I have reviewed the Issue that was raised to fix the build errors - https://github.com/PalisadoesFoundation/talawa-api/issues/1916 but saw that was not able to successfully fix all errors in the build process due to stability issues.

@palisadoes shall we continue with dev images only as we are doing currently? They will 3x the size of production docker images (600MB -> 1.8GB) but this is the only viable option imo right now

varshith257 commented 3 months ago

Why 1.8GB image size is required? The multi-stage build has been done to reduce image size, it is now ~500MB of dev and prod (~200MB).

varshith257 commented 3 months ago

@varshith257 Did you fix the build issues as I am not able to get the server up and running in production mode, here is the list of commands I run -

It won't because we are still pointing it to a stale file. There needs a refactoring of the codebase to support the production environment. I have closed the issue by fixing some of them, But it needs to be fixed with tsconfig and fixing of the errors emitted. The compiler from ts-> js is accompanied by incompatibility and it breaks down due to some libraries being built of ECMA.

palisadoes commented 3 months ago
  1. We just need an instance up and running on the GoDaddy server.
  2. It must work whenever we update the main / or develop branch. (I'm not sure which yet in my mind)
  3. We are paying for a docker hub account that we are not using. I'd prefer to shut it down until we are ready. Can I do that?
varshith257 commented 3 months ago

@vasujain275 You can go with testing it with dev image. As we are developing this for developers dev image could be right one to go with IMO

vasujain275 commented 3 months ago

@palisadoes

  1. Okay, I will go with dev images only as the build process is not ready yet.
  2. Yes CI/CD pipeline will ensure to update the instance on every new commit.
  3. Why are we paying for a Docker Hub account?? I specifically mentioned that we don't need a docker hub paid account, please downgrade that account to a free account.
palisadoes commented 3 months ago
  1. We had to use a paid account as we needed to register as an organization.
  2. Individuals can get free accounts, but we don't want to be tied to someone who could walk away from the project.

Do we need a docker account at this stage?

vasujain275 commented 3 months ago

@palisadoes If we need a paid account for docker hub then I can implement the pipeline without docker hub. Docker hub would make this a little easier but we can implement the pipeline without it too.

palisadoes commented 3 months ago

OK. I'll shutdown the docker account over the weekend.

vasujain275 commented 3 months ago

@palisadoes The ci/cd is ready but it is failing often when there are several commits in small time intervals. How about if we update the cloud instance every 6 or 12 hours? It will improve the stability and uptime of the instance significantly.