Open palisadoes opened 9 months ago
@noman2002 @kb-0311 I can give this issue a try
there is file named push-documentation.yml
in github/workflows
folder. Do we need to make changes here or create a new file?
@kirtanchandak Please proceed with this. You can use that file, you just need to do some minor changes. But I would suggest please check other solutions like render.com instead of Heroku.
Hi @noman2002 @kirtanchandak, I have been working on this same issue containerization talawa-api using docker here - https://github.com/PalisadoesFoundation/talawa-api/pull/1418 that was merged yesterday and was working on developing a ci/cd pipeline for this exact use case, @kirtanchandak can I work on this issue if you haven't made any significant progress yet?
One issue per person
One issue per person
Okay
This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.
@kirtanchandak Are you still working on this?
@palisadoes I can work on this if @kirtanchandak is not working on this
Assigning to@vasujain275. Please work with @noman2002 and @kb-0311 for comments, advice and reviewing the PR
Hi @noman2002 and @kb-0311, I did some research and found that the MongoDB Atlas free tier is good enough for our use case. I looked into redis free tier at render.com, it has 25 MB of memory with a 50 connection limit and no persistence storage. We can also host our API on render.com but it will be shut down after an inactivity of 5 minutes and will be restarted when called again. Will that render.com's inactivity delay or redis free tier will be a problem?
@vasujain275
@kb-0311 We can use the AWS free tier, which will have a t2.micro ec2 instance free for 12 months per account ( we will need a credit card to avail this 12-month free tier ). We can use the docker file and docker-compose file I created in a recent issue, we can set a ci/cd pipeline to push our image to the docker hub on each push and then pull to e2c using GitHub actions. We can then run docker-compose on e2c also using GitHub ci/cd that can run containers of both redis and MongoDB, although I think it will be best to run MongoDB on Atlas as it has a good free tier and will reduce the load on our t2.micro ec2 instance if we run it on atlas. We can also use azure/gcp, they also have this exact free tier of 12 months that requires a credit card on signup.
@vasujain275 Ideally we also want:
Other:
@palisadoes
Q. How can github actions do all the changes in our aws instance? Ans. Docker Containers, we will never install the talwa api directly to our aws instance instead we will use docker container of our api that can be rebuilt and restarted every time we push changes to our repo.
@noman2002 @kb-0311 , appreciate your feedback on possible improvements in this approach
@vasujain275
@palisadoes
@palisadoes
@kb-0311 I will start working on this by creating a new branch "deployment" in my forked repo and will follow the process you mentioned in the previous comment.
@palisadoes Okay I will keep that in mind
What's the status on this? It's an important part of our 2023 road map. We need this up and running by very early January.
@palisadoes
Sorry for the delay, I have programmed deploy.yml locally but having some issues with my credit card on aws so not able to test it. I will positively resolve it within a couple of days.
@vasujain275 If AWS is a difficulty, what would be the minimal Ubuntu VPS required with GoDaddy to get a test instance running? We could provide access via a public/private SSH key for security.
We really need to get this going.
@palisadoes I got the AWS ec2 instance working, I only had debit cards and it was not accepting them. I found a credit card and it worked, I have been testing the code for the past couple of hours. Creating a PR in couple of minutes
Please answer my question on the GoDaddy VPS.
With that solution we would have a fixed cost and not have to worry about unexpected cloud charges.
Let me know
@palisadoes
If we had a VPS we could host mongo on the local server for simplicity.
What would your minimum be?
Remember, we will be resetting the DB every 24 hours. So we don't need a lot of disk.
@palisadoes I believe that hosting MongoDB, Redis, and Talawa API on the same VPS is feasible. In this scenario, I think the basic plan (1 vCPU / 2GB RAM / 40GB Storage) would be sufficient.
@vasujain275
@kb-0311
@palisadoes
This issue did not get any activity in the past 10 days and will be closed in 180 days if no update occurs. Please check if the develop branch has fixed it and report again or close the issue.
@kb-0311 @palisadoes So moving forward we need -
Yes.
os.system
YAML
. This may be another PR@vasujain275 @kb-0311
The workflow is failing.
This needs to be fixed. It states there is no space on the system, this could also be an authorization issue as I have seen erroneous space related errors related to access.
@palisadoes
The cloud instance of talawa-api was out of storage due to old containers and images created by workflow:
Added the step to remove all the unwanted images and containers after the talawa-api containers are up and running in deploy.yaml workflow in this pr - https://github.com/PalisadoesFoundation/talawa-api/pull/1687
Already fixed the deploy.sh file on the GoDaay VPS and tested the workflow too.
Do not put the checks in the YAML. This may be another PR
@palisadoes Should I only add python scripts to pr for now and leave deploy.yaml workflow unchanged? Please clarify.
We will need to move the functionality of deploy.yaml
over to push.yaml
where all our other push logic is located.
@vasujain275
The workflow is failing again:
@palisadoes It failed because I was testing python scripts on the vps, otherwise it is working fine
deploy.yml
functionality over to push.yml
@palisadoes
deploy.yml
to push.yml
.Yes, that can be done with a separate python deployment script just for creating the .env
file. It can
.env
fileargparse
for each required variable retrieved from the appropriate GitHub secrets@palisadoes The deploy action to Cloud API will be down for a few hours as I am migrating it to a new process and python scripts. I will raise a PR once I am done testing the new scripts thoroughly.
@palisadoes The deploy action to Cloud API will be down for a few hours as I am migrating it to a new process and python scripts. I will raise a PR once I am done testing the new scripts thoroughly.
Hi, actually I saw that there are no tests for Redis Caching Services. I have an idea to do this, @palisadoes told me to discuss it with you:
1) We would not do unit tests as mocking can bypass the tests easily.
2) Rather we can use integration tests in which we can directly contact with a testing client which would be running on a different port.
3) Dependency injection can be done with tsyringe
Your insights would be helpful! @xoldyckk @EshaanAgg
@Manik2708, great idea, but we need a better rate limiter for our API right now. I have thought about implementing it with Redis. Currently, I am working on two different issues. Maybe you can first implement the rate limiting using Redis. After that, we can test Redis caching, as the rate limiter is of higher priority and will be beneficial to the project a lot.
@Manik2708, great idea, but we need a better rate limiter for our API right now. I have thought about implementing it with Redis. Currently, I am working on two different issues. Maybe you can first implement the rate limiting using Redis. After that, we can test Redis caching, as the rate limiter is of higher priority and will be beneficial to the project a lot.
Is the issue opened for this? Like can I get more context about rate limiting?
@Manik2708, great idea, but we need a better rate limiter for our API right now. I have thought about implementing it with Redis. Currently, I am working on two different issues. Maybe you can first implement the rate limiting using Redis. After that, we can test Redis caching, as the rate limiter is of higher priority and will be beneficial to the project a lot.
Is the issue opened for this? Like can I get more context about rate limiting?
Also, I saw that Dependency Injection through tsyringe
doesn't provide control over file execution. For better and strong architecture we have to use Nest.js
I also raised the issue regarding this in their library: https://github.com/microsoft/tsyringe/issues/235, So there is still a problem of Dependency Injection!
We need to have a cloud based API instance for developers to test against. It must:
Please coordinate these efforts with @noman2002 and @kb-0311. Ask to be assigned this task by them.