search

openassistive / OpenATBackend

Back end api for Open Assistive built using nodeJS.
https://api.openassistive.org
4 stars 1 forks source link

A solution for continuous deployment #10

Open hosseinzoda opened 7 years ago

hosseinzoda commented 7 years ago

Why It's needed?

  1. Netlify has request rate limit. https://www.netlify.com/docs/api/#rate-limiting
  2. Making it reliable and scalable

My suggestion

By setting up deploy window we can fix this issue.

What's good about it

  1. We can determine the next deploy time and show it to the user.
  2. The current setup is compatible with it.

Core: Depending on few variables and the state of deployment determine next deploy or git push trigger

hosseinzoda commented 7 years ago

This also helps with handling concurrent add requests.

hosseinzoda commented 7 years ago

To clarify. I mean we should manage when to actually do add project. https://github.com/openassistive/OpenATBackend/blob/master/js/savejson/savejson.controller.js#L43

So first thing comes to mind is to create a queue of add projects until deploy window finishes. And then push to git

hosseinzoda commented 7 years ago

Anyway this is not an issue for now.

willwade commented 7 years ago

There are problems creating our own CI system:

  1. Time and effort
  2. Where is it hosted. We currently run the api off heroku off a low-priced tier. Its poor for hosting content.
  3. We could use github pages - however its difficult to get SSL on this. We want this to be as low-cost and as flexible to re-host as possible. Using cloudflare, etc to get round these issues is a headache..

So the real issue is:

  1. Netlify has rate limiting. In the first 6 months we may have a lot of adds' causing us to break the limit. thats not good

And on a side-note:

  1. Im worried about spam/poorly written entries. i.e. there is no moderation. Equally though I'm concerned that in a moderated system we won't get round to moderating stuff quick enough..

Hmm. How about:

AND

hosseinzoda commented 7 years ago

we tag all entries as "to-be-moderated" if an item has "to-be-moderated" tag then we colour it or label it different on the frontend

Correct, This helps a lot on filtering contents. Also we can associate an ip address of issuer to it. So we can easily remove spams.

There are problems creating our own CI system:

I did not suggest to create another CI. My suggestion was close your instructions. By limiting amount of deploy request this will get fixed. (Also con-current requests will get fixed) Here's the instruction for my suggestion

  1. On /save add the save content to the queue. And set a cron-job on the server for next check for processing deploy.
  2. On deploy commit to github.

Now next time to process a deploy may vary depending on few parameters and state of deployment.

willwade commented 7 years ago

Ah ok. Makes sense. I think we are agreeing then!

One small problem though with a queue.. What if someone goes and adds something directly to GitHub inbetween. Wouldn't this create a conflict

(Maybe so unlikely we ignore this edgecase)

Instead - commit everything to GitHub as we do. And only deploy once an hour. To reduce over hitting gitHub however keep the local queue to 1 minute.

(Just leaving this here: https://devcenter.heroku.com/articles/scheduler#installing-the-add-on. Basically we just need a service e.g. /deploy that is accessible via the web. Would be useful to restrict this to an IP)

hosseinzoda commented 7 years ago

/save does know what to do on conflict. It will replace it.

(Just leaving this here: https://devcenter.heroku.com/articles/scheduler#installing-the-add-on. Basically we just need a service e.g. /deploy that is accessible via the web. Would be useful to restrict this to an IP)

Correct.

willwade commented 7 years ago

/save does know what to do on conflict. It will replace it.

Yep. Whats your thoughts on:

Instead - commit everything to GitHub as we do. And only deploy once an hour. To reduce over hitting gitHub however keep the local queue to 1 minute.

?

hosseinzoda commented 7 years ago

Instead - commit everything to GitHub as we do. And only deploy once an hour. To reduce over hitting gitHub however keep the local queue to 1 minute.

I think It's dependent to build time (time it takes to complete a deploy process). Like if It's 10 seconds then margin between deploy triggers can be one minute.

Github does not have much to do on changes occur, I think It's inexpensive task.

Anyway one or few build(s) every minute will not exceed rate limit.

hosseinzoda commented 7 years ago

We can write some code to figure this out. depending on previous build time.

willwade commented 7 years ago

Its quick right now. My guess under 2 minutes. but yeah - we can work out exact timing as we go