search

Shyp / rickover

A job queue and scheduler written in Go, backed by Postgres, and available over HTTP
https://godoc.org/github.com/Shyp/rickover
MIT License
182 stars 21 forks source link

Proposal: Cache jobs config on start #12

Open caseycrites opened 7 years ago

caseycrites commented 7 years ago

Since job configuration requires a restart to be loaded, cannot (currently) be updated, and is very small (would likely scale to 100's of jobs at least), I'd like to propose caching the jobs config on start. In addition to removing a database query when getting job info, this would make supplying job config information to the downstream worker a breeze (#5).

I was initially thinking this could be done here, but am open to other suggestions.

Thoughts overall? Happy to take this on if we can agree this is a good idea.

kevinburke commented 7 years ago

This seems reasonable; I wish I could add updating but it'd be easy enough to switch later as well.

I was initially thinking this could be done here, but am open to other suggestions.

It's probably a bad idea to make network calls in the init() package, which is usually used to initialize in-memory things your library needs like flags or variables. It would also mean that we'd have to ensure the database is configured and connections are established before that init() runs and in general you can't guarantee ordering of init() function calls - you can define as many of them as you want.

https://golang.org/doc/effective_go.html#init

caseycrites commented 7 years ago

Yeah, I agree that init likely isn't a great place to make network calls. The other two options in my mind (at this moment) seem to be:

  1. Lazy load each jobs' config (which is fine, just seems unnecessary), or
  2. create some sort of Configure (or whatever) method that does the caching (and is easy to forget to do)

and neither one of those sounds great to me atm. I'll think a bit more about it and see if there are other options.