search

taskcluster / cloud-mirror

cloud mirror
Mozilla Public License 2.0
0 stars 7 forks source link

Single queue #13

Closed jhford closed 8 years ago

jhford commented 8 years ago

@jonasfj i reworked the queue to make it so we have a single queue per environment. There's also a couple of things related to putting the frontend on heroku. This isn't ready to deploy into production, but I just wanted to see what your thoughts are.

I'm also going to split out the QueueManager class into its own library at some point because

  1. it's generally useful
  2. there's no 'simple' sqs system
  3. the unit tests are really delicate because of queue purging
  4. it shouldn't change as frequently as the rest of cloud-mirror

The whole registerQueue() function thing is because there's a bidirectional dependency between cache and queue managers. I correctly handle the case where we have more than one queue registered as well as the case where I have multiple backends that share an id.

jonasfj commented 8 years ago

I gave this a quick skim... Didn't get into details with all the files.

Note: I think we need instrumentation before we ship this. @imbstack could be helpful with a lot of hints on how to easily had this... Particularly cache miss and timeouts where we redirect to original URL.

jonasfj commented 8 years ago

note, I'm sort of hoping @imbstack is willing to help increase the bus factor on this project a bit. It's a very critical piece and I'm not quiet sure I follow everything that's going on (I'll admit some of that can attributed to laziness on my part).

imbstack commented 8 years ago

I'm happy to help out adding tc-lib-monitor to this as well for instrumentation purposes. The readme should explain most of how to use it, but I'm sure something confusing will come up.

imbstack commented 8 years ago

How's this looking? I'm happy to help out however with this.