search

wlandau / crew

A distributed worker launcher
https://wlandau.github.io/crew/
Other
123 stars 4 forks source link

future work #18

Closed wlandau closed 1 year ago

wlandau commented 2 years ago

Is this a good time to look at cloud backends? Maybe an AWS Batch queue with an EFS data store? Or should crew step aside for rrq?

wlandau commented 2 years ago

I got some promising ideas from David Kretch and Adam Banker from a chat a couple weeks ago.

ElastiCache can host a Redis server with a bunch of networking stuff already figured out, but it's managed and pricey. (Note: configured with Terraform.)

For the Redis server approach, all workers need to share the same VPC and allow traffic on the appropriate ports. Security groups can handle the ports.

Security: for a local connection to the Redis server, authentication, a one-time generated password, and secure TLS encryption is apparently as good as it gets. Apparently TLS encryption is the hardest part because you need a domain name. Apparently ElastiCache handles that too. But apparently, RStudio images are deployed like this.

I wonder about efficiency. If the Redis server is itself in a remote worker, that's a lot of throughput from the Redis server to the local controlling process. If the Redis server could somehow (securely) exist locally, would that be faster? Would it parallelize the potential bottleneck, or would it just generate a ton more requests on the slowest piece of the network? A remote Redis server is more secure anyway, so I suppose I should try that first and see if it's a problem.

At this point I am venturing into unfamiliar territory, so I am taking a long break from implementation in order to educate myself. I plan to slowly work my way through AWS in Action and see if I can get AWS promotional credits for prototyping. Given all the other projects I have, I expect this to be extremely slow. I do care about this problem and intend to see it through. But if anyone else with more knowledge or free time speeds ahead of me, I will not mind.

wlandau commented 1 year ago

Closing in favor of #21.