Deployment strategy for Voyager

[aef-]

We obviously need a better way to manage deployments. This is less about potential scaling issues, which will eventually be bottlenecked by the number of jobs on HPC, and more about creating an easier and more reliable way to deploy.

Based on what HPC supports, I am thinking we could move most of the services/DBs to AWS and have a couple of processes running on HPC that we interact with through something like gRPC/OpenAPI (formerly Swagger) that handle commands like bsub/bjobs, checksums, copying files, etc.

The thinking behind this is:

• We'll be able to easily deploy/rollback and track releases. As of now, if we wanted to rollback to our previous Ridgeback Queue, it takes a degree of diligence on the part of whoever is doing it to make sure they deploy the correct celery service, revert the ENVARs, etc. It also becomes difficult to tell which def built which service.
• Potentially, anyone could spin up a new instance easily, e.g. a branch instance.
• We can let something like Kubernetes handle stopping and starting services as it switches over from one version to another, something that's been causing issues. It's also self-healing, if a service slows down/errors out, Kubernetes will restart it with in an effort to keep the service running in good health, while we can spend time debugging the issue.
• We get some degree of APM support -- esp with logging. I'm not entirely sure what AWS has for span/error tracing but it's something we can look into.
• We can assume that while we may add support for additional commands on HPC, they typically won't change once we do, so having a robust deployment structure for that is less a concern.
• We'd be able to separate out a celery box from the web box. Celery slows down web.

Some reasons not to do this/questions:

• Do we get enough from the additional level of abstraction/time spent to implement this -- esp. w.r.t having some type of RPC-like service that we'd have to manage to some degree?
• Can we feel confident that those RPC-like services can handle the # of requests we'll be sending them?
• Would this make integrating with a file manager like Minio easier or more difficult?

[aef-]

After speaking with Ronak -- here's another option:

ACCESS has a couple of servers with root access that Lohit set up. We could set up everything there and w/e infra we'd like to use and mount JUNO to it. I'm not 100% on how viable this is -- or what type of configuration is required on Lohit's part and if he'd be okay with it.