Overhaul deployment

[cbizon]

There is currently a lot of effort involved in deploying an automat. There are numerous helm charts, because each plater has its own helm chart (or at least its own values file). So there is a lot of updating of values in different places.

This was annoying but manageable in the past, but we are now moving into a situation where we want to have multiple deployments of automat, each with its own set of platers. Specifically, we will have one for robokop and one for translator. We might put, for instance, some things in the robokop U24 instance that are not appropriate for Translator. Or they may need to be on different versions for a while.

One approach that we have been considering: Creation of a deployment metadata document. This could be a simple yaml file that points at individual versioned graphs, and the minimal things that are going to change from version to versoin. Then something reads that document and templates out the relevant helm charts

[cbizon]

@EvanDietzMorris @jdr0887 @YaphetKG what are your thoughts?

[EvanDietzMorris]

There is a lot to talk about here but I thought I'd first add the most simple overview of the issue to help bring @jdr0887 up to speed on the current situation.

Right now ORION starts with a graph spec yaml and creates a bunch of graphs (a neo4j dump, kgx files, and a few metadata files) and outputs a list of the graph ids and new versions it just built that looks like: Graph_1 abc123abc123 Graph_2 def345def354 Graph_3 asdqw1231awds

I have a script that takes that list and moves over all the files to our publicly exposed directory on stars and spits out all the values that would go into the helm charts, like this: dataUrl: https://stars.renci.org/var/plater/bl-3.2.1/RobokopKG/b0ed1e26f7f3726e/graph_b0ed1e26f7f3726e.db.dump metadataUrl: https://stars.renci.org/var/plater/bl-3.2.1/RobokopKG/b0ed1e26f7f3726e/RobokopKG.meta.json metaKGUrl: https://stars.renci.org/var/plater/bl-3.2.1/RobokopKG/b0ed1e26f7f3726e/meta_kg.json testingDataUrl: https://stars.renci.org/var/plater/bl-3.2.1/RobokopKG/b0ed1e26f7f3726e/sri_testing_data.json

Right now, I have to copy and paste those values into the existing helm charts. It would be easy to write a script to overwrite the helm chart values files with those, but currently the helm charts are named something different than the graphs themselves so it'd be slightly more complex than generating file paths with the graph IDs. Then we deploy them all manually with helm commands (there are separate issues about how sometimes it isn't enough to do a helm upgrade, sometimes we have to full reinstall).

It would be really nice if we could, in ORION, somehow specify which helm charts need to be updated and do all of this automatically, and also decouple the graphs from specific deployments, so like Chris said maybe separate files from the Graph Spec, a Deployment Spec, that could specify a number of different deployment endpoints.