search

FredHutch / batchman

Batchman - Nextflow on Fargate x AWS Batch
6 stars 1 forks source link

Enable "headless" launch of workflows #81

Open nkrumm opened 4 years ago

nkrumm commented 4 years ago

For production use at UW, we need to be able to launch workflows entirely via the API. We're planning on having a workflow like this:

S3 triggers -> Lambda -> Batchman API call

Currently, when a user submits a launch request, the POST route for /api/v1/workflow looks up to see if the user is submitting to an appropriate workgroup (the workgroup also defines which AWS queue to use etc). Since an API-drive approach means there is no user session (no JWT token), this current behavior will prevent the launch from happening.

What I would propose here is that we revive the old API_KEY functionality (originally used to secure the logging endpoint, but after we moved logging to within the VPC only, it was deprecated). This API_KEY is workgroup specific, so the external lambdas/triggers attempting to start the workflow could still be authorized only for specific workgroups (by knowing the API key). Note that this functionality does not expose the launch API in an unprotected manner-- the API is still private to the VPC.

So, at the start of the /api/v1/workflow POST route, there would be a check to see that either the user is credentialed for the workgroup, OR there is a matching workgroup-specific API key. Users can only see the API keys for their own workgroup(s) so the key can't leak to users in other workgroups. To reiterate, this key does not allow workflow launches from unauthenticated web sessions.

nkrumm commented 4 years ago

Started work in https://github.com/FredHutch/batchman/tree/081-headless-launch