search

aws / amazon-genomics-cli

https://aws.github.io/amazon-genomics-cli/
Apache License 2.0
146 stars 82 forks source link

Automated job submission within AWS #287

Open multimeric opened 2 years ago

multimeric commented 2 years ago

Description

I have a use case where I want to submit pipeline jobs through the AGC, but doing so from AWS. This may seen redundant, because in theory I could submit the head node job in the same way that the AGC does. However, AGC provides a nice abstraction layer over this, so I'd rather use it. Therefore, I'm interested in a way to run the AGC on AWS.

Use Case

There are several web apps where users can submit files for analysis. Off the top of my head:

If such an app were to be hosted on AWS, we would need a way for the web app to automatically submit jobs. This could be done by building a lambda that simply runs the AGC with a set of config files, but there is currently no pre-built docker image that contains the AGC. Its dependencies are non-trivial to install.

Proposed Solution

multimeric commented 2 years ago

Actually, having a prebuilt AGC docker image may not help that much, because you would still need the lambda handler to be installed into the image, which is not trivial to add on top of an existing docker image. Since the pre-compiled release zip files have very minimal installation requirements, it would probably be easier to install them into a given lambda docker image instead.

multimeric commented 2 years ago

Perhaps a shared lambda layer?

multimeric commented 2 years ago

No, a shared layer doesn't work. Indeed a lambda zip can't ever work because a complete AGC installation is 578 Mb unzipped, which exceeds the 250 Mb maximum allowed for zip lambdas. It seems like you would have to use a docker image lambda for this.

github-actions[bot] commented 2 years ago

Greetings! Sorry to say but this is a very old issue that is probably not getting as much attention as it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to open a new one.