Migrating from Dockerhub to ECR

ababaian commented 4 years ago

Currently each instance is pulling an image from the serratusbio dockerhub repository. We can likely download this faster by pulling directly from ECR.

This can be accomplished by:

[ ] Modify the build_containers.sh script to push to ECR if the environmental variable ECR_ACCOUNT is set in the same way that DOCKERHUB_USER works. Related, if the ECR repository does not exist, it should be created on that account: serratus-dl, serratus-align etc...
[ ] For each instance, the user_data is defined in terraform launches the appropriate image/container. These will need to be changed to ECR, which may require a ECR read permission policy be added to the worker IAM
[ ] Finally, within the user_data for each instance, ECR should be tried first, and if it is unavailable, dockerhub will be a fallback. This ensures if there's a permission/config error on the instance, at least something gets booted up and the instance doesn't stall out without a container which is the worst possible case.

kobi86 commented 2 years ago

Hi @ababaian, nice to meet :) I wanted to ask how can I help with this issue? :)

ababaian commented 2 years ago

Hey @kobi86, sounds great!

Right now Serratus is still is quite challenging to get up and running (related is #22). The first step is to boot-up a Serratus cluster on another AWS account.

I have a tutorial written up on this, and it would be great to get feedback on this and make sure that there is enough information there to get it up and running. Once you can boot up a small cluster (say 1 node), we can go through the above steps to migrate the docker images to ECR.

Pop into the Slack if you'd like, we can chat about it.

avinashsingh77 commented 1 year ago

Hey @ababaian . Is this issue still open? If yes, I am interested to pick this up. Also, please let me know on how can we connect on Slack or GMeet.

ababaian commented 1 year ago

Hi @avinashsingh77 , this is actually non-critical these days. Most relevant issues for starting out are probably here: https://github.com/serratus-bio/serratus.io/issues

Shoot me an email artem AT rRNA dot ca and I can get you an invite to the slack :) Welcome!

avinashsingh77 commented 1 year ago

Hey @ababaian , I have sent you an email from my gmail id for the slack invite. Thanks!!

ababaian commented 1 year ago

You should have gotten an invite a week ago or so, check spam

ababaian / serratus

Migrating from Dockerhub to ECR #164