How use Airbyte on EC2 and ECR to pull custom connectors

[marcosmarxm]

Tell us about the documentation you'd like us to add or update.

Slack convo show users how to leverage the power of Airbyte on AWS.

If applicable, add links to the relevant docs that should be updated

• Link to relevant docs page

[schenkd]

Hey, for all those who are looking for a quick solution I have the following to offer:

• Installs amazon-ecr-credential-helper on the host
• Creates a .docker/config.json file for example like this:

  cat > /root/.docker/config.json << EOF
  {
  "credHelpers" : {
      "public.ecr.aws" : "ecr-login",
      "<your-account>.dkr.ecr.<your-region>.amazonaws.com" : "ecr-login"
  },
  "credsStore" : "ecr-login"
  }
  EOF

• Mount the binary and the config.json into the airbyte-scheduler as volume. In a script with sed it could look like this:

  sed -i $"73i\      - /root/.docker:/root/.docker" docker-compose.yaml
  sed -i $"74i\      - /bin/docker-credential-ecr-login:/bin/docker-credential-ecr-login" docker-compose.yaml

Ready! You could now pull your images from your ECR repo. Of course you have to add the appropriate IAM policy to the ec2 instance profile!

Best David

[jyates-om1]

@schenkd - Thank you for posting this. I can confirm that it still works, and this makes our workflow using airbyte much smoother

[andresgutgon]

What would be the way of doing this in 2023? We're using a newer Airbyte and looks like airbyte-scheduler is not there anymore