search

awslabs / dynamodb-cross-region-library

A library to facilitate cross-region replication with Amazon DynamoDB Streams.
Apache License 2.0
275 stars 98 forks source link

ElasticBeanstalk RED due to JDK7 end of life #23

Closed leevlad closed 8 years ago

leevlad commented 8 years ago

Following the steps in the official guide here: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.CrossRegionRepl.Walkthrough.Step2.html

The ElasticBeanstalk application is RED due to the error below. Problem is that the Dockerfile used for the coordinator doesn't version-lock ubuntu, so the latest images are built against xenial, which no longer includes openjdk-7-jre. Ref: https://github.com/awslabs/dynamodb-cross-region-library/blob/master/cloud-formation/coordinatorContainer/Dockerfile#L1

I am putting together a PR to fix this. Edit: here it is https://github.com/awslabs/dynamodb-cross-region-library/pull/24

I'm more surprised that noone noticed until now.

Error:

 ---> Running in ae2ccaa9a39a
  Reading package lists...
  Building dependency tree...
  Package openjdk-7-jre is not available, but is referred to by another package.
  This may mean that the package is missing, has been obsoleted, or
  is only available from another source

  E: Package 'openjdk-7-jre' has no installation candidate
  The command '/bin/sh -c apt-get -y install openjdk-7-jre python-setuptools wget jq' returned a non-zero code: 100
  Failed to build Docker image aws_beanstalk/staging-app, retrying...
  Sending build context to Docker daemon 12.29 kB
dymaws commented 8 years ago

Thanks for the PR, good catch.

leevlad commented 8 years ago

When can I expect this to be deployed in Prod?

dymaws commented 8 years ago

It's been deployed.

leevlad commented 8 years ago

Thanks so much for the prompt response. Cheers!

leevlad commented 8 years ago

@dymaws I believe this docker image needs to be updated: https://hub.docker.com/r/dynamodbecosystemdev/dynamodbcrossregionreplication/tags/

dymaws commented 8 years ago

That image is used exclusively by Amazon EC2 Container service, it's built from a previous Ubuntu release with JDK7 then remains frozen thereafter so I don't think it needs to be updated.

leevlad commented 8 years ago

@dymaws I am still getting the same error when spinning this up. When I download the DynamoDBReplicationCoordinatorApplicationBundle.zip from ElasticBeanstalk, the Dockerfile doesn't include latest changes.

dymaws commented 8 years ago

@leevlad I am not sure if rebuilding the Elastic Beanstalk environment forces a re-download of the Docker container zip. However, if you check the zip source manually, you can see the change has been deployed: https://s3.amazonaws.com/dynamodb-cross-region/DynamoDBReplicationCoordinatorApplicationBundle.zip

If all else fails, you may have to delete your entire CloudFormation stack and re-launch it for the new zip source to be picked up by EB.

leevlad commented 8 years ago

thanks for the info. I've actually torn down my entire CloudFormation stack this morning and this is the result I got. I will try to re-launch it again

leevlad commented 8 years ago

@dymaws just re-launched the entire CF stack into a clean environment from here: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.CrossRegionRepl.Walkthrough.Step2.html

The zip file does not include the changes made in this PR and still pulls the latest version of ubuntu (xenial), causing it to fail at the jdk7 step, same as before. This means that the zip file is either cached somewhere, or it's sourced from somewhere other than the S3 bucket you pointed at.

leevlad commented 8 years ago

I found the problem. The S3 template found here defines the location of the zip file incorrectly: https://dynamodb-cross-region.s3.amazonaws.com/dynamodb-replication-coordinator.template

Line 347-374

In reality the path it pulls the zip file from is: https://s3.amazonaws.com/dynamodb-cross-region-us-east-1/DynamoDBReplicationCoordinatorApplicationBundle.zip

and not the one you posted. The S3 object above does not contain the latest Dockerfile.

Two ways to solve this would be to update the template mentioned above or update the zip file above with the latest code.

As a side note, appending region name to the bucket name is not how S3 regionalization is supposed to be implemented. There are separate endpoints for each region, according to the docs: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

dymaws commented 8 years ago

Thanks for the detailed analysis - looks like different buckets for each region was created because Elastic Beanstalk only supports bundle zips from the local region, hence there is a separate copy in each regional bucket. All zip files should be deployed in all regions now, sorry about the confusion.

dymaws commented 8 years ago

BTW, here is a blog post outlining the scenario for Elastic Beanstalk http://blog.mikebabineau.com/2013/08/21/multi-region-gotcha-on-elastic-beanstalk/