Failed to deploy on AWS Elastic Beanstalk

mohataher commented 6 years ago

Failed to deploy on AWS Elastic Beanstalk

When deploying on aws using command

jhipster aws

An error occurs ERROR! Resource is not in the state dBInstanceAvailable. Here's a stack trace.

It seems like the issues is at this line where the DB URL is created.

....

Create S3 bucket
Bucket myappwebapp already exists

Upload WAR to S3
War uploaded successful

Create database
Database myappwebapp already exists

Waiting for database (This may take several minutes)
/home/mohataher/dev/web_projects/myappwebapp/node_modules/aws-sdk/lib/request.js:31
            throw err;
            ^

Error: ERROR! Resource is not in the state dBInstanceAvailable
    at Environment.error (/home/mohataher/dev/web_projects/myappwebapp/node_modules/yeoman-environment/lib/environment.js:157:40)
    at module.exports.error (/home/mohataher/dev/web_projects/myappwebapp/node_modules/generator-jhipster/generators/generator-base.js:2064:18)
    at rds.createDatabaseUrl (/home/mohataher/dev/web_projects/myappwebapp/node_modules/generator-jhipster/generators/aws/index.js:200:30)
    at Response.rds.waitFor (/home/mohataher/dev/web_projects/myappwebapp/node_modules/generator-jhipster/generators/aws/lib/rds.js:55:13)
    at Request.<anonymous> (/home/mohataher/dev/web_projects/myappwebapp/node_modules/aws-sdk/lib/request.js:364:18)
    at Request.callListeners (/home/mohataher/dev/web_projects/myappwebapp/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/home/mohataher/dev/web_projects/myappwebapp/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/home/mohataher/dev/web_projects/myappwebapp/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/home/mohataher/dev/web_projects/myappwebapp/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/mohataher/dev/web_projects/myappwebapp/node_

Motivation for or Use Case

This is done at the deployment of the app on Elastic Beanstalk

Reproduce the error

Create new project using MySQL at production environment.
run jhipster upgrade to use latest version.
Create an entity (optional)
Deploy using command jhipster aws. Following steps in this guide.

JHipster Version(s)

##### **Operating System** Using Ubuntu 16.

jdubois commented 6 years ago

It looks like this is because your database already exists and/or cannot be reached. Could you try with no database?

mohataher commented 6 years ago

It seems like I didn't have appropriate permission for my AWS account. On Elasticbeanstalk, I got this error message

A Service Role is required in this region. Please add a Service Role option to the environment. Refer to the Elastic Beanstalk documentation for more information.

Once I gave it the appropriate permission, the jhipster aws command terminated successfully.

Thanks for the help.

mohataher commented 6 years ago

I will have to reopen this issue. We still have the same problem. The deployment fails because there is no supplied (or created) service role in the deployment region.

A Service Role is required in this region. Please add a Service Role option to the environment. Refer to the Elastic Beanstalk documentation for more information.

To get a deeper look at the problem, I reviewed eb.js code and seems like there is no supplied service role.

Chatting with AWS guys, I got this answer

I have checked the EB(Elastic Beanstalk) events for the mentioned environment name and found that there was no service role attached it the EB Environment and I assume that is most likely to be the reason it is failing to create an EB environment.

... When you create an environment by using the Elastic Beanstalk API, and don't specify a service role, Elastic Beanstalk creates a service-linked role. This is a unique type of service role that is predefined by Elastic Beanstalk to include all the permissions that the service requires to call other AWS services on your behalf. So ,EB uses service role to interact with other services and this has nothing to do with the IAM user permissions.

To get this sorted, a service role needs to be added here.

A proposed change would be something like this:

const applicationParams = {
        ApplicationName: applicationName,
        VersionLabel: versionLabel,
        AutoCreateApplication: true,
        ResourceLifecycleConfig: {
             ServiceRole: 'STRING_VALUE' // IAM role that has full access to S3 and EB on required region
        }
        SourceBundle: {
            S3Bucket: bucketName,
            S3Key: warKey
        }
};

Now two questions:

Could the service role be the same as the IAM role passed on application-prod.yarn file? I'm still investigating this atm. If so, how could this IAM role be accessed in eb.js file?
What kind of acceptable test procedure to JHipster community should I take to ensure this works properly?

FYI, I'm currently using JHipster with Java and Angular 4 and all my tests will be based on that. I do have access to AWS and could happily test that.

jdubois commented 6 years ago

This looks pretty strange to me:

Please note I don't know AWS Beanstalk, and that the person who coded this doesn't seem to be willing to work on this anymore
But we have quite a lot of deployments, for at least 2 years, using this sub-generator, so it looks pretty solid

So unfortunately I don't think any of us can help you here, as we have no AWS credit to test this, but I'm very surprised if you are the only one with this issue.

Now if you're willing to work on this and improve/fix this sub-generator, you are most welcome, as I don't think anybody else can help you here.

jdubois commented 6 years ago

Oh great news: I could test this :-) So you are totally correct - and I have no idea what other people do, because that must fail all the time.

A fix from you would be most welcome - and maybe some better documentation on this? And for the test procedure, unfortunately we have nothing automated on AWS - so I will test this manually and merge if that works for me.

Please tell me if you are available for a fix, or I'll have a look myself.

mohataher commented 6 years ago

Julien, thank you for looking at this. I have tried several steps to resolve this issue with no success.

Created a local service role manually on my AWS environment. That didn't change anything.
Injected the service role in eb.js in OptionSettings. That didn't solve the issue.

At this stage, neither the customer support at AWS nor myself know what to try next. So if you can take this any further, I would be really grateful. I'm also happy to test this out since I have some credit on AWS.

jdubois commented 6 years ago

Unfortunately I already spent a couple of hours on this and I have no idea what to do... I'm guessing AWS changed their APIs, as this has worked for a long time without any issue, but now we're kind of stuck :-(

mohataher commented 6 years ago

Thank you Julien, could you please explain what steps you made and their outcome? That will be a good resource for any future work on this issue.

jdubois commented 6 years ago

Well, I'm lost with this:

ResourceLifecycleConfig: {
             ServiceRole: 'STRING_VALUE' // IAM role that has full access to S3 and EB on required region
}

I have no idea what to put as ServiceRole, or which roles I should give to my user. This looks like the only issue, otherwise it should work well.

I'm sorry, that's the first time I try this, and AWS is very complex.

ruddell commented 6 years ago

I also spent some time on this and couldn't figure it out. It errors at the same point.

We do have a new aws-containers subgenerator coming out hopefully in v5.

jdubois commented 6 years ago

Yes @ruddell we're working on the aws-containers sub-generator and I hope it can be ready very soon. However, it's a different sub-generator, as it relies on Amazon Fargate:

It's a totally different approach: you have a lot more flexibility, but with a much more complex configuration
It's probably going to cost a lot more $$
It takes much more time to run an application, as we launch many more services
It's only available in us-west-1 at the moment, as it's still very young

For me we should have:

jhipster aws with Beanstalk for simple applications
jhipster aws-containers with Fargate for complex applications

They are different approaches and markets.

ggotti commented 6 years ago

The ServiceRole issue is interesting. If you've previously create an ElasticBeastalk application via the AWS Console, then it will automagically create a "aws-elasticbeanstalk-ec2-role" instance profile role, and a "aws-elasticbeanstalk-service-role" service role for you. When you run this generator afterwards, which doesn't define an explicit role, then it will work. If you haven't previously run anything via the UI, then you're currently out of luck.

The service role appears to be used for things like cleaning up instances and monitoring. The error you've reported is briefly mentioned here, and I suspect it has something to do with the permissions of the account you're using to provision from. Are you able to create a standard Java app from the UI? The error is a little misleading because roles are global.

@jdubois once the other PR has gone in, we might want to look at porting some of the environment checking behaviour into here to resolve some of these types of issues.

jdubois commented 6 years ago

Hi @ggotti - now that aws-containers is merged, would you have time to work on this?

ggotti commented 6 years ago

@jdubois I'll have a look at it now :)

jhipster / generator-jhipster