search

villasv / aws-airflow-stack

Turbine: the bare metals that gets you Airflow
https://victor.villas/aws-airflow-stack/
MIT License
377 stars 69 forks source link

cfn-init missing region parameter fails outside us-east-1 #168

Closed ztyree42 closed 4 years ago

ztyree42 commented 4 years ago

When creating stack in us-west-1 cfn-init fails with stack-id not found.

Adding --region to the UserData:Fn property on LaunchConfiguration solves.

villasv commented 4 years ago

Hmm, for some reason the deployment worked for us-east-2 (supposedly, because it's in the test suite). But I'll check it out, thanks for reporting!

villasv commented 4 years ago

Confirmed. Deploying on us-west-1 failed with:

Error occurred while GetObject.
S3 Error Code: PermanentRedirect.
S3 Error Message: The bucket is in this region: us-east-1.
Please use this region to retry the request

Tests pass because taskcat by default has s3_regional_buckets enabled.

villasv commented 4 years ago

In order to solve this I'm going to need to adjust the CD pipeline to publish the lambda functions package to a different bucket for every region. I'm going to take a look at how other AWS QuickStarts implement this.

villasv commented 4 years ago

Adding --region to the UserData:Fn property on LaunchConfiguration solves.

This is bugging me. It shouldn't be enough to fix it, since in my case the deployment failed on the CloudWatch metric and not the EC2 related resources. It worries me that you might be using an outdated template file. Are you using a particular release version? If so, which one?

ztyree42 commented 4 years ago

I cloned your master. And you're right, my apologies; I forgot to mention that I grabbed the lambda code from your us-east bucket and copied it into a bucket in us-west. But even doing that is not enough without also explicitly passing region to userdata otherwise it doesn't see the stack-id.

On Sat, Apr 25, 2020, 8:00 PM Victor Villas notifications@github.com wrote:

Adding --region to the UserData:Fn property on LaunchConfiguration solves.

This is bugging me. It shouldn't be enough to fix it, since in my case the deployment failed on the CloudWatch metric and not the EC2 related resources. It worries me that you might be using an outdated template file. Are you using a particular release version? If so, which one?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/villasv/aws-airflow-stack/issues/168#issuecomment-619472668, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECDZPPWNFN3T2LZV3EYBETROOPVPANCNFSM4MQQG32Q .

villasv commented 4 years ago

Still, AFAIK when the script in UserData fails, the stack deployment won't fail in CloudFormation. When you say "stack creation fails", what error do you experience? In the cfn-ini script?

ztyree42 commented 4 years ago

It isn't the init, stack creation fails when attempting to create the cloudwatchmetriclambda resource at which point a rollback occurs and all the stack resources are deleted. I get this information by looking at the events login they cloudformation UI.

But if I don't also add region to user data then cfn-init fails with cannot find stack name or some such.

sorry I'm on my phone right now so I can't give you the exact error message.

On Sun, Apr 26, 2020, 2:06 PM Victor Villas notifications@github.com wrote:

Still, AFAIK when the script in UserData fails, the stack deployment won't fail in CloudFormation. When you say "stack creation fails", what error do you experience? In the cfn-ini script?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/villasv/aws-airflow-stack/issues/168#issuecomment-619624881, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECDZPJTZE7NLYBNUOMSPZTROSO5RANCNFSM4MQQG32Q .

villasv commented 4 years ago

No problem, you've guided me quite a lot already. I was able to fix the CloudWatch metric thing, so we're good with the deployment thing. But now I've confirmed that the cfn-init script won't run correctly if I don't specify the region.

I created a separate issue for the lambda thing, so I hope you don't mind if I rename this issue to be specifically about the region parameter thing.

ztyree42 commented 4 years ago

Of course not, glad I could help. I'm excited to get the cluster stack deployed into my vpc.

On Sun, Apr 26, 2020, 4:38 PM Victor Villas notifications@github.com wrote:

No problem, you've guided me quite a lot already. I was able to fix the CloudWatch metric thing already, so we're good with the deployment thing. But now I've confirmed that the cfn-init script won't run correctly if I don't specify the region.

I created a separate issue for the lambda thing, so I hope you don't mind if I rename this issue to be specifically about the region parameter thing.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/villasv/aws-airflow-stack/issues/168#issuecomment-619642747, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECDZPMKZ37UJJEEQXZ2OT3ROTAWPANCNFSM4MQQG32Q .