search

scylladb / scylla-machine-image

Apache License 2.0
18 stars 25 forks source link

Cannot get EC2 user data after enabled IMDSv2 #498

Closed vincentwenatsa closed 4 months ago

vincentwenatsa commented 6 months ago

Dec 19 15:38:09 ip-172-19-20-73 scylla_post_start.py[3760]: 2023-12-19 15:38:09,968 - [user_data] - WARNING - Error getting user data: HTTP Error 401: Unauthorized. Will use defaults!

Dec 19 15:38:09 ip-172-19-20-73 scylla_post_start.py[3760]: Error getting user data: HTTP Error 401: Unauthorized. Will use defaults!

the scylla_post_start.py script failed to fetch user data after enabled IMDSv2. It needs to fetch a IMDSv2 token https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html

fruch commented 6 months ago

it was introduced a year ago in: https://github.com/scylladb/scylla-machine-image/commit/b1c8990248e37039323539fa2ea32668173561b2

which version of scylla AMI you are using ?

vincentwenatsa commented 6 months ago

5.2.10 @fruch

vincentwenatsa commented 6 months ago

Will it be backport to any 5.2 version?

marqueurs404 commented 6 months ago

I am encountering this on 5.4.0 scylla AMI as well, on both the scylla_post_start and scylla-image-setup services

marqueurs404 commented 6 months ago

I presume it's because the fetched token wasn't used in https://github.com/scylladb/scylla-machine-image/blob/26b93d5cebd2762d612b79d4642705958e2eb804/lib/scylla_cloud.py#L819 and https://github.com/scylladb/scylla-machine-image/blob/26b93d5cebd2762d612b79d4642705958e2eb804/lib/scylla_cloud.py#L821

This bug was likely only noticed recently as EC2 defaults to IMDSv2-only since November 2023 https://aws.amazon.com/blogs/aws/amazon-ec2-instance-metadata-service-imdsv2-by-default/

fruch commented 6 months ago

@syuu1228

We should consider setting the v2 usage in the AMI once this issue is fixed

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-IMDS-new-instances.html

fruch commented 6 months ago

@marqueurs404

Thanks for spotting it out, seems you are correct, and some calls were missed

mykaul commented 6 months ago

https://github.com/scylladb/seastar/issues/1051 https://github.com/scylladb/scylla-manager/issues/3244

(and we have some others, that are not completely IMDSv2 friendly)

fruch commented 6 months ago

scylladb/seastar#1051 scylladb/scylla-manager#3244

(and we have some others, that are not completely IMDSv2 friendly)

exactly why I've suggested enabling in the AMI, to flush all those out.

vincentwenatsa commented 5 months ago

Will this issue get fixed any time soon?

yaronkaikov commented 5 months ago

@syuu1228 ping

yaronkaikov commented 4 months ago

@vincentwenatsa once this fix is promoted. we will backport to 5.4 and 5.2, it will be part of next release