search

drone / autoscaler

Automatically adds or removes instances based on build volume
https://autoscale.drone.io
Other
193 stars 89 forks source link

Update outdated default image for AWS/GCP #75

Closed guessi closed 2 years ago

guessi commented 4 years ago

AMI information comes from upstream Ubuntu Cloud Image Finder

Or alternative, retrieve from scripts:

AWS:

curl -fsSL https://cloud-images.ubuntu.com/locator/releasesTable | \
    awk -F '"' '/Amazon.*focal.*amd64.*20220706*/{print$4" "$18}' | \
    sed 's/\>//;s/\<\/a\>//' | \
    column -t | \
    sort

GCP:

curl -fsSL https://cloud-images.ubuntu.com/locator/releasesTable | \
    awk -F '"' '/Google.*amd64.*ubuntu-2004-focal-v20220712.*/{print$4" "$16}' | \
    column -t | \
    sort

Last Update: 2022.07.16

mtb-xt commented 4 years ago

@bradrydzewski @tboerger is there any chance we could merge this? It would be awesome if we could have AWS SSM agent in our drone agent instances...

ademariag commented 3 years ago

Any chance that we could push this at one point?

guessi commented 3 years ago

rebase, and update to current latest images

ademariag commented 3 years ago

@bradrydzewski @tboerger is there any chance we could merge this?

tboerger commented 3 years ago

I can't merge on this repo, it's up to @bradrydzewski

ademariag commented 3 years ago

Is this project dead? I don't see any updates lately and lots of valuable contributions are not being accepted. @bradrydzewski could you add more maintainers perhaps from the community?

guessi commented 3 years ago

FYI, just update the default images available on AWS/GCP

guessi commented 2 years ago

Two years later, it finally got approved 🎉

For long-term consideration, it feels like we need an auto discovery mechanism for discovering new image id for specific requirement of the platform. Those image ids are now 6 month old 👶 🍼 since my last commit.

Let me know if image ids update is required before it get merged.

guessi commented 2 years ago

FYI, just refreshed AMI IDs and squash commits, also I added support for AWS new region: ap-southeast-3 (Asia Pacific, Jakarta).

bradrydzewski commented 2 years ago

@guessi thanks, can you please provide the link where you are sourcing the AMI list? For security purposes, we need to manually verify before we can merge (sorry, this is a security requirement, trust but verify ...)

guessi commented 2 years ago

@bradrydzewski sure, security always first !!!

Let me know where should I put the link, put it here as PR description or as comment in code?

Update: I've update the PR description for where the AMI info comes from.

bradrydzewski commented 2 years ago

I noticed these images are being labeled as hvm-ssd but the autoscaler does not use ssd disk types by default. Have you tested any of these new amis with default settings to ensure there is no risk of regression?

guessi commented 2 years ago

@bradrydzewski I personally only use SSD nowadays and I have used these image as source for a long time :-)

If this is really a concern, we can alternatively retrieve image id from awscli instead, which might be much reliable for user?

aws ssm get-parameters \
    --names /aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id

From trustworthy source mentioned in Ubuntu Official Documentation[1].

or

aws ec2 describe-images \
    --owners aws-marketplace \
    --filters "Name=owner-id,Values=679593333241" "Name=name,Values=*ubuntu-focal-20.04-amd64*" \
    --query "sort_by(Images, &CreationDate)[-1:].[Name, ImageId]" \
    --region us-east-1 \
    --output text

Where "679593333241" is Canonical's ID, where you can find how to verify it is released by Canonical here[2].

Or even more better, retrieve from AWS API.

Ref:

bradrydzewski commented 2 years ago

I have no problem with ssd or even ssd by default, I just want to make sure that if someone runs the autoscaler with default options it will work out of the box and not error. If we use images labeled hvm-ssd do we need to make any changes to volume type or will everything just work? I am asking because I do not use AWS so the answer to this is non-obvious to me.

guessi commented 2 years ago

Your concern is not exist, since hvm-ssd have been used for quite long time.

To verify that we can easily find existed image ids for us-east-1[1], us-west-1[2] and you will find that current images are hvm-ssd already.

AWS US-EAST-1

$ aws ec2 describe-images --region us-east-1 --image-ids "ami-43a15f3e" --output json | grep "ImageLocation" # image id for us-east-1
"ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180306",

AWS US-WEST-1

% aws ec2 describe-images --region us-west-1 --image-ids "ami-925144f2" --output json | grep "ImageLocation" # image id for us-west-1
"ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180306",

** 810c77f is current master commit hash.

bradrydzewski commented 2 years ago

you will find that current images are hvm-ssd already.

thanks! I think this is the context I was missing

guessi commented 2 years ago

Just update once again after month.

guessi commented 2 years ago

Umm... looks like I need to resolve conflict and bump the image version again. Let me handle it.

guessi commented 2 years ago

Just update the the image id for AWS and GCP once again.

At the moment of I created this PR (Sep 11, 2020) the latest version available was Ubuntu Server 20.04 LTS and 2 years later, it's now 2022, Ubuntu Server 22.04 LTS was released on Apr 22, 2022.

Feel free to let me know if it is needed to update to Ubuntu Server 22.04 LTS.

maxgruebneraeroqual commented 2 years ago

We've run into an issue with builds that was sorted by switching to Ubuntu 22.04 AMIs.

https://community.harness.io/t/drone-autoscaler-amis-too-old-for-net-6-application/12249/5

Further, we've just got an email from AWS about the 16.04 Ubuntu AMIs being deprecated.

tphoney commented 2 years ago

thanks for the continued work on this PR !

solid work !!