Amazon linux 2023 fix - Githubissues

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.

Apache License 2.0

636 stars 116 forks source link

Amazon linux 2023 fix #358

Open pragnesh opened 1 year ago

pragnesh commented 1 year ago

This PR makes the following changes:

added ebs-volume-type config which allow ebs volume type from configuration since amazon linux 2023 by default comes with gp3 volume image
updated setup-ephemeral-storage.py since new version of lsblk doesn't produce same output for setup.
moved python package installation before running setup-ephemeral-storage.py
instance data IMDSv2 fix for spark public hostname

I tested this PR by... launching spark cluster with latest amazon linux 2023 and amazon linux 2 image. both working fine

nchammas commented 9 months ago

Thanks for this PR @pragnesh and sorry to create a conflict.

Could you explain the motivation for the following changes? They seem unrelated to the subject of this PR.

moved python package installation before running setup-ephemeral-storage.py

instance data IMDSv2 fix for spark public hostname

nchammas commented 9 months ago

Did you test this PR against a private VPC btw? I'm wondering if #346 is affected at all by these changes.

pragnesh commented 9 months ago

Hi @nchammas ,

setup-ephemeral-storage.py is not running without python package,

amazon linux 2 has default python installed so it was working earlier but amazon linux 2023 need python to be installed so moved it before setting running setup-ephemeral-storage.py

instance data IMDSv2 is enabed by default for amazon linux 2023, so SPARK_PUBLIC_DNS need to be adjusted for functional spark web UI

link for Instance data difference between amazon linux 2 and amazon linux 2023

pragnesh commented 9 months ago

Did you test this PR against a private VPC btw? I'm wondering if #346 is affected at all by these changes.

no i have not. we do you VPC but , launch it with public ip enabled.

I have face similar issue with IMDSV2 so hence this fix for SPARK_PUBLIC_DNS.

I have rebased my PR.

pragnesh commented 9 months ago

Did you test this PR against a private VPC btw? I'm wondering if #346 is affected at all by these changes.

I have updated PR to correctly set SPARK_PUBLIC_DNS in private VPC. I have tested change on amazon linux 2 ami. ~I have tested amazon linux 2023 ami with java preinstalled since Adoptium does not work with amazon linux 2023.~ Hardcoded releasever to 2 since Adoptium does not have amazon linux 2023 jdk I have tried testing private vpc but due to some vpc network configuration i am unable to launch but SPARK_PUBLIC_DNS is set correctly within private vpc.

nchammas commented 7 months ago

Sorry to leave this PR hanging. It is still on my mind and I will hopefully pick it up after I get Flintrock 2.1 out.