search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
636 stars 116 forks source link

AdoptOpenJDK.jfrog.io has been deprecated #360

Closed rlaabs closed 9 months ago

rlaabs commented 11 months ago

https://adoptium.net/blog/2023/07/adoptopenjdk-jfrog-io-has-been-deprecated/

Scheduled brownouts happening now will cause cluster launches to fail.

HDFS setup also doesn't seem to work with java-openjdk11 available from amazon-linux-extras

ashizhou commented 11 months ago

@rlaabs I came across the same problems as well. Have you managed to find a solution for it?

rlaabs commented 11 months ago

@rlaabs I came across the same problems as well. Have you managed to find a solution for it?

@ashizhou Yes, I think the easiest workaround is to just install Java on your Amazon Machine Image; that way flintrock will skip the Java installation step where this is a problem.

I used the following: -- OpenJDK 11.0.19 -- Spark 3.4.1 -- HDFS 3.3.6

Don't forget to set JAVA_HOME. That was the silly mistake I initially made.

nchammas commented 9 months ago

Thanks for reporting this and including a link to the deprecation announcement. Would you like to test the fix over on #361 before I merge it in?

nchammas commented 9 months ago

I'll go ahead and merge this PR in, actually, but of course feel free to test the fix in your environment any time. You can easily install Flintrock directly from GitHub.

dorienh commented 4 months ago

Yes, I think the easiest workaround is to just install Java on your Amazon Machine Image; that way flintrock will skip the Java installation step where this is a problem.

I used the following: -- OpenJDK 11.0.19

Forgive me for asking this noob question: how do I do this? Do I specify OpenJDK in the config file somewhere? Can I still use a standard aws linux 2 image?

nchammas commented 4 months ago

@dorienh - Are you still having problems installing Java on the latest version of Flintrock (2.1.0)?

If you'd like you make your own AMI, you'll need to work through this guide. And yes, your starting AMI -- the one from which you will create your own AMIs -- can be the standard Amazon Linux 2 image.