search

halcyon / asdf-java

A Java plugin for asdf-vm.
MIT License
455 stars 86 forks source link

Automatic update of release data #87

Closed joschi closed 4 years ago

joschi commented 4 years ago

Instead of fetching the release data of each OpenJDK distribution separately and parsing the individual formats or maintaining a list of releases manually (Amazon Corretto, Zulu Community), this PR introduces the use of a central release "database".

The release data in the ./data directory is updated automatically by a scheduled GitHub Actions workflow and the clients only have to fetch the respective text file from GitHub (in TSV format).

This has the following advantages:

Disclaimer: The source of the Java release metadata is a project of mine. https://joschi.github.io/java-metadata/index.html

joschi commented 4 years ago

The scheduled workflow fails here because the java-metadata branch only exists in my fork of this repository.

See also: https://github.com/joschi/asdf-java/runs/622011631

donbeave commented 4 years ago

That's really good job!

joschi commented 4 years ago

@halcyon Should we/How should we move on with this?

halcyon commented 4 years ago

@joschi this is an interesting and promising idea. I really like the reduction of complexity, reduction of required dependencies, and concentration of java metadata requests to a single source. I do have some hesitation around increasing our dependence on github (not git, as one can host a git repository anywhere!). I'll be in touch soon to continue the discussion. Thank you for being patient with me as I learn more and consider. As always, I appreciate all your ideas and contributions!

donbeave commented 4 years ago

@halcyon actually not need to rely on Github, it can be done by any CI (GitLab is good also, or Travis CI is a good alternative too).

halcyon commented 4 years ago

@halcyon actually not need to rely on Github, it can be done by any CI (GitLab is good also, or Travis CI is a good alternative too).

@donbeave how much work would be required to rewrite Github Actions to work on one of these other platforms? Let's include moving java-metadata in that estimate too, as anything that would cause asdf-java to need to be hosted elsewhere could impact it too.

The java-metadata project is very impressive. It pulls down metadata for all architectures across an expanded number of implementations. The number of requests it makes are higher than an individual asdf-java client, as we currently pull down metadata for the architecture the client is on. While I understand that compared to many clients making independent requests, java-metadata would result in an overall reduction of requests, it could still be considerably high for the java-metadata instance.

donbeave commented 4 years ago

I think these questions better address to @joschi. I didn't check much java-metadata project, but this PR commits looks quite simple, nothing special I can see here what can not run on GitLab or Travis CI. Only one special feature is required from CI is the ability to run cron jobs, both have this option: 1) https://docs.gitlab.com/ee/ci/pipelines/schedules.html 2) https://docs.travis-ci.com/user/cron-jobs/

joschi commented 4 years ago

Thanks for getting the discussion going!

how much work would be required to rewrite Github Actions to work on one of these other platforms? Let's include moving java-metadata in that estimate too, as anything that would cause asdf-java to need to be hosted elsewhere could impact it too.

Both, java-metadata and the update_data.bash script in this PR, are bash scripts which could run on pretty much every operating system with bash and some GNU tools. I ran both of them on macOS during development and they are running on Ubuntu 18.04 in the GitHub Actions workflow.

Are there limits, request-rate or otherwise that java-metadata might hit within Github Actions?

It's currently running in a GitHub Free account with 2,000 minutes of execution time per month (https://github.com/pricing).

If there are limits, how much headroom does java-metadata have?

The update is triggered every 6 hours and I haven't seen any problems since starting the project in the beginning of March 2020.

The requests to the collected data are served through GitHub Pages with its respective limits (https://help.github.com/en/github/working-with-github-pages/about-github-pages#guidelines-for-using-github-pages), e. g. a soft bandwidth limit of 100GB per month.

In the unlikely case that this would be exceeded, it would still be possible to put a CDN in front of these files.

With regards to asdf-java, it might be useful to put the generated TSV files into a separate branch and serve them through GitHub Pages.

If asdf-java were to depend upon java-metadata, would it be a logical next step to consider consolidation?

I don't think it's necessary or a logical next step since the goals of asdf-java and java-metadata are quite different.

Of course you're free to fork java-metadata and run it yourself.

joschi commented 4 years ago

@halcyon Do you have other questions about or around this PR?

It would also resolve the following issues and PRs:

45/#47 (as well as other machine architectures: list)

57

61/#67

90

93

Current output on macOS ``` # asdf list-all java adoptopenjdk-8.0.181+13 adoptopenjdk-8.0.192+12 adoptopenjdk-8.0.202+8 adoptopenjdk-8.0.202+8.openj9-0.12.0 adoptopenjdk-8.0.202+8.openj9-0.12.1 adoptopenjdk-8.0.212+3 adoptopenjdk-8.0.212+3.openj9-0.14.0 adoptopenjdk-8.0.212+4 adoptopenjdk-8.0.212+4.openj9-0.14.2 adoptopenjdk-8.0.222+10.1 adoptopenjdk-8.0.222+10.1.openj9-0.15.1 adoptopenjdk-8.0.232+9.1 adoptopenjdk-8.0.232+9.1.openj9-0.17.0 adoptopenjdk-8.0.242+8.1 adoptopenjdk-8.0.242+8.1.openj9-0.18.1 adoptopenjdk-8.0.252+9.1 adoptopenjdk-8.0.252+9.1.openj9-0.20.0 adoptopenjdk-9.0.0+181 adoptopenjdk-10.0.2+13.1 adoptopenjdk-11.0.0+28 adoptopenjdk-11.0.1+13 adoptopenjdk-11.0.1+13.openj9-0.11.0 adoptopenjdk-11.0.2+7 adoptopenjdk-11.0.2+9 adoptopenjdk-11.0.2+9.openj9-0.12.0 adoptopenjdk-11.0.2+9.openj9-0.12.1 adoptopenjdk-11.0.3+7 adoptopenjdk-11.0.3+7.openj9-0.14.0 adoptopenjdk-11.0.3+7.openj9-0.14.3 adoptopenjdk-11.0.4+11.1 adoptopenjdk-11.0.4+11.1.openj9-0.15.1 adoptopenjdk-11.0.5+10.1 adoptopenjdk-11.0.5+10.1.openj9-0.17.0 adoptopenjdk-11.0.6+10.1 adoptopenjdk-11.0.6+10.1.openj9-0.18.0 adoptopenjdk-11.0.6+10.1.openj9-0.18.1 adoptopenjdk-11.0.7+10.1 adoptopenjdk-11.0.7+10.1.openj9-0.20.0 adoptopenjdk-12.0.0+33 adoptopenjdk-12.0.0+33.openj9-0.13.0 adoptopenjdk-12.0.1+12 adoptopenjdk-12.0.1+12.openj9-0.14.1 adoptopenjdk-12.0.2+10.1 adoptopenjdk-12.0.2+10.1.openj9-0.15.1 adoptopenjdk-12.0.2+10.3.openj9_0.15.1 adoptopenjdk-13.0.0+33.1 adoptopenjdk-13.0.0+33.1.openj9-0.16.0 adoptopenjdk-13.0.1+9.1 adoptopenjdk-13.0.1+9.1.openj9-0.17.0 adoptopenjdk-13.0.2+8.1 adoptopenjdk-13.0.2+8.1.openj9-0.18.0 adoptopenjdk-14.0.0+36.1 adoptopenjdk-14.0.0+36.1.openj9-0.19.0 adoptopenjdk-14.0.1+7.1 adoptopenjdk-14.0.1+7.1.openj9-0.20.0 adoptopenjdk-large_heap-8.0.202+8.openj9-0.12.1 adoptopenjdk-large_heap-8.0.212+3.openj9-0.14.0 adoptopenjdk-large_heap-8.0.212+4.openj9-0.14.2 adoptopenjdk-large_heap-8.0.222+10.1.openj9-0.15.1 adoptopenjdk-large_heap-8.0.232+9.1.openj9-0.17.0 adoptopenjdk-large_heap-8.0.242+8.1.openj9-0.18.1 adoptopenjdk-large_heap-8.0.252+9.1.openj9-0.20.0 adoptopenjdk-large_heap-11.0.2+9.openj9-0.12.0 adoptopenjdk-large_heap-11.0.2+9.openj9-0.12.1 adoptopenjdk-large_heap-11.0.3+7.openj9-0.14.0 adoptopenjdk-large_heap-11.0.3+7.openj9-0.14.3 adoptopenjdk-large_heap-11.0.4+11.1.openj9-0.15.1 adoptopenjdk-large_heap-11.0.5+10.1.openj9-0.17.0 adoptopenjdk-large_heap-11.0.6+10.1.openj9-0.18.0 adoptopenjdk-large_heap-11.0.6+10.1.openj9-0.18.1 adoptopenjdk-large_heap-11.0.7+10.1.openj9-0.20.0 adoptopenjdk-large_heap-12.0.0+33.openj9-0.13.0 adoptopenjdk-large_heap-12.0.1+12.openj9-0.14.1 adoptopenjdk-large_heap-12.0.2+10.1.openj9-0.15.1 adoptopenjdk-large_heap-13.0.0+33.1.openj9-0.16.0 adoptopenjdk-large_heap-13.0.1+9.1.openj9-0.17.0 adoptopenjdk-large_heap-13.0.2+8.1.openj9-0.18.0 adoptopenjdk-large_heap-14.0.0+36.1.openj9-0.19.0 adoptopenjdk-large_heap-14.0.1+7.1.openj9-0.20.0 graalvm-19.0.0 graalvm-19.0.2 graalvm-19.1.0 graalvm-19.1.1 graalvm-19.2.0 graalvm-19.2.0.1 graalvm-19.2.1 graalvm-19.3.0+java8 graalvm-19.3.0+java11 graalvm-19.3.0.2+java8 graalvm-19.3.0.2+java11 graalvm-19.3.1+java8 graalvm-19.3.1+java11 graalvm-19.3.2+java8 graalvm-19.3.2+java11 graalvm-20.0.0+java8 graalvm-20.0.0+java11 graalvm-20.1.0+java8 graalvm-20.1.0+java11 liberica-1.8.0 liberica-8u202 liberica-8u212 liberica-8u222 liberica-8u232 liberica-8u232+10 liberica-8u242+7 liberica-8u252+9 liberica-11.0.1 liberica-11.0.2 liberica-11.0.3 liberica-11.0.4 liberica-11.0.5 liberica-11.0.5+11 liberica-11.0.6+10 liberica-11.0.7+10 liberica-12 liberica-12.0.1 liberica-12.0.2 liberica-13 liberica-13.0.1 liberica-13.0.1+10 liberica-13.0.2+9 liberica-14+36 liberica-14.0.1+8 liberica-javafx-8u242+7 liberica-javafx-8u252+9 liberica-javafx-11.0.6+10 liberica-javafx-11.0.7+10 liberica-javafx-13.0.2+9 liberica-javafx-14+36 liberica-javafx-14.0.1+8 liberica-lite-11.0.1 liberica-lite-11.0.2 liberica-lite-11.0.3 liberica-lite-11.0.4 liberica-lite-11.0.5 liberica-lite-11.0.5+11 liberica-lite-11.0.6+10 liberica-lite-11.0.7+10 liberica-lite-12 liberica-lite-12.0.1 liberica-lite-12.0.2 liberica-lite-13 liberica-lite-13.0.1 liberica-lite-13.0.1+10 liberica-lite-13.0.2+9 liberica-lite-14+36 liberica-lite-14.0.1+8 openjdk-9.0.4 openjdk-10 openjdk-10.0.1 openjdk-10.0.2 openjdk-11 openjdk-11.0.1 openjdk-11.0.2 openjdk-12 openjdk-12.0.1 openjdk-12.0.2 openjdk-13 openjdk-13.0.1 openjdk-13.0.2 openjdk-14 openjdk-14.0.1 sapmachine-11.0.6 sapmachine-11.0.6.0.1 sapmachine-11.0.7 sapmachine-13.0.2 sapmachine-14 sapmachine-14.0.1 zulu-7.13.0.1 zulu-7.14.0.5 zulu-7.15.0.1 zulu-7.16.0.1 zulu-7.17.0.5 zulu-7.18.0.3 zulu-7.20.0.3 zulu-7.21.0.3 zulu-7.22.0.3 zulu-7.23.0.1 zulu-7.24.0.1 zulu-7.25.0.5 zulu-7.27.0.1 zulu-7.29.0.5 zulu-7.31.0.5 zulu-7.34.0.5 zulu-7.36.0.5 zulu-7.38.0.11 zulu-8.12.0.1 zulu-8.13.0.5 zulu-8.14.0.1 zulu-8.15.0.1 zulu-8.17.0.3 zulu-8.19.0.1 zulu-8.20.0.5 zulu-8.21.0.1 zulu-8.23.0.3 zulu-8.25.0.1 zulu-8.27.0.7 zulu-8.28.0.1 zulu-8.30.0.1 zulu-8.31.0.1 zulu-8.33.0.1 zulu-8.34.0.1 zulu-8.36.0.1 zulu-8.38.0.13 zulu-8.40.0.25 zulu-8.42.0.21 zulu-8.42.0.23 zulu-8.44.0.11 zulu-8.46.0.19 zulu-9.0.0.15 zulu-9.0.1.3 zulu-9.0.4.1 zulu-9.0.7.1 zulu-10.1+11 zulu-10.2+3 zulu-10.3+5 zulu-11.2.3 zulu-11.29.3 zulu-11.29.11 zulu-11.31.11 zulu-11.33.15 zulu-11.35.13 zulu-11.35.15 zulu-11.37.17 zulu-11.39.15 zulu-12.1.3 zulu-12.2.3 zulu-12.3.11 zulu-13.27.9 zulu-13.28.11 zulu-13.29.9 zulu-13.31.11 zulu-14.27.1 zulu-14.28.21 zulu-javafx-8.33.0.1 zulu-javafx-8.36.0.1 zulu-javafx-8.38.0.13 zulu-javafx-8.40.0.25 zulu-javafx-8.42.0.23 zulu-javafx-8.44.0.13 zulu-javafx-8.46.0.19 zulu-javafx-11.2.5 zulu-javafx-11.29.3 zulu-javafx-11.31.11 zulu-javafx-11.33.15 zulu-javafx-11.35.15 zulu-javafx-11.37.17 zulu-javafx-11.37.19 zulu-javafx-11.39.15 zulu-javafx-13.29.11 zulu-javafx-13.31.11 ```
halcyon commented 4 years ago

how much work would be required to rewrite Github Actions to work on one of these other platforms? Let's include moving java-metadata in that estimate too, as anything that would cause asdf-java to need to be hosted elsewhere could impact it too.

Both, java-metadata and the update_data.bash script in this PR, are bash scripts which could run on pretty much every operating system with bash and some GNU tools. I ran both of them on macOS during development and they are running on Ubuntu 18.04 in the GitHub Actions workflow.

My question has nothing to do with changing operating systems. I asked about the impact this would have on our ability to change hosting. I'm concerned about the tighter coupling and increased vendor lock-in with GitHub.

Are there limits, request-rate or otherwise that java-metadata might hit within Github Actions?

It's currently running in a GitHub Free account with 2,000 minutes of execution time per month (https://github.com/pricing).

It's good to know that GitHub Actions currently has generous head-room for the number of minutes of execution. I'm unconcerned with the nominal traffic used by asdf-java clients querying GitHub Pages. However, I am concerned about the far larger, and ever increasing ingress traffic used within GitHub Actions by java-metadata.

joschi commented 4 years ago

My question has nothing to do with changing operating systems.

I understood your question being about runtime environments.

And my response was: Both scripts (update_data.bash in this project and java-metadata as a whole) will run in any *nix-like environment which provides Bash 5.x and some additional programs (sed, perl, curl, jq, jo).

The GitHub Actions workflows are executing those scripts, not more.

halcyon commented 4 years ago

@halcyon Do you have other questions about or around this PR?

It would also resolve the following issues and PRs:

45/#47 (as well as other machine architectures: list)

57

61/#67

90

93

Current output on macOS

Despite my reservations around vendor lock-in, I think this is an innovative approach and has a lot of promise. I appreciate this work, and apologize for not getting back with you sooner. I've done a lot of thinking, and for the next steps I'd like to see asdf-java do the data collection directly that java-metadata is doing. This would enable asdf-java to continue to be a stand-alone project, which I think is important to facilitating future support requests and keeping complexity to a minimum.

alex-popov-tech commented 4 years ago

hey guys @joschi @halcyon sorry for interrupting, any updates on that? looks really promising! i will switch to that feature branch for now

joschi commented 4 years ago

@alex-popov-tech I've been tracking the PR in the master branch of https://github.com/joschi/asdf-java/ in which the release data is being updated regularly, if you want to give it a test drive.

aheritier commented 4 years ago

@joschi it's better with your master

I can see and install the version adoptopenjdk-11.0.7+10.1

The name changed from adopt-openjdk-11.0.7+10 which is a bit disturbing because asdf is failing to install adopt-openjdk-11.0.7+10 with the error

curl: Remote file name has no length!
curl: try 'curl --help' or 'curl --manual' for more information

(It might be great to handle it more gracefully if possible).

Otherwise it's good

joschi commented 4 years ago

@aheritier Good suggestion, thanks!

I've updated the branch to print an error message and exit the installation script if the release wasn't found:

# asdf install java zulu-14.99
Unknown release: zulu-14.99
aheritier commented 4 years ago

perfect @joschi Thanks a lot

halcyon commented 4 years ago

@joschi is this ready to be merged?

alex-popov-tech commented 4 years ago

i just verified that my case works on that branch

joschi commented 4 years ago

@joschi is this ready to be merged?

@halcyon With the latest commit (which makes this PR fail because the upstream TSV files aren't there yet) it's good to merge, in my opinion.