search

vemonet / setup-spark

:octocat:✨ Setup Apache Spark in GitHub Action workflows
https://github.com/marketplace/actions/setup-apache-spark
MIT License
20 stars 12 forks source link

Error downloading the Spark binary #6

Closed panicoenlaxbox closed 3 years ago

panicoenlaxbox commented 3 years ago

Describe the bug Error downloading the Spark binary

Which version of the action are you using?

v1

Environment GitHub-hosted

Spark Versions 3.1.1

To Reproduce

    - uses: vemonet/setup-spark@v1
      with:
        spark-version: '3.1.1'
        hadoop-version: '3.2'

Screenshots image

Notes It seems that the requested file is not right now at the specified location.

https://www.apache.org/dyn/closer.lua/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz?as_json returns 

"path_info": "spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz",
"preferred": "https://ftp.cixug.es/apache/"

but I think it's wrong, If I go to this location I cann't find this file :(

image

If I go manually to https://www.apache.org/dyn/closer.lua/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz (no ?tojson query string), a message appears that says "The requested file or directory is not on the mirrors. The object is in our archive : https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz"_

May be a parameter in the GitHub Action to enter manually the url (if user wants to use it) could solve the problem now and for future changes too

Thank you so much

vemonet commented 3 years ago

Hi @panicoenlaxbox thanks for this report

3.1.1 was there a few days (or weeks) ago, before they moved on to 3.1.2

I am now realizing how fast Apache Spark updates the versions available on their mirrors (and they usually discontinue older versions)

I will make the README clearer about this issue (that the users need to check for the latest version before using it)

As you proposed to be able to provide the direct URL to the Spark binary download would be a must-have for anyone relying on a specific version of Spark, I will take a look into adding it when I have the time

vemonet commented 3 years ago

Hi @panicoenlaxbox , I just updated the action, it's now using a better URL so a lot of versions like 3.1.1 and 3.1.2 work out of the box without the need for an URL

But you can also now provide the download URL to the .tgz using the spark-url parameter: https://github.com/vemonet/setup-spark#usage

Thanks a lot for your report!

panicoenlaxbox commented 3 years ago

Hi @vemonet, An excelent job, thank you so much for your support, your GitHub action is awesome :)