search

awslabs / aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Other
642 stars 304 forks source link

Fix spark distribution url. #88

Open yoshihitoh opened 3 years ago

yoshihitoh commented 3 years ago

Issue #, if available:

Description of changes: Fix spark distribution url.

Tested with commands below.

before this PR

HTTP/1.1 404 Not Found for both urls.

$ curl -I https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-1.0/spark-2.4.3-bin-hadoop2.8.tgz1
HTTP/1.1 404 Not Found
x-amz-request-id: QV5X09B338ZDSHQT
x-amz-id-2: M+qU+wPMd40bKAv34Q4zfkHJ0AyOu6pIshgw+yiaMJx7HOsSdpyJvj6On2jRVEJofeqbgIwB89I=
Content-Type: application/xml
Date: Mon, 05 Jul 2021 05:48:21 GMT
Server: AmazonS3

$ curl -I https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-2.0/spark-2.4.3-bin-hadoop2.8.tgz1
HTTP/1.1 404 Not Found
x-amz-request-id: GQJ1YTQTT9D1620P
x-amz-id-2: 2mEUf9Kf1IZvev9Ozhd/74nEim2wlglxax3SLePgxX90gmXlj9sa6eUxw7KfnD6flTNfigXmeM4=
Content-Type: application/xml
Date: Mon, 05 Jul 2021 05:49:01 GMT
Server: AmazonS3

after this PR

HTTP/1.1 200 OK for both urls.

$ curl -I https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-1.0/spark-2.4.3-bin-hadoop2.8.tgz
HTTP/1.1 200 OK
x-amz-id-2: gHENl0y5zmSVfj/2jyaYJGapXGDhwuZU7qnMcfuDoiAjGFHYnLAsPAOaODGFkAFtzZHfEN8lpsM=
x-amz-request-id: GYARPZEPJ8VR5TP9
Date: Mon, 05 Jul 2021 05:49:25 GMT
Last-Modified: Tue, 17 Sep 2019 17:47:19 GMT
ETag: "bff157160f1ae778baabb801d113915e-13"
x-amz-version-id: null
Accept-Ranges: bytes
Content-Type: application/x-gzip
Server: AmazonS3
Content-Length: 206432845

$ curl -I https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-2.0/spark-2.4.3-bin-hadoop2.8.tgz1
HTTP/1.1 200 OK
x-amz-id-2: Ji5ocVb9LQyr8s+C7OODGU6Aqk/sbz3Uj2CQIxDRLfDTvjung6pQ/KCQzwM23PSUm7WQ1gThTkE=
x-amz-request-id: JTZHVNMG477RYJEE
Date: Mon, 05 Jul 2021 05:49:48 GMT
Last-Modified: Tue, 08 Jun 2021 23:51:45 GMT
ETag: "3b600756fddc7c2733f777bda010484b-13"
x-amz-version-id: nAoVakG3r_B1JeMD8skkNUJ6MygX_74Q
Accept-Ranges: bytes
Content-Type: application/x-gzip
Server: AmazonS3
Content-Length: 206432845

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

tbugfinder commented 2 years ago

HI @yoshihitoh , could you resolve the conflicts?

yoshihitoh commented 2 years ago

Hi @tbugfinder , thank you for checking this PR.

I resolved the conflicts by rebasing with the master branch, and changing the base branch from glue-0.9 to master. Maybe I made a mistake in choosing the branch when I opened this.

Confirmed those URLs work.

$ curl -I https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-1.0/spark-2.4.3-bin-hadoop2.8.tgz
HTTP/1.1 200 OK
x-amz-id-2: pTB3cZBwBOhU0DlYP0FUruGuZ/Eabm8EorLcjXFkoZpROtTrDQJJxmSmkBsFNv8Q4NXChuJ+BYo=
x-amz-request-id: 0BS6NMDKYXTYVNBZ
Date: Tue, 17 May 2022 06:01:53 GMT
Last-Modified: Tue, 17 Sep 2019 17:47:19 GMT
ETag: "bff157160f1ae778baabb801d113915e-13"
x-amz-version-id: null
Accept-Ranges: bytes
Content-Type: application/x-gzip
Server: AmazonS3
Content-Length: 206432845

$ curl -I https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-2.0/spark-2.4.3-bin-hadoop2.8.tgz
HTTP/1.1 200 OK
x-amz-id-2: pTB3cZBwBOhU0DlYP0FUruGuZ/Eabm8EorLcjXFkoZpROtTrDQJJxmSmkBsFNv8Q4NXChuJ+BYo=
x-amz-request-id: 0BS6NMDKYXTYVNBZ
Date: Tue, 17 May 2022 06:01:53 GMT
Last-Modified: Tue, 17 Sep 2019 17:47:19 GMT
ETag: "bff157160f1ae778baabb801d113915e-13"
x-amz-version-id: null
Accept-Ranges: bytes
Content-Type: application/x-gzip
Server: AmazonS3
Content-Length: 206432845