search

kgyrtkirk / hive-dev-box

21 stars 14 forks source link

Cant find a valid url: apache-tez ... #9

Open javiroman opened 4 years ago

javiroman commented 4 years ago

Hi, any help with this error?

 * installing some basic stuff...
[main] INFO hu.rxd.toolbox.switcher.GenericComponent - downloading: http://xenia.sote.hu/ftp/mirrors/www.apache.org/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
[main] INFO hu.rxd.toolbox.qtest.diff.CachedURL - downloading: http://xenia.sote.hu/ftp/mirrors/www.apache.org/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
[main] INFO hu.rxd.toolbox.switcher.GenericComponent - failed to download: http://xenia.sote.hu/ftp/mirrors/www.apache.org/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
[main] INFO hu.rxd.toolbox.switcher.GenericComponent - downloading: https://archive.apache.org/dist/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
[main] INFO hu.rxd.toolbox.qtest.diff.CachedURL - downloading: https://archive.apache.org/dist/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
[main] INFO hu.rxd.toolbox.switcher.GenericComponent - failed to download: https://archive.apache.org/dist/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
[main] INFO hu.rxd.toolbox.switcher.GenericComponent - downloading: https://rxd.hu/misc/preview/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
[main] INFO hu.rxd.toolbox.qtest.diff.CachedURL - downloading: https://rxd.hu/misc/preview/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
[main] INFO hu.rxd.toolbox.switcher.GenericComponent - failed to download: https://rxd.hu/misc/preview/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
Exception in thread "main" java.io.IOException: Cant find a valid url; tried: [http://xenia.sote.hu/ftp/mirrors/www.apache.org/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz, https://archive.apache.org/dist/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz, https://rxd.hu/misc/preview/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz]
        at hu.rxd.toolbox.switcher.GenericComponent.downloadArtifact(GenericComponent.java:114)
        at hu.rxd.toolbox.switcher.TezComponent.provideComponent(TezComponent.java:51)
        at hu.rxd.toolbox.switcher.GenericComponent.ensurePresence(GenericComponent.java:57)
        at hu.rxd.toolbox.switcher.GenericComponent.switchTo(GenericComponent.java:31)
        at hu.rxd.toolbox.HiveDevBoxSwitcher.main(HiveDevBoxSwitcher.java:12)
Error on line 74

It looks the url downloader is not working fine (the archive https://archive.apache.org/dist/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz is accesible with wget, however is raising this error).

kgyrtkirk commented 4 years ago

I think the xenia mirror is functioning as well... I would guess that it's some network issue with your docker installation have you tried downloading something from the container?

javiroman commented 4 years ago

Yes, the networking is working within the container:

[javiroman@gopher hive-dev-box]$ docker run -ti --entrypoint=/bin/sh hive-dev-box
$ ls
$ wget https://archive.apache.org/dist/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
--2020-08-04 06:42:00--  https://archive.apache.org/dist/tez/0.9.1/apache-tez-0.9.1-bin.tar.gz
Resolving archive.apache.org (archive.apache.org)... 138.201.131.134, 2a01:4f8:172:2ec5::2
Connecting to archive.apache.org (archive.apache.org)|138.201.131.134|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 61074757 (58M) [application/x-gzip]
Saving to: ‘apache-tez-0.9.1-bin.tar.gz’

apache-tez-0.9.1-bin.tar.gz       100%[=============================================================>]  58.25M  45.4MB/s    in 1.3s    

2020-08-04 06:42:02 (45.4 MB/s) - ‘apache-tez-0.9.1-bin.tar.gz’ saved [61074757/61074757]

Right now doing some tests to find out what is happening.

kgyrtkirk commented 4 years ago

this is highly unfortunate... it seems like the exception is discarded :facepalm:

could you check that /work/download is writeable?

javiroman commented 4 years ago

mmm, yes probably there is a problem with permissions.

dev@21932771a6a8:/work/downloads$ touch test
touch: cannot touch 'test': Permission denied
dev@21932771a6a8:/work/downloads$ pwd
/work/downloads
dev@21932771a6a8:/work/downloads$
javiroman commented 4 years ago

Apparently, the problem is hive-dev-box is only ready for running as root user.

sudo ./hdb run hive-test runs as expected.

kgyrtkirk commented 4 years ago

oh...ok; I think the root directory of the work volume is incorrect.

hive-dev-box uses the work "volume" to cache stuff / and share between the containers... it should be owned by the dev user...I don't know how that went south...I've tried to run hdb with sudo - but it still run as the dev user (as it should be)

anyway - I think you could probably use chown -R dev /work to fix the permissions