[SDKTECHNO-106] Check and repare all DL from Cloudera

[finalspy]

Since Cloudera changed its policies : https://www.cloudera.com/downloads/paywall-expansion.html Downloads from archive.cloudera.com doesnt work anymore.

So this implies to either use internal archive or use apache clients

Needs first to summarize images and binaries impacted. Then find a workaround.

[github-actions[bot]]

:+1: JIRA Issue created : SDKTECHNO-106

[finalspy]

Impacted images :

• saagie/bash:debian-9-stretch and derived (aws/azure/gcp)
• saagie/r-image all versions
• saagie/jupyter-python-nbk and derived (spark/scala)
• saagie/rstudio all versions
• saagie/sqoop

[finalspy]

Cloudera archives concerned :

• https://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.1.tar.gz
• https://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.13.0.tar.gz
• https://archive.cloudera.com/cdh5/cdh/5/hive-1.1.0-cdh5.13.0.tar.gz
• https://archive.cloudera.com/cdh5/cdh/5/hive-1.1.0-cdh5.13.3.tar.gz
• https://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.6-cdh5.13.0.tar.gz

I also noticed references to the following repos :

• deb [arch=amd64, trusted=yes] https://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh5 contrib
• deb-src [trusted=yes] https://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh5 contrib

[finalspy]

3 URLs starting with downloads.cloudera.com are still downloadable at the time I wrote this comment. But all parent directories are unreachable and trying to get them results in a 404 http response or an empty directory listing depending on the final /

• https://downloads.cloudera.com/connectors/ClouderaHive_ODBC_2.6.4.1004/Debian/clouderahiveodbc_2.6.4.1004-2_amd64.deb
• https://downloads.cloudera.com/connectors/impala_odbc_2.5.41.1029/Debian/clouderaimpalaodbc_2.5.41.1029-2_amd64.deb
• https://downloads.cloudera.com/connectors/ClouderaImpala_ODBC_2.6.8.1008/Debian/clouderaimpalaodbc_2.6.8.1008-2_amd64.deb

[finalspy]

Jupyter-minimal image uses cloudera archives to install hdfs-sentry-plugin. This results in the following packages installed from Cloudera :

• avro-libs/now 1.7.6+cdh5.16.2+143-1.cdh5.16.2.p0.23~xenial-cdh5.16.2 all [installed,local]
• bigtop-jsvc/now 0.6.0+cdh5.16.2+937-1.cdh5.16.2.p0.22~xenial-cdh5.16.2 amd64 [installed,local]
• bigtop-tomcat/now 0.7.0+cdh5.16.2+0-1.cdh5.16.2.p0.22~xenial-cdh5.16.2 all [installed,local]
• bigtop-utils/now 0.7.0+cdh5.16.2+0-1.cdh5.16.2.p0.22~xenial-cdh5.16.2 all [installed,local]
• hadoop/now 2.6.0+cdh5.16.2+2863-1.cdh5.16.2.p0.26~xenial-cdh5.16.2 all [installed,local]
• hadoop-0.20-mapreduce/now 2.6.0+cdh5.16.2+2863-1.cdh5.16.2.p0.26~xenial-cdh5.16.2 amd64 [installed,local]
• hadoop-client/now 2.6.0+cdh5.16.2+2863-1.cdh5.16.2.p0.26~xenial-cdh5.16.2 all [installed,local]
• hadoop-hdfs/now 2.6.0+cdh5.16.2+2863-1.cdh5.16.2.p0.26~xenial-cdh5.16.2 all [installed,local]-
• hadoop-mapreduce/now 2.6.0+cdh5.16.2+2863-1.cdh5.16.2.p0.26~xenial-cdh5.16.2 all [installed,local]
• hadoop-yarn/now 2.6.0+cdh5.16.2+2863-1.cdh5.16.2.p0.26~xenial-cdh5.16.2 all [installed,local]
• hive/now 1.1.0+cdh5.16.2+1450-1.cdh5.16.2.p0.28~xenial-cdh5.16.2 all [installed,local]
• hive-jdbc/now 1.1.0+cdh5.16.2+1450-1.cdh5.16.2.p0.28~xenial-cdh5.16.2 all [installed,local]
• parquet/now 1.5.0+cdh5.16.2+200-1.cdh5.16.2.p0.24~xenial-cdh5.16.2 all [installed,local]
• parquet-format/now 2.1.0+cdh5.16.2+22-1.cdh5.16.2.p0.22~xenial-cdh5.16.2 all [installed,local]
• sentry/now 1.5.1+cdh5.16.2+573-1.cdh5.16.2.p0.23~xenial-cdh5.16.2 all [installed,local]
• sentry-hdfs-plugin/now 1.5.1+cdh5.16.2+573-1.cdh5.16.2.p0.23~xenial-cdh5.16.2 all [installed,local]
• solr/now 4.10.3+cdh5.16.2+537-1.cdh5.16.2.p0.26~xenial-cdh5.16.2 all [installed,local]

[finalspy]

Here's a summary of jar libs differences I found between cloudera and Apache Hive ans Sqoop tat.gz distributions https://docs.google.com/spreadsheets/d/1oZToezEXyACnVOWB0F7qr2bG5h6YqcaecF9kqkwZPJU/edit?usp=sharing