Compatibility issue between Hadoop and Hive (Hive not working)

InseeFrLab / images-datascience

Collection of Docker images to build the data science catalog of the Onyxia project

MIT License

24 stars 23 forks source link

Compatibility issue between Hadoop and Hive (Hive not working) #117

Open nashugame opened 1 year ago

nashugame commented 1 year ago

I was trying to configure Hadoop and Hive. Hadoop is working fine. But for hive to work with current version of Hadoop, it should be >= 3.1.3. Screenshot from 2023-05-31 15-36-09

Current version of hive in Onyxia setup: 2.3.9 Also not able to configure hive-site.xml (Edit/delete/change permission) even with root privilege. Screenshot from 2023-05-31 15-36-22

olevitt commented 1 year ago

Hi !

Thanks for your feedback ! I transfered the issue to this project as it's not related to Onyxia itself but rather to the images used by it.
Maybe @fcomte @avouacr have any thoughts on this issue ?

fcomte commented 1 year ago

Hello !

Thx for the report We should :

[ ] build spark with hive.version=3.1.3 in stead of 2.3.9
[ ] separate our spark image into two distincts docker images : one for the hive-metastore server and one for the spark environnement

nashugame commented 1 year ago

Thnx @fcomte @olevitt for all the help. I also request you to make hive-site.xml file in /hive/conf editable, so that we can customise the configurations according to our requirements, currently it's read-only and can't be modified even with root privileges.

fcomte commented 1 year ago

yes this part will be done in our helm-charts : Can you write an issue to track evolution there ?

https://github.com/InseeFrLab/helm-charts-interactive-services/issues

nashugame commented 1 year ago

Sure