search

gbif / stackable

GBIF Stackable Infrastructure
Apache License 2.0
4 stars 0 forks source link

WebHDFS impl #17

Closed zaultooz closed 6 months ago

zaultooz commented 1 year ago

To enable users to download their newly constructed archives through the GBIF api, we need to expose the hdfs over HTTP and create rewrite rules for dev2.

We would need to create an image based on the setup guide on: https://hadoop.apache.org/docs/stable/hadoop-hdfs-httpfs/ServerSetup.html and create a service that exposes the server on one of the nodes.

I would suggest label it so it follows the namenodes. The core-site.xml and hdfs-site.xml should be mounted into the container through the normal cm volume setup.

Stackable already have something that looks like webhdfs in their testing-image so it could maybe be used.

zaultooz commented 6 months ago

Should be done through another issue, https://github.com/gbif/stackable/tree/master/charts/gbif-hadoop-httpfs WebHDFS charts.