pndaproject / platform-data-mgmnt

PNDA components that manage aspects of datasets
Other
3 stars 20 forks source link

PNDA-3427: Configure data-service with the webhdfs services. #22

Closed stephanesan closed 6 years ago

stephanesan commented 6 years ago

In HDP the following setting is exposed by the rest API: fs.defaultFS = hdfs://HDFS-HA and so can not be used to determine where the httpfs proxies are running. In HDP the httpfs proxy is currently configured to run on all 4 managers through salt but there is no information in the Hadoop REST API that exposes these service locations. So currently, the data-service will be configured with the WebHDFS services when running in HDP mode.