search

apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.77k stars 6.79k forks source link

HDFS supported wheel. #19097

Open franciszzj opened 4 years ago

franciszzj commented 4 years ago

Description

A HDFS support wheel. USE_HDFS=1. Like pip install mxnet=1.6.0-hdfs.

github-actions[bot] commented 4 years ago

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue. Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly. If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

Xiaoxiong-Liu commented 3 years ago

Has this feature been implemented now?

leezu commented 3 years ago

No. But contributions are welcome. You can add code at https://github.com/apache/incubator-mxnet/tree/master/tools/dependencies to create static build of the hdfs library and modify the config files in https://github.com/apache/incubator-mxnet/tree/master/config/distribution to enable hdfs for the wheel build.

szha commented 3 years ago

@leezu I don't see USE_HDFS flag in the CMakeLists.txt. Where will such flag be passed in cmake?

leezu commented 3 years ago

https://github.com/dmlc/dmlc-core/blob/4cabfeafb4f8f8e574de89cea0780c993c20c2d8/CMakeLists.txt#L16