search

NVIDIA / DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html
Apache License 2.0
5.12k stars 619 forks source link

Conda packages #1274

Open mrocklin opened 5 years ago

mrocklin commented 5 years ago

As DALI gains adoption within the Python community it would be helpful to distribute conda packages. Is this in-scope for this project?

To start thinking about this, it would be interesting to know what dependencies this software has and waht compilation environment it requires.

cc @datametrician @jakirkham

awolant commented 5 years ago

Hi, thanks for the question. Thanks to the contribution from IBM we have a conda recipe ready.

Conda packages are maintained and distributed by IBM. For details look here.

JanuszL commented 5 years ago

IBM conda package is for POWER architecture. Using conda it is still possible to install pip whl and this works in DALI case (even if it is not the most recommended way by the conda community). @mrocklin have you encountered any problems using DALI whl in conda environment?

jakirkham commented 5 years ago

even if it is not the most recommended way by the conda community

To be clear, this is strongly discouraged in the Conda community (not only not recommend). It raises issues with binary compatibility as wheels frequently vendor all library dependencies within the wheel. So two wheels can supply the same dependencies with different versions (like two BLAS libraries) and this will result in symbol collisions that will crash users' programs. The same problem can happen when adding a wheel in a Conda environment.

JanuszL commented 5 years ago

Right, I forgot about binary compatibility problems. Libs itself are not a big deal - DALI hashes the lib names and do other magic to prevent symbol clash for the libraries it ships in its whl. But system libraries could be a problem. We are discussing shipping DALI conda package but we are far from the decision now.

otaj commented 3 years ago

Hi, any update on this one? It's been almost two years at the moment since "discussing shipping"

JanuszL commented 3 years ago

Hi @otaj,

I'm sorry, we don't have any updates we can share now. The DALI conda release is a part of a bigger strategy which details we can't share here.

buchholzmd commented 2 months ago

Has there been any update to this? DALI seems to be a good option for multiprocess data loading for use with JAX and it would be very convenient if shipped as a conda package

EDIT: I see that there is version 1.32.0 available through https://github.com/open-ce/DALI-feedstock

JanuszL commented 2 months ago

Hi @buchholzmd,

Thank you for reaching out. Indeed, DALI is a part of the Open Cognitive Environment (Open-CE), but the effort is not run by NVIDIA. Regarding the original ask, we still don't have any updates we can share, sorry.