search

daphne-eu / daphne

DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines
Apache License 2.0
65 stars 58 forks source link

Containers with TensorFlow and PyTorch #705

Open pdamme opened 5 months ago

pdamme commented 5 months ago

Since today, DaphneLib supports efficient data exchange with TensorFlow and PyTorch. Currently, they are optional dependencies, i.e., if they are installed, the related DaphneLib features can be used, if they are not installed, DaphneLib still works, but the TF/PT-related features don't. Moreover, the test cases for the data exchange with TF/PT are only run when these libraries are installed.

Hence, users and developers are not forced to install TF/PT unless they really want to interoperate with them or run the respective test cases. That also means, there is no urgent problem.

Currently, the DAPHNE container images do not contain TF/PT. To provide users the opportunity to play around with the data exchange with TF/PT and to allow developers (or the CI) to run all test cases (including the ones that require TF/PT), we could add TF/PT to our containers by:

pip install tensorflow torch
pip install --upgrade pandas

Note that the version of pandas currently available in the containers (installed via apt) is too old.

However, TF/PT are quite large libraries. I built a container including them on my system and it has a size of ~14 GB, while the current daphne-dev container image is just ~4 GB. We should not put the burden of downloading such a large container image on every user/developer. Thus, we could discuss the introduction of a separate container image that contains TF/PT (maybe similar to how it's already done for CUDA). For the CI, it would be great to run an image that has TF/PT, such that all test cases execute.

corepointer commented 4 months ago

I'll see that I get these python packages integrated in the github-actions container and that we have another flavor besides daphne and daphne-dev.

corepointer commented 1 week ago

Update on this one: Can't integrate these packages into python as they pull in CUDA and therefore increase the container size to 10+ GB. I suggest to either have another flavor of DAPHNE container (besides daphne, daphne-dev and github-actions) or to integrate these packages into the CUDA version of daphne-dev, as this one's also quite large already as it contains the CUDA SDK and CUDNN.