search

microsoft / planetary-computer-containers

Container definitions for the Planetary Computer
MIT License
52 stars 12 forks source link

update tensorflow #34

Closed ngam closed 2 years ago

ngam commented 2 years ago

Is there a specific reason why TensorFlow is being pulled from main as opposed to conda-forge? We have TensorFlow 2.7.0 functioning in conda-forge now, so if you don't object, let's update it.

https://github.com/microsoft/planetary-computer-containers/blob/b5e229fb86dc98e8e5a3337945f63f40d3b88778/gpu-tensorflow/environment.yml#L108

Also, in the future, it may be worthwhile to consider pulling in NVIDIA's containers as base instead of pangeo --- nothing against pangeo (I am a big fan!) but for GPU-related activities, I think it's safer to rely on NVIDIA or the package providers (Google/TensorFlow or PyTorch) as they do more rigorous testing.

The issue is that, oftentimes, the set of volunteers at conda-forge (pangeo builds on conda-forge; I am a contributor at conda-forge) cannot keep up with the demanding builds of PyTorch and TensorFlow (currently, PyTorch is good, but we do not have the full ecosystem; for TensorFlow, we have 2.7.0, but we are sort of stuck --- we also don't have the full ecosystem). Any little tweak (especially with TensorFlow, e.g. trying 2.8.0 through pip) will break everything as one would need to stitch together the cuda-related packages. Pulling in a GPU-ready container through NVIDIA or Google will likely be safer, but will almost definitely be larger in terms of storage.

Thanks for the good work!!

ngam commented 2 years ago

(btw, if you decide to pull in tf 2.7 from conda-forge, we will need to do some additional work to ensure it pulls in the gpu package --- I can help set that up in a PR, but first, let me know if you're open to the idea)

ngam commented 2 years ago

Closing as there seems to be no interest in this...