intel-analytics / analytics-zoo

Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
https://analytics-zoo.readthedocs.io/
Apache License 2.0
17 stars 4 forks source link

Add pip setup extra_require for tensorflow (1.x, 2.x) and pytorch #426

Open yangw1234 opened 3 years ago

leonardozcm commented 3 years ago

Collect extra pakages used in examples:

# pytorch
matplotlib
ray=0.8.6
Pillow=8.1.0
torch 
torchvision
jep==3.9.0
h5py==2.9.0
cloudpickle==1.6.0

# tensorflow 1.15
h5py==2.9.0
tensorlfow==1.15
Pillow=8.1.0
pypandoc==1.5
pandas
scikit-learn
tensorflow_datasets==3.2.0
psutil
tensorflow-gan

# tensorflow 2
tensorlfow==2.3.0
conda install -y cmake==3.16.0 -c conda-forge

Any suggestions?

yangw1234 commented 3 years ago

remove the conda install, this is only for horovod

yangw1234 commented 3 years ago

I think for tensorflow 1.15 only tensorflow==1.15.0 and tensorflow_datasets==3.2.0 is enough.

yangw1234 commented 3 years ago

@hkvision @qiuxin2012 any suggestions on the PyTorch dependencies?

hkvision commented 3 years ago

ray and jep shouldn't be needed at the same time; also we may not need pillow for general pytorch applications?

hkvision commented 3 years ago

why is matplotlib needed? h5py and cloudpickle is only needed for spark backend right?

leonardozcm commented 3 years ago

why is matplotlib needed? h5py and cloudpickle is only needed for spark backend right?

Pillow and matplot are needed in torch examples. And Pillow are also imported in orca.image.file

hkvision commented 3 years ago

why is matplotlib needed? h5py and cloudpickle is only needed for spark backend right?

Pillow and matplot are needed in torch examples. And Pillow are also imported in orca.image.file

But they would only be needed for image applications; other applications don't need them. Shouldn't add them as a must.

leonardozcm commented 3 years ago

why is matplotlib needed? h5py and cloudpickle is only needed for spark backend right?

Pillow and matplot are needed in torch examples. And Pillow are also imported in orca.image.file

But they would only be needed for image applications; other applications don't need them. Shouldn't add them as a must.

ok, I'll not mark them as necessaray

yangw1234 commented 3 years ago

ray and jep shouldn't be needed at the same time; also we may not need pillow for general pytorch applications?

any idea to differentiate ray and spark backend?