search

ctuning / ck-mlperf

This repository is outdated! Join the open MLPerf workgroup to participate in the development of the next generation of automation workflows for MLPerf benchmarks:
https://bit.ly/mlperf-edu-wg
BSD 3-Clause "New" or "Revised" License
32 stars 23 forks source link

ImageNet preprocessing fails with SciPy 1.3.0 #30

Closed kaul84 closed 4 years ago

kaul84 commented 5 years ago

Hi

Thank you for the latest docker file. The build is failing on my system with the following error at imageNet inference stage

  -----------------------------------
Installing to /home/dvdt/CK_TOOLS/dataset-imagenet-preprocessed
From: /home/dvdt/CK_TOOLS/dataset-imagenet-ilsvrc2012-val-min , To: /home/dvdt/CK_TOOLS/dataset-imagenet-preprocessed , Size: 224 , Crop: 87.5 , InterSize: 0 , 2GU: 0,  2BGR: False, OFF: 0, VOL: '', FOF: image_list
.txt, DTYPE: uint8, EXT: rgb8, IMG: 
Traceback (most recent call last):
  File "/home/dvdt/CK_REPOS/ck-env/package/dataset-imagenet-preprocessed/preprocess_image_dataset.py", line 186, in <module>
    output_filenames = preprocess_files(selected_filenames, source_dir, destination_dir, crop_percentage, square_side, inter_size, guentherization_mode, convert_to_bgr, data_type, new_file_extension)
  File "/home/dvdt/CK_REPOS/ck-env/package/dataset-imagenet-preprocessed/preprocess_image_dataset.py", line 134, in preprocess_files
    convert_to_bgr = convert_to_bgr)
  File "/home/dvdt/CK_REPOS/ck-env/package/dataset-imagenet-preprocessed/preprocess_image_dataset.py", line 87, in load_image
    img = scipy.misc.imread(image_path)
**AttributeError: module 'scipy' has no attribute 'misc'**

Details
CK version: 1.9.7
Python version used by CK: 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0]

docker --version
Docker version 18.09.6, build 481bc77156

OS - Centos7 on GCP
psyhtest commented 5 years ago

Thanks for reporting @gauravkaul! So Docker is not 100% bullet-proof, as I was able to build this image successfully only yesterday.

This error usually happens when Pillow is not installed properly. Can you please comment out this line for now, build the image, run it into bash and check:

$ ck show env --tags=python-package,pillow

We can try to change the order of dependencies to ensure that Pillow gets resolved before SciPy. I will ping you when it's done.

psyhtest commented 5 years ago

I've got a bit more info in the notification email:

*** Dependency 3 = lib-python-numpy (Python NumPy library):
Resolved. CK environment UID = 6407184f44fb392e (version 1.16.3)
*** Dependency 4 = lib-python-scipy (Python SciPy library):
Resolved. CK environment UID = fb965c485b8577e4 (version 1.3.0)
*** Dependency 5 = lib-python-pillow (Python Pillow library):
Resolved. CK environment UID = e4448d6aa2c4bd0a (version 6.0.0)

I don't recall seeing SciPy 1.3.0 previously, only 1.2.1. Maybe it started failing with this new version.

You could try to be more specific about the CentOS version on top of the file e.g. centos7.6 instead of centos7.

psyhtest commented 5 years ago

Indeed, from the CentOS-7 image we built yesterday:

$ sudo docker run -it --rm test_centos_7 bash
[dvdt@987f2869f1dc ~]$ ck show env --tags=python-package,pillow
Env UID:         Target OS: Bits: Name:                 Version: Tags:

f035a4ca09467e35   linux-64    64 Python Pillow library 6.0.0    64bits,PIL,host-os-linux-64,lib,needs-python,needs-python-3.7.3,pillow,python-package,target-os-linux-64,v6,v6.0,v6.0.0,vmaster

[dvdt@987f2869f1dc ~]$ ck show env --tags=python-package,scipy 
Env UID:         Target OS: Bits: Name:                Version: Tags:

6e94283575684243   linux-64    64 Python SciPy library 1.2.1    64bits,host-os-linux-64,lib,needs-python,needs-python-3.7.3,python-package,scipy,target-os-linux-64,v1,v1.2,v1.2.1,vmaster

[dvdt@987f2869f1dc ~]$ cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
psyhtest commented 5 years ago

Or it may be a quirk of GCP somehow serving more recent versions of Python packages. Which version do you get installed with:

$ python3 -m pip install scipy --user
psyhtest commented 5 years ago

Oh, we are so (un)lucky! SciPy 1.3.0 was released on 2019-05-17, that is precisely yesterday! We will investigate how to fix the workflow (or the SciPy version).

kaul84 commented 5 years ago

Hi Anton,

Thanks for the feedback. On my machine (non GCP) Ubuntu the docker container is being built fine. But the same problem due to the SciPy version mismatch is happening on Ubuntu 18.04 LTS, again on GCP. Apologise the previous was for Centos7 indeed as my default instance is Centos. On a new Ubuntu 18.04 GCP instance I am seeing the same issue. I think this is a imagenet issue with SciPy 1.3 as the error suggests.

psyhtest commented 5 years ago

Yes, it's the same error for all OS images now. But I've prepared a workaround - a stable CentOS-7 image (instructions; Dockerfile) that installs Python packages of last-known-good versions, specifically SciPy 1.2.1. Could you please give it a try. If it works for you, we will prepare similar images for the other OSes.

psyhtest commented 5 years ago

You can download a prebuilt Ubuntu 18.04 image with all MLPerf image classification models from the Docker Hub.

ens-lg4 commented 4 years ago

We no longer use scipy for preprocessing, so I'm closing the issue. Please retry.