TensorFlow dependencies

[psyhtest]

I installed TensorFlow 1.10.1 on a clean Ubuntu 18.04 system:

$ ck install package:lib-tensorflow-1.10.1-cpu

On the first run of program:image-classification-tf-py, however, SciPy got detected and corrupted the same environment entry:

$ ck show env --tags=tensorflow
Env UID:         Target OS: Bits: Name:                                Version: Tags:

5eaeb467192af25c   linux-64    64 SciPy Python library (prebuilt, cpu) 1.10.1   64bits,channel-stable,host-os-linux-64,lib,needs-python,needs-python-3.6.5,python-package,scipy,target-os-linux-64,tensorflow,tensorflow-cpu,v1,v1.10,v1.10.1,vcpu,vprebuilt

(see the tags containing vcpu,vprebuilt on the one hand and scipy on the other.)

What's weird is that even after removing this env entry:

$ ck rm env:5eaeb467192af25c
Are you sure to delete CK entry "SciPy Python library (prebuilt, cpu)"
    env:5eaeb467192af25c (9b9b3208ac44b891:5eaeb467192af25c) ? (y/N): y
   Entry "SciPy Python library (prebuilt, cpu)"
    env:5eaeb467192af25c (9b9b3208ac44b891:5eaeb467192af25c) was successfully deleted!

CK still detects it and assigns the same env entry:

$ ck detect soft:lib.python.scipy
...
  Detecting and sorting versions (ignore some work output) ...

    * /home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib/scipy/__init__.py   (Version 1.10.1)

  Found pre-recorded CK installation info ...

  Registering in the CK (/home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib/scipy/__init__.py) ...

  Software entry found: lib.python.scipy (4460bdb0ade2a3df)

Moreover, detecting NumPy overwrites it:

$ ck detect soft:lib.python.numpy
...
  Detecting and sorting versions (ignore some work output) ...

    * /home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib/numpy/__init__.py   (Version 1.10.1)

  Found pre-recorded CK installation info ...

  Registering in the CK (/home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib/numpy/__init__.py) ...

  Software entry found: lib.python.numpy (6a10047b1bcd16fc)

Environment entry updated (5eaeb467192af25c)!
  Successfully registered with UID: 5eaeb467192af25c

It seems that for this program I need to have all the three dependencies resolved: TensorFlow, NumPy and SciPy, but can only have one at a time!

This probably happens because of ck-install.json containing:

  "env_data_uoa": "5eaeb467192af25c",

and all the three dependencies are being located under lib/.

[psyhtest]

Removing the above line for env_data_uoa allowed me to register three separate environments:

$ ck show env --tags=tensorflow
Env UID:         Target OS: Bits: Name:                                Version: Tags:

2b13628df1484c9a   linux-64    64 TensorFlow library (prebuilt, cpu)   1.10.1   64bits,channel-stable,host-os-linux-64,lib,needs-python,needs-python-3.6.5,target-os-linux-64,tensorflow,tensorflow-cpu,v1,v1.10,v1.10.1,vcpu,vprebuilt
0da8e1d4cf1ac35e   linux-64    64 SciPy Python library (prebuilt, cpu) 1.10.1   64bits,channel-stable,host-os-linux-64,lib,needs-python,needs-python-3.6.5,python-package,scipy,target-os-linux-64,tensorflow,tensorflow-cpu,v1,v1.10,v1.10.1,vcpu,vprebuilt
f145fbbd4742bbae   linux-64    64 NumPy Python library (prebuilt, cpu) 1.10.1   64bits,channel-stable,host-os-linux-64,lib,needs-python,needs-python-3.6.5,numpy,python-package,target-os-linux-64,tensorflow,tensorflow-cpu,v1,v1.10,v1.10.1,vcpu,vprebuilt

I believe, however, I had to detect them in the following order: NumPy, TensorFlow, SciPy.

I don't think that NumPy and SciPy should be discoverable by the tensorflow tag. Also, the names (containing (prebuilt, cpu)) and the versions (all 1.10.1) are incorrect too.

[psyhtest]

Still, running the program failed:

$ ck run program:image-classification-tf-py
...
*** Dependency 7 = lib-python-numpy (NumPy python library):

    Resolved. CK environment UID = f145fbbd4742bbae (version 1.10.1)

*** Dependency 8 = lib-python-scipy (SciPy python library):

    Resolved. CK environment UID = 0da8e1d4cf1ac35e (version 1.10.1)
  -----------------------------------

  (pre processing via CK (/home/anton/CK_REPOS/ck-tensorflow/script/cached-benchmark, preprocess)

  (pre processing script via CK failed: problem loading python code: can't load module code (path=/home/anton/CK_REPOS/ck-tensorflow/script/cached-benchmark, name=preprocess, err=No module named 'numpy'))

[psyhtest]

$ ck show env --tags=scipy
Env UID:         Target OS: Bits: Name:                                Version: Tags:

0da8e1d4cf1ac35e   linux-64    64 SciPy Python library (prebuilt, cpu) 1.10.1   64bits,channel-stable,host-os-linux-64,lib,needs-python,needs-python-3.6.5,python-package,scipy,target-os-linux-64,tensorflow,tensorflow-cpu,v1,v1.10,v1.10.1,vcpu,vprebuilt

$ cat `ck find env:0da8e1d4cf1ac35e`/env.sh
#! /bin/bash
# CK generated script

if [ "$1" != "1" ]; then if [ "$CK_ENV_PYTHON_PACKAGE_SCIPY_SET" == "1" ]; then return; fi; fi

# Soft UOA           = lib.python.scipy (4460bdb0ade2a3df)  (lib,python-package,scipy,needs-python,needs-python-3.6.5,tensorflow,tensorflow-cpu,v1,v1.10,v1.10.1,vprebuilt,vcpu,channel-stable,host-os-linux-64,target-os-linux-64,64bits)
# Host OS UOA        = linux-64 (4258b5fe54828a50)
# Target OS UOA      = linux-64 (4258b5fe54828a50)
# Target OS bits     = 64
# Tool version       = 1.10.1
# Tool split version = [1, 10, 1]

export PYTHONPATH=/home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib:${PYTHONPATH}

export CK_ENV_PYTHON_PACKAGE_SCIPY_SET=1

[psyhtest]

$ ck show env --tags=numpy
Env UID:         Target OS: Bits: Name:                                Version: Tags:

f145fbbd4742bbae   linux-64    64 NumPy Python library (prebuilt, cpu) 1.10.1   64bits,channel-stable,host-os-linux-64,lib,needs-python,needs-python-3.6.5,numpy,python-package,target-os-linux-64,tensorflow,tensorflow-cpu,v1,v1.10,v1.10.1,vcpu,vprebuilt

$ cat `ck find env:f145fbbd4742bbae`/env.sh
#! /bin/bash
# CK generated script

if [ "$1" != "1" ]; then if [ "$CK_ENV_PYTHON_PACKAGE_NUMPY_SET" == "1" ]; then return; fi; fi

# Soft UOA           = lib.python.numpy (6a10047b1bcd16fc)  (lib,python-package,numpy,needs-python,needs-python-3.6.5,tensorflow,tensorflow-cpu,v1,v1.10,v1.10.1,vprebuilt,vcpu,channel-stable,host-os-linux-64,target-os-linux-64,64bits)
# Host OS UOA        = linux-64 (4258b5fe54828a50)
# Target OS UOA      = linux-64 (4258b5fe54828a50)
# Target OS bits     = 64
# Tool version       = 1.10.1
# Tool split version = [1, 10, 1]

export PYTHONPATH=/home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib:${PYTHONPATH}

export CK_ENV_PYTHON_PACKAGE_NUMPY_SET=1

[psyhtest]

As the above NumPy and SciPy env entries show, PYTHONPATH gets set to /home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib in both cases.

[psyhtest]

It's a bit more complicated for TensorFlow itself:

$ cat `ck find env:2b13628df1484c9a`/env.sh
#! /bin/bash
# CK generated script

if [ "$1" != "1" ]; then if [ "$CK_ENV_LIB_TF_SET" == "1" ]; then return; fi; fi

# Soft UOA           = lib.tensorflow (30db354f469bb178)  (lib,tensorflow,needs-python,needs-python-3.6.5,tensorflow-cpu,v1,v1.10,v1.10.1,vprebuilt,vcpu,channel-stable,host-os-linux-64,target-os-linux-64,64bits)
# Host OS UOA        = linux-64 (4258b5fe54828a50)
# Target OS UOA      = linux-64 (4258b5fe54828a50)
# Target OS bits     = 64
# Tool version       = 1.10.1
# Tool split version = [1, 10, 1]

export PYTHONPATH=/home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib:/home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib/external/protobuf_archive/python:${PYTHONPATH}
export CK_ENV_LIB_TF=/home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64
export CK_ENV_LIB_TF_LIB=/home/anton/CK_TOOLS/tensorflow-prebuilt-cpu-1.10.1-compiler.python-3.6.5-linux-64/lib

export CK_ENV_LIB_TF_SET=1

[psyhtest]

As a workaround for the preprocessing failure, I can do:

$ ck virtual env --tags=lib,scipy --shell_cmd="ck run program:image-classification-tf-py"

[ens-lg4]

@psyhtest , I believe this has been now fixed?

[psyhtest]

Closing as we now have more robust mechanisms to handle this.