search

conda-forge / tensorflow-feedstock

A conda-smithy repository for tensorflow.
BSD 3-Clause "New" or "Revised" License
91 stars 81 forks source link

2.15 does not use GPU. Other versions do #365

Closed edwardyehuang closed 5 months ago

edwardyehuang commented 6 months ago

Solution to issue cannot be found in the documentation.

Issue

2.15 does not use GPU. Other versions do

Installed packages

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   2.0.0              pyhd8ed1ab_0    conda-forge
aiohttp                   3.9.1           py311h459d7ec_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
blinker                   1.7.0              pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py311hb755f60_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.24.0               hd590300_0    conda-forge
ca-certificates           2023.11.17           hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.3.2              pyhd8ed1ab_0    conda-forge
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py311hb3a22ac_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
cryptography              41.0.7          py311hcb13ee4_1    conda-forge
cuda-cudart               12.0.107             h59595ed_1    conda-forge
cuda-nvcc-tools           12.0.76              h59595ed_1    conda-forge
cuda-nvrtc                12.0.76              hd3aeb46_2    conda-forge
cuda-nvtx                 12.0.76              h59595ed_1    conda-forge
cuda-version              12.0                 hffde075_2    conda-forge
cudnn                     8.8.0.121            h264754d_4    conda-forge
flatbuffers               23.5.26              h59595ed_1    conda-forge
frozenlist                1.4.1           py311h459d7ec_0    conda-forge
gast                      0.5.4              pyhd8ed1ab_0    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
google-auth               2.25.2             pyhca7485f_0    conda-forge
google-auth-oauthlib      1.2.0              pyhd8ed1ab_0    conda-forge
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
grpcio                    1.59.3          py311ha6695c7_0    conda-forge
h5py                      3.10.0          nompi_py311hebc2b07_101    conda-forge
hdf5                      1.14.3          nompi_h4f84152_100    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
importlib-metadata        7.0.1              pyha770c72_0    conda-forge
keras                     2.15.0             pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
libaec                    1.1.2                h59595ed_1    conda-forge
libblas                   3.9.0           20_linux64_openblas    conda-forge
libcblas                  3.9.0           20_linux64_openblas    conda-forge
libcublas                 12.0.1.189           hd3aeb46_3    conda-forge
libcufft                  11.0.0.21            hd3aeb46_2    conda-forge
libcurand                 10.3.1.50            hd3aeb46_1    conda-forge
libcurl                   8.5.0                hca28451_0    conda-forge
libcusolver               11.4.2.57            hd3aeb46_2    conda-forge
libcusparse               12.0.0.76            hd3aeb46_2    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgfortran-ng            13.2.0               h69a702a_3    conda-forge
libgfortran5              13.2.0               ha4646dd_3    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
libgrpc                   1.59.3               hd6c4280_0    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           20_linux64_openblas    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libnvjitlink              12.0.76              hd3aeb46_2    conda-forge
libopenblas               0.3.25          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libprotobuf               4.24.4               hf27288f_0    conda-forge
libre2-11                 2023.06.02           h7a70373_0    conda-forge
libsqlite                 3.44.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libtensorflow             2.15.0           cpu_h8d94747_1    conda-forge
libtensorflow_cc          2.15.0           cpu_h2f60d71_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
markdown                  3.5.1              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3           py311h459d7ec_1    conda-forge
ml_dtypes                 0.2.0           py311h320fe9a_2    conda-forge
multidict                 6.0.4           py311h459d7ec_1    conda-forge
nccl                      2.19.4.1             h3a97aeb_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
numpy                     1.26.2          py311h64a7726_0    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openssl                   3.2.0                hd590300_1    conda-forge
opt_einsum                3.3.0              pyhc1e730c_2    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
protobuf                  4.24.4          py311h46cbc50_0    conda-forge
pyasn1                    0.5.1              pyhd8ed1ab_0    conda-forge
pyasn1-modules            0.3.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyjwt                     2.8.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 23.3.0             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.7          hab00c5b_1_cpython    conda-forge
python-flatbuffers        23.5.26            pyhd8ed1ab_0    conda-forge
python_abi                3.11                    4_cp311    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
re2                       2023.06.02           h2873b5e_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rsa                       4.9                pyhd8ed1ab_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
tensorboard               2.15.1             pyhd8ed1ab_0    conda-forge
tensorboard-data-server   0.7.0           py311h63ff55d_1    conda-forge
tensorflow                2.15.0          cuda120py311h5cbd639_1    conda-forge
tensorflow-base           2.15.0          cuda120py311h43b5e44_1    conda-forge
tensorflow-estimator      2.15.0          cuda120py311hf663016_1    conda-forge
tensorflow-gpu            2.15.0          cuda120py311hb76ca00_1    conda-forge
termcolor                 2.3.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2023d                h0c530f3_0    conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    conda-forge
werkzeug                  3.0.1              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.1          py311hd4cff14_1    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yarl                      1.9.3           py311h459d7ec_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Environment info

active environment : tf215
    active env location : /home/edwardyehuang/miniconda3/envs/tf215
            shell level : 3
       user config file : /home/edwardyehuang/.condarc
 populated config files : /home/edwardyehuang/.condarc
          conda version : 4.12.0
    conda-build version : not installed
         python version : 3.9.12.final.0
       virtual packages : __cuda=12.2=0
                          __linux=6.2.0=0
                          __glibc=2.35=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/edwardyehuang/miniconda3  (writable)
      conda av data dir : /home/edwardyehuang/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/edwardyehuang/miniconda3/pkgs
                          /home/edwardyehuang/.conda/pkgs
       envs directories : /home/edwardyehuang/miniconda3/envs
                          /home/edwardyehuang/.conda/envs
               platform : linux-64
             user-agent : conda/4.12.0 requests/2.27.1 CPython/3.9.12 Linux/6.2.0-32-generic ubuntu/22.04.3 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
hmaarrfk commented 6 months ago

I don’t doubt what you are saying but It would really help if you provided the code to validate that the GPU isn’t being used.

edwardyehuang commented 6 months ago 
import tensorflow as tf
a = tf.constant(2) + 2

In 2.15, only the CPU is used, and the above code works fine in <=2.14

All versions are installed via conda install -c conda-forge tensorflow-gpu==xxxx

hmaarrfk commented 6 months ago

If you can make the above an assert statement, we can add it to our tests suite. Can you try to have an assert statement in your test please.

hmaarrfk commented 6 months ago

The test I found that fails with tensorflow 2.15.0 is:

import tensorflow as tf
assert len(tf.config.list_physical_devices(device_type='CPU')), "No CPU devices found"
assert len(tf.config.list_physical_devices(device_type='GPU')), "No GPU devices found"

with tensorflow 2.13 + cuda 11.8 on my machine it passes.

hmaarrfk commented 6 months ago

@jakirkham @xhochy any clues (just in case you are looking to burn free time today!)

njzjz commented 6 months ago 
>>> import tensorflow as tf
2023-12-29 16:17:51.727737: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> tf.test.is_built_with_cuda()
False
tensorflow                2.15.0          cuda120py311h5cbd639_1    conda-forge
tensorflow-base           2.15.0          cuda120py311h43b5e44_1    conda-forge
tensorflow-estimator      2.15.0          cuda120py311hf663016_1    conda-forge
njzjz commented 6 months ago

The GPU versions have the same size as the CPU versions, which should be incorrect:

image

hmaarrfk commented 6 months ago

Good observation.

Sorry for not catching this earlier.

We should probably mark the GPU builds here as broken.

edwardyehuang commented 5 months ago

Thank you. I tested it and it worked.