search

janclemenslab / das

Deep Audio Segmenter
http://janclemenslab.org/das/
28 stars 10 forks source link

On Windows, TensorFlow/CUDA lib not found #36

Closed avakiai closed 2 years ago

avakiai commented 2 years ago

Hi Jan & Team,

I'm trying to train my model through the gui and am running into a problem with some of das's dependencies. Apologies if this post is inappropriate here, but I thought it might be helpful if any other users encouter a similar issue.

I'll emphasize that I think this is not a das issue, but rather an issue to do with the tensorflow and/or CUDA (NVIDI's GPU driver) packages not looking in the right places for a needed library.

Here is the problem: The training initializes fine, but then I get the following error: 

INFO:das.train:Will save to F:/Ava/Experiment_2/Exp2_DAS_selections/training_set.res/20220705_144815.
INFO:das.train:start training
2022-07-05 14:48:15.133820: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
Epoch 1/400
2022-07-05 14:48:21.231684: I tensorflow/compiler/xla/service/service.cc:171] XLA service 0x21d111ef360 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-07-05 14:48:21.231839: I tensorflow/compiler/xla/service/service.cc:179]   StreamExecutor device (0): GeForce RTX 2080 SUPER, Compute Capability 7.5
2022-07-05 14:48:21.243508: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:75] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
2022-07-05 14:48:21.243765: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:76] Searched for CUDA in the following directories:
2022-07-05 14:48:21.244053: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:79]   ./cuda_sdk_lib
2022-07-05 14:48:21.244239: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:79]   C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.3
2022-07-05 14:48:21.244466: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:79]   /usr/local/cuda
2022-07-05 14:48:21.244651: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:79]   .
2022-07-05 14:48:21.244850: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:81] You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions.  For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
2022-07-05 14:48:21.245787: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:327] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2022-07-05 14:48:21.245978: I tensorflow/compiler/jit/xla_compilation_cache.cc:363] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
2022-07-05 14:48:21.246128: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at xla_ops.cc:432 : Internal: libdevice not found at ./libdevice.10.bc
2022-07-05 14:48:21.327881: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:210] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2022-07-05 14:48:24.722848: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8201
2022-07-05 14:48:26.940443: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:327] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2022-07-05 14:48:26.978341: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at xla_ops.cc:432 : Internal: libdevice not found at ./libdevice.10.bc
Traceback (most recent call last):
  File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\xarray_behave\gui\utils.py", line 222, in run
    self.fn(*self.args, **self.kwargs)
  File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\das\train.py", line 400, in train
    fit_hist = model.fit(
  File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1193, in fit
    tmp_logs = self.train_function(iterator)
  File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 3039, in __call__
    return graph_function._call_flat(
  File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 1963, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\function.py", line 591, in call
    outputs = execute.execute(
  File "C:\Users\AC-Lab-AKK\anaconda3\envs\das\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError:  libdevice not found at ./libdevice.10.bc
         [[{{node cluster_1_1/xla_compile}}]] [Op:__inference_train_function_13833]

The critical line appears to be: 

2022-07-05 14:48:21.243765: W tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:76] Searched for CUDA in the following directories:

This appears to be a known issue when using tensor/cuda in anaconda, see here and here. However, I haven't been able to solve it. (There are some suggestions for how to set XLA's search path, but I have not been able to change the search path whether through the terminal or a py script.)

Before I proceed with debugging this, I wanted to ask what the latest version of TensorFlow and CUDA is that might still be compatible with das? Or, if you happen to have run into this problem already, what you might recommend.

For reference, my conda das environment has the following (again, recently downloaded as per install instructions and updated xarray via pip): 


(das) C:\Users\AC-Lab-AKK>conda list -n das
# packages in environment at C:\Users\AC-Lab-AKK\anaconda3\envs\das:
#
# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu    anaconda
abseil-cpp                20210324.2           h0e60522_0    conda-forge
absl-py                   1.1.0              pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.1            py39hb82d6ee_1    conda-forge
aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
aom                       3.3.0                h0e60522_1    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
astor                     0.8.1              pyh9f0ad1d_0    conda-forge
asttokens                 2.0.5              pyhd8ed1ab_0    conda-forge
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
audioread                 2.1.9            py39hcbf5309_3    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.1                pyhd3eb1b0_0    anaconda
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
blinker                   1.4                        py_1    conda-forge
blosc                     1.21.1               h74325e0_3    conda-forge
bokeh                     2.4.3            py39hcbf5309_0    conda-forge
brotli                    1.0.9                h8ffe710_7    conda-forge
brotli-bin                1.0.9                h8ffe710_7    conda-forge
brotlipy                  0.7.0           py39hb82d6ee_1004    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
c-blosc2                  2.1.1                hdf67494_2    conda-forge
ca-certificates           2022.6.15            h5b45459_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                4.2.4              pyhd8ed1ab_0    conda-forge
certifi                   2022.6.15        py39hcbf5309_0    conda-forge
cffi                      1.15.0           py39h0878f49_0    conda-forge
cfitsio                   4.1.0                h5a969a9_0    conda-forge
charls                    2.3.4                h39d44d4_0    conda-forge
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
click                     8.1.3            py39hcbf5309_0    conda-forge
cloudpickle               2.1.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
colorcet                  3.0.0              pyhd8ed1ab_0    conda-forge
cryptography              37.0.2           py39h7bc7c5c_0    conda-forge
cudatoolkit               11.3.1              h280eb24_10    conda-forge
cudnn                     8.2.1.32             h754d62a_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.11.2           py39hb82d6ee_2    conda-forge
das                       0.26.4                   py39_0    ncb
dask                      2022.6.0           pyhd8ed1ab_0    conda-forge
dask-core                 2022.6.0           pyhd8ed1ab_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defopt                    6.3.0              pyhd8ed1ab_0    conda-forge
distributed               2022.6.0           pyhd8ed1ab_0    conda-forge
docutils                  0.18.1           py39hcbf5309_1    conda-forge
executing                 0.8.3              pyhd8ed1ab_0    conda-forge
fasteners                 0.17.3             pyhd8ed1ab_0    conda-forge
flammkuchen               0.9.2                      py_0    conda-forge
flatbuffers               2.0.6                h0e60522_0    conda-forge
fonttools                 4.33.3           py39hb82d6ee_0    conda-forge
freeglut                  3.2.2                h0e60522_1    conda-forge
freetype                  2.10.4               h546665d_1    conda-forge
frozenlist                1.3.0            py39hb82d6ee_1    conda-forge
fsspec                    2022.5.0           pyhd8ed1ab_0    conda-forge
future                    0.18.2           py39hcbf5309_5    conda-forge
gast                      0.4.0              pyh9f0ad1d_0    conda-forge
giflib                    5.2.1                h8d14728_2    conda-forge
google-auth               1.35.0             pyh6c4a22f_0    conda-forge
google-auth-oauthlib      0.4.6              pyhd8ed1ab_0    conda-forge
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
grpcio                    1.46.3           py39hb76b349_0    conda-forge
h5py                      3.6.0           nompi_py39hd4deaf1_100    conda-forge
hdf5                      1.12.1          nompi_h2a0e4a3_104    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
icu                       68.2                 h0e60522_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
imagecodecs               2022.2.22        py39h43fe4e9_6    conda-forge
imageio                   2.19.3             pyhcf75d05_0    conda-forge
importlib-metadata        4.11.4           py39hcbf5309_0    conda-forge
importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
intel-openmp              2022.1.0          h57928b3_3787    conda-forge
ipython                   8.4.0            py39hcbf5309_0    conda-forge
jasper                    2.0.33               h77af90b_0    conda-forge
jedi                      0.18.1           py39hcbf5309_1    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h8ffe710_1    conda-forge
jxrlib                    1.1                  h8ffe710_2    conda-forge
keras-preprocessing       1.1.2              pyhd8ed1ab_0    conda-forge
keras-tuner               1.1.0              pyhd8ed1ab_0    conda-forge
kiwisolver                1.4.3            py39h2e07f2f_0    conda-forge
krb5                      1.19.3               h1176d77_0    conda-forge
kt-legacy                 1.0.4                    pypi_0    pypi
lcms2                     2.12                 h2a16943_0    conda-forge
lerc                      3.0                  h0e60522_0    conda-forge
libaec                    1.0.6                h39d44d4_0    conda-forge
libavif                   0.10.1               h8ffe710_0    conda-forge
libblas                   3.9.0              15_win64_mkl    conda-forge
libbrotlicommon           1.0.9                h8ffe710_7    conda-forge
libbrotlidec              1.0.9                h8ffe710_7    conda-forge
libbrotlienc              1.0.9                h8ffe710_7    conda-forge
libcblas                  3.9.0              15_win64_mkl    conda-forge
libclang                  11.1.0          default_h5c34c98_1    conda-forge
libcurl                   7.83.1               h789b8ee_0    conda-forge
libdeflate                1.12                 h8ffe710_0    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libflac                   1.3.4                h0e60522_0    conda-forge
libiconv                  1.16                 he774522_0    conda-forge
liblapack                 3.9.0              15_win64_mkl    conda-forge
liblapacke                3.9.0              15_win64_mkl    conda-forge
libogg                    1.3.5                h2bbff1b_1    anaconda
libopencv                 4.5.1            py39h27d8466_0    conda-forge
libopus                   1.3.1                h8ffe710_1    conda-forge
libpng                    1.6.37               h1d00b33_2    conda-forge
libprotobuf               3.17.2               h7755175_1    conda-forge
librosa                   0.9.1              pyhd8ed1ab_0    conda-forge
libsndfile                1.0.31               h0e60522_1    conda-forge
libssh2                   1.10.0               h680486a_2    conda-forge
libtiff                   4.4.0                h2ed3b44_1    conda-forge
libvorbis                 1.3.7                h0e60522_0    conda-forge
libwebp                   1.2.2                h57928b3_0    conda-forge
libwebp-base              1.2.2                h8ffe710_1    conda-forge
libxcb                    1.13              hcd874cb_1004    conda-forge
libxml2                   2.9.14               hf5bbc77_0    conda-forge
libxslt                   1.1.35               h34f844d_0    conda-forge
libzlib                   1.2.12               h8ffe710_1    conda-forge
libzopfli                 1.0.3                h0e60522_0    conda-forge
llvmlite                  0.38.1           py39ha0cd8c8_0    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.0.0            py39h0878066_2    conda-forge
lz4-c                     1.9.3                h8ffe710_1    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
markdown                  3.3.7              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.1            py39hb82d6ee_1    conda-forge
matplotlib                3.5.2            py39hcbf5309_0    conda-forge
matplotlib-base           3.5.2            py39h581301d_0    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
matplotlib-scalebar       0.8.1              pyhd8ed1ab_0    conda-forge
mkl                       2022.1.0           h6a75c08_874    conda-forge
msgpack-python            1.0.4            py39h2e07f2f_0    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
multidict                 6.0.2            py39hb82d6ee_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mypy_extensions           0.4.3            py39hcbf5309_5    conda-forge
networkx                  2.8.4              pyhd8ed1ab_0    conda-forge
numba                     0.55.2           py39hb8cd55e_0    conda-forge
numcodecs                 0.9.1            py39h415ef7b_3    conda-forge
numexpr                   2.7.3            py39h2e25243_2    conda-forge
numpy                     1.22.4           py39h0948cea_0    conda-forge
oauthlib                  3.2.0              pyhd8ed1ab_0    conda-forge
opencv                    4.5.1            py39hcbf5309_0    conda-forge
openjpeg                  2.4.0                hb211442_1    conda-forge
openssl                   1.1.1o               h8ffe710_0    conda-forge
opt_einsum                3.3.0              pyhd8ed1ab_1    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.4.2            py39h2e25243_2    conda-forge
param                     1.12.1             pyh6c4a22f_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.2.0              pyhd8ed1ab_0    conda-forge
peakutils                 1.3.3                      py_0    conda-forge
pickleshare               0.7.5           py39hde42818_1002    conda-forge
pillow                    9.1.1            py39ha53f419_1    conda-forge
pip                       22.1.2             pyhd8ed1ab_0    conda-forge
pockets                   0.9.1                      py_0    conda-forge
pooch                     1.6.0              pyhd8ed1ab_0    conda-forge
portaudio                 19.6.0               h0e60522_5    conda-forge
prompt-toolkit            3.0.29             pyha770c72_0    conda-forge
protobuf                  3.17.2           py39h415ef7b_0    conda-forge
psutil                    5.9.1            py39hb82d6ee_0    conda-forge
pthread-stubs             0.4               hcd874cb_1001    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
py-opencv                 4.5.1            py39h832f523_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.8                      py_0    anaconda
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyct                      0.4.6                      py_0    conda-forge
pyct-core                 0.4.6                      py_0    conda-forge
pygments                  2.12.0             pyhd8ed1ab_0    conda-forge
pyjwt                     2.4.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyqt                      5.12.3           py39hcbf5309_8    conda-forge
pyqt-impl                 5.12.3           py39h415ef7b_8    conda-forge
pyqt5-sip                 4.19.18          py39h415ef7b_8    conda-forge
pyqtchart                 5.12             py39h415ef7b_8    conda-forge
pyqtgraph                 0.12.4             pyhd8ed1ab_0    conda-forge
pyqtwebengine             5.12.1           py39h415ef7b_8    conda-forge
pyside2                   5.13.2           py39hc084194_4    conda-forge
pysocks                   1.7.1            py39hcbf5309_5    conda-forge
pysoundfile               0.10.3.post1       pyhd3deb0d_0    conda-forge
pytables                  3.7.0            py39hbcfe41f_0    conda-forge
python                    3.9.13          h9a09f29_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-flatbuffers        1.12               pyhd8ed1ab_1    conda-forge
python-sounddevice        0.4.1              pyh9f0ad1d_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyvideoreader             0.5.6              pyhfe9d6e3_0    conda-forge
pywavelets                1.3.0            py39h5d4886f_1    conda-forge
pyyaml                    6.0              py39hb82d6ee_4    conda-forge
qt                        5.12.9               h5909a2a_4    conda-forge
qtpy                      2.1.0              pyhd8ed1ab_0    conda-forge
requests                  2.28.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
resampy                   0.2.2                      py_0    conda-forge
rsa                       4.8                pyhd8ed1ab_0    conda-forge
samplestamps              0.4.4                    py39_0    ncb
scikit-image              0.19.3           py39h2e25243_0    conda-forge
scikit-learn              1.1.1            py39he931e04_0    conda-forge
scipy                     1.8.1            py39hc0c34ad_0    conda-forge
setuptools                62.6.0           py39hcbf5309_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.9                h82413e6_1    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-napoleon    0.7                        py_0    conda-forge
sqlite                    3.38.5               h8ffe710_0    conda-forge
stack_data                0.3.0              pyhd8ed1ab_0    conda-forge
tabulate                  0.8.9              pyhd8ed1ab_0    conda-forge
tbb                       2021.5.0             h2d74725_1    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
tensorboard               2.6.0              pyhd8ed1ab_1    conda-forge
tensorboard-data-server   0.6.0            py39hcbf5309_2    conda-forge
tensorboard-plugin-wit    1.8.1              pyhd8ed1ab_0    conda-forge
tensorflow                2.6.0           gpu_py39he88c5ba_0    anaconda
tensorflow-base           2.6.0           gpu_py39hb3da07e_0    anaconda
tensorflow-estimator      2.6.0              pyh7b7c402_0    anaconda
tensorflow-gpu            2.6.0                h17022bd_0    anaconda
termcolor                 1.1.0                      py_2    conda-forge
terminaltables            3.1.10             pyhd8ed1ab_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tifffile                  2022.5.4           pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h8ffe710_0    conda-forge
toolz                     0.11.2             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py39hb82d6ee_3    conda-forge
tqdm                      4.64.0             pyhd8ed1ab_0    conda-forge
traitlets                 5.3.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.2.0                hd8ed1ab_1    conda-forge
typing_extensions         4.2.0              pyha770c72_1    conda-forge
typing_inspect            0.7.1              pyh6c4a22f_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
ucrt                      10.0.20348.0         h57928b3_0    conda-forge
unicodedata2              14.0.0           py39hb82d6ee_1    conda-forge
urllib3                   1.26.9             pyhd8ed1ab_0    conda-forge
vc                        14.2                 hb210afc_6    conda-forge
vs2015_runtime            14.29.30037          h902a5da_6    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
werkzeug                  2.1.2              pyhd8ed1ab_1    conda-forge
wheel                     0.35.1             pyh9f0ad1d_0    conda-forge
win_inet_pton             1.1.0            py39hcbf5309_4    conda-forge
wrapt                     1.14.1           py39hb82d6ee_0    conda-forge
xarray                    2022.3.0           pyhd8ed1ab_0    conda-forge
xarray-behave             0.33.0                   pypi_0    pypi
xorg-libxau               1.0.9                hcd874cb_0    conda-forge
xorg-libxdmcp             1.1.3                hcd874cb_0    conda-forge
xz                        5.2.5                h62dcd97_1    conda-forge
yaml                      0.2.5                h8ffe710_2    conda-forge
yarl                      1.7.2            py39hb82d6ee_2    conda-forge
zarr                      2.11.3             pyhd8ed1ab_0    conda-forge
zfp                       0.5.5                h0e60522_8    conda-forge
zict                      2.2.0              pyhd8ed1ab_0    conda-forge
zipp                      3.8.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.12               h8ffe710_1    conda-forge
zlib-ng                   2.0.6                h8ffe710_0    conda-forge
zstd                      1.5.2                h6255e5f_1    conda-forge

Thanks a lot!

postpop commented 2 years ago

Hi, yes that looks familiar - I also had found those unhelpful stackoverflow posts :-) We recently enabled jit compilation in tensorflow, hoping that would make DAS a little faster. But jit causes issues on some windows machines. We therefore disabled it in the latest version 0.26.6. Re-creating the env or updating DAS via pip install das --upgrade --no-deps should fix the issue.

avakiai commented 2 years ago

Amazing! Thank you so much! Indeed it's running now!

Just thought I'd also leave here two further possible fixes (not tried):

(In the meantime, I had started training in a colab file. I'm curious now which completes faster. :) One tangential question: Is it possible to also set a data_splits parameter when using das.train.train in a py script? Or is this only available in the gui version?)

Thank you!

postpop commented 2 years ago

Glad this worked!

Splits are created when making the dataset - if you want to try different splits, you have to create different datasets.