HDDM - Githubissues

laraPPr commented 7 months ago

link to support ticket: #2024041060000012
website: https://hddm.readthedocs.io/en/latest/index.html#
installation docs: https://hddm.readthedocs.io/en/latest/index.html#installation
toolchain: ?
easyblock to use: PythonBundle
at least v0.9.8
required dependencies:
- [x] cython
- [x] scipy
- [x] seaborn
- [x] statsmodels
- [x] scikit-learn
- [x] tqdm
- [x] Pymc3
- [x] matplotlib
- [x] git+https://github.com/AlexanderFengler/ssm_simulators@main
- [x] git+https://github.com/hddm-devs/kabuki
optional dependencies:
- [x] PyTorch
notes:
- ...
effort: (TBD)
other install methods
- conda: yes (link?) / no
- container image: yes (link?) / no
- pre-built binaries (RHEL8 Linux x86_64): yes (link?) / no
- easyconfig outside EasyBuild: yes / no

pavelToman commented 7 months ago

https://github.com/easybuilders/easybuild-easyconfigs/pull/20370

pavelToman commented 7 months ago

I was unable to make it work for newer toolchain because of its dependency PyMC >= 2.3.3, < 3.0.0 - this one is quiet old. I try to use newer PyMC, but it does not work with 3.0.0 and newer. And I was unable to do PyMC 2.3.8 for newer toolchain. Example of error with newer PyMC: error: missing pymc.stochastic_from_dist() -> there is no such function for v3.0 and latter, there is in 2.3.8

boegel commented 6 months ago

Researcher reported that HDDM is missing a dependency on PyTorch:

HDDM: pytorch module seems missing. No LAN functionality can be loaded.
It seems that you do not have pytorch installed. You cannot use the network_inspector module.
It seems that you do not have pytorch installed.The HDDMnn, HDDMnnRegressor, HDDMnnStimCoding, HDDMnnRL and HDDMnnRLRegressorclasses will not work
It seems that you do not have pytorch installed.The HDDMnn, HDDMnnRegressor, HDDMnnStimCoding, HDDMnnRL and HDDMnnRLRegressorclasses will not work
It seems that you do not have pytorch installed.The HDDMnn, HDDMnnRegressor and HDDMnnStimCodingclasses will not work
It seems that you do not have pytorch installed.The HDDMnn, HDDMnnRL, HDDMnnRegressor, HDDMnnStimCoding, HDDMnnRL and HDDMnnRLRegressorclasses will not work
It seems that you do not have pytorch installed.The HDDMnn, HDDMnnRegressor, HDDMnnStimCoding, HDDMnnRL and HDDMnnRLRegressorclasses will not work
1
Couldn't execute load_torch_mlp()...
Option 1: pytorch not installed or version older than 1.7?
Option 2: pytorch model for your model string is not yet available

This basically means that we'll need to try and get HDDM installed with a foss toolchain, and preferably a recent one.

@pavelToman Can you take another look at this? Maybe try using foss/2021b first, and see if you can fix whatever problems you hit there with PyMC, since we already have a sufficiently recent PyTorch available with foss/2021b.

Ideally we also get this installed with a more recent toolchain (>= foss/2023a)

pavelToman commented 6 months ago

https://github.com/easybuilders/easybuild-easyconfigs/pull/20440

boegel commented 6 months ago

@pavelToman Please add some info on the problems you saw with building PyMC 2.x with recent toolchain, and then this can be closed (and WIP easyconfigs cleaned up)

pavelToman commented 6 months ago

PyMC-foss-2023a: Building wheel for pymc (setup.py): finished with status 'error' Full log is there: https://github.com/vscentrum/vsc-software-stack/blob/d04f5ae7631ca240f9d06782ba6ca7f2b69b4849/323_HDDM/log_pymc-foss-2023a.txt

WARN: CCompilerOpt.dist_test[636] : CCompilerOpt._dist_test_spawn[770] : Command (gcc -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -I/apps/gent/RHEL8/cascadelake-ib/software/FFTW/3.3.10-GCC-12.3.0/include -I/apps/gent/RHEL8/cascadelake-ib/software/FlexiBLAS/3.3.1-GCC-12.3.0/include -I/apps/gent/RHEL8/cascadelake-ib/software/FlexiBLAS/3.3.1-GCC-12.3.0/include/flexiblas -fPIC -DDETECT_FEATURES=1 -I/apps/gent/RHEL8/cascadelake-ib/software/Python/3.11.3-GCCcore-12.3.0/include/python3.11 -c /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.c -o /tmp/vsc47063/eb-hxe8d7ho/tmp08ao2fwk/kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.o -MMD -MF /tmp/vsc47063/eb-hxe8d7ho/tmp08ao2fwk/kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.o.d -march=native -Werror) failed with exit status 1 output ->
      In file included from /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/immintrin.h:79,
                       from /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.c:14:
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/avx512vpopcntdqintrin.h: In function main:
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/avx512vpopcntdqintrin.h:65:1: error: inlining failed in call to always_inline _mm512_popcnt_epi64: target specific option mismatch
         65 | _mm512_popcnt_epi64 (__m512i __A)
            | ^~~~~~~~~~~~~~~~~~~
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.c:24:9: note: called from here
         24 |     a = _mm512_popcnt_epi64(a);
            |         ^~~~~~~~~~~~~~~~~~~~~~
      In file included from /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/immintrin.h:91:
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/avx512bitalgintrin.h:39:1: error: inlining failed in call to always_inline _mm512_popcnt_epi8: target specific option mismatch
         39 | _mm512_popcnt_epi8 (__m512i __A)
            | ^~~~~~~~~~~~~~~~~~
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.c:22:9: note: called from here
         22 |     a = _mm512_popcnt_epi8(a);
            |         ^~~~~~~~~~~~~~~~~~~~~
      In file included from /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/immintrin.h:81:
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/avx512vbmi2intrin.h:252:1: error: inlining failed in call to always_inline _mm512_shrdv_epi64: target specific option mismatch
        252 | _mm512_shrdv_epi64 (__m512i __A, __m512i __B, __m512i __C)
            | ^~~~~~~~~~~~~~~~~~
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.c:20:9: note: called from here
         20 |     a = _mm512_shrdv_epi64(a, a, _mm512_setzero_si512());
            |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/avx512vpopcntdqintrin.h:65:1: error: inlining failed in call to always_inline _mm512_popcnt_epi64: target specific option mismatch
         65 | _mm512_popcnt_epi64 (__m512i __A)
            | ^~~~~~~~~~~~~~~~~~~
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.c:24:9: note: called from here
         24 |     a = _mm512_popcnt_epi64(a);
            |         ^~~~~~~~~~~~~~~~~~~~~~
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/avx512bitalgintrin.h:39:1: error: inlining failed in call to always_inline _mm512_popcnt_epi8: target specific option mismatch
         39 | _mm512_popcnt_epi8 (__m512i __A)
            | ^~~~~~~~~~~~~~~~~~
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.c:22:9: note: called from here
         22 |     a = _mm512_popcnt_epi8(a);
            |         ^~~~~~~~~~~~~~~~~~~~~
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/avx512vbmi2intrin.h:252:1: error: inlining failed in call to always_inline _mm512_shrdv_epi64: target specific option mismatch
        252 | _mm512_shrdv_epi64 (__m512i __A, __m512i __B, __m512i __C)
            | ^~~~~~~~~~~~~~~~~~
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_icl.c:20:9: note: called from here
         20 |     a = _mm512_shrdv_epi64(a, a, _mm512_setzero_si512());
            |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

      WARN: CCompilerOpt.feature_test[1575] : testing failed

...

WARN: CCompilerOpt.dist_test[636] : CCompilerOpt._dist_test_spawn[770] : Command (gcc -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -I/apps/gent/RHEL8/cascadelake-ib/software/FFTW/3.3.10-GCC-12.3.0/include -I/apps/gent/RHEL8/cascadelake-ib/software/FlexiBLAS/3.3.1-GCC-12.3.0/include -I/apps/gent/RHEL8/cascadelake-ib/software/FlexiBLAS/3.3.1-GCC-12.3.0/include/flexiblas -fPIC -I/apps/gent/RHEL8/cascadelake-ib/software/Python/3.11.3-GCCcore-12.3.0/include/python3.11 -c /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_knl.c -o /tmp/vsc47063/eb-hxe8d7ho/tmp08ao2fwk/kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_knl.o -MMD -MF /tmp/vsc47063/eb-hxe8d7ho/tmp08ao2fwk/kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_knl.o.d -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512er -mavx512pf -Werror) failed with exit status 1 output ->
      In file included from /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/immintrin.h:53,
                       from /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_knl.c:14:
      In function _mm512_mask_prefetch_i64scatter_pd,
          inlined from main at /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_knl.c:23:5:
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/GCCcore/12.3.0/lib/gcc/x86_64-pc-linux-gnu/12.3.0/include/avx512pfintrin.h:180:3: error: base may be used uninitialized [-Werror=maybe-uninitialized]
        180 |   __builtin_ia32_scatterpfqpd (__mask, (__v8di) __index, __addr, __scale,
            |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        181 |                                __hint);
            |                                ~~~~~~~
      <built-in>: In function main:
      <built-in>: note: by argument 3 of type const void * to __builtin_ia32_scatterpfqpd declared here
      /kyukon/home/apps/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/distutils/checks/cpu_avx512_knl.c:18:9: note: base declared here
         18 |     int base[128];
            |         ^~~~
      cc1: all warnings being treated as errors

      WARN: CCompilerOpt.feature_test[1575] : testing failed

...

error: Command "gcc -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -I/apps/gent/RHEL8/cascadelake-ib/software/FFTW/3.3.10-GCC-12.3.0/include -I/apps/gent/RHEL8/cascadelake-ib/software/FlexiBLAS/3.3.1-GCC-12.3.0/include -I/apps/gent/RHEL8/cascadelake-ib/software/FlexiBLAS/3.3.1-GCC-12.3.0/include/flexiblas -fPIC -DHAVE_CBLAS -DNO_ATLAS_INFO=1 -I/apps/gent/RHEL8/cascadelake-ib/software/FlexiBLAS/3.3.1-GCC-12.3.0/include/flexiblas -Ibuild/src.linux-x86_64-3.11/build/src.linux-x86_64-3.11/pymc -I/apps/gent/RHEL8/cascadelake-ib/software/SciPy-bundle/2023.07-gfbf-2023a/lib/python3.11/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.11/numpy/distutils/include -I/apps/gent/RHEL8/cascadelake-ib/software/Python/3.11.3-GCCcore-12.3.0/include/python3.11 -c build/src.linux-x86_64-3.11/pymc/flibmodule.c -o build/temp.linux-x86_64-cpython-311/build/src.linux-x86_64-3.11/pymc/flibmodule.o -MMD -MF build/temp.linux-x86_64-cpython-311/build/src.linux-x86_64-3.11/pymc/flibmodule.o.d -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq -mavx512vnni" failed with exit status 1
      INFO:
      ########### EXT COMPILER OPTIMIZATION ###########
      INFO: Platform      :
        Architecture: x64
        Compiler    : gcc

      CPU baseline  :
        Requested   : 'native'
        Enabled     : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX
        Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq -mavx512vnni
        Extra checks: AVX512F_REDUCE AVX512BW_MASK AVX512DQ_MASK

      CPU dispatch  :
        Requested   : 'max -xop -fma4'
        Enabled     : AVX512_CNL AVX512_ICL AVX512_SPR
        Generated   : none

pavelToman commented 6 months ago

PyMC-foss-2022a: Building wheel for pymc (setup.py): finished with status 'error' Full log is there: https://github.com/vscentrum/vsc-software-stack/blob/dddd532a461dc565a767a6d93237e25eb5860c56/323_HDDM/log_pymc-foss-2022a.txt ...errors are same as for 2023a WARN: CCompilerOpt.feature_test[1490] : testing failed WARN: CCompilerOpt.feature_test[1490] : testing failed

error: Command "gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -I/apps/gent/RHEL8/cascadelake-ib/software/FFTW/3.3.10-GCC-11.3.0/include -I/apps/gent/RHEL8/cascadelake-ib/software/FlexiBLAS/3.2.0-GCC-11.3.0/include -I/apps/gent/RHEL8/cascadelake-ib/software/FlexiBLAS/3.2.0-GCC-11.3.0/include/flexiblas -fPIC -I/apps/gent/RHEL8/cascadelake-ib/software/SciPy-bundle/2022.05-foss-2022a/lib/python3.10/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.10/numpy/distutils/include -I/apps/gent/RHEL8/cascadelake-ib/software/Python/3.10.4-GCCcore-11.3.0/include/python3.10 -c pymc/LazyFunction.c -o build/temp.linux-x86_64-cpython-310/pymc/LazyFunction.o -MMD -MF build/temp.linux-x86_64-cpython-310/pymc/LazyFunction.o.d -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mavx512cd -mavx512vl -mavx512bw -mavx512dq -mavx512vnni" failed with exit status 1
      INFO:
      ########### EXT COMPILER OPTIMIZATION ###########
      INFO: Platform      :
        Architecture: x64
        Compiler    : gcc

      CPU baseline  :
        Requested   : 'native'
        Enabled     : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX
        Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mavx512cd -mavx512vl -mavx512bw -mavx512dq -mavx512vnni
        Extra checks: AVX512F_REDUCE AVX512BW_MASK AVX512DQ_MASK

      CPU dispatch  :
        Requested   : 'max -xop -fma4'
        Enabled     : AVX512_KNL AVX512_KNM AVX512_CNL AVX512_ICL
        Generated   : none

vscentrum / vsc-software-stack

HDDM #323