Failing setup and mnist installation when following 'Getting started'

[guspih]

Gets errors when running 'bash setup_environment.sh' which presumably causes further errors when attempting to Download mnist ('python util/data/get_a_dataset.py --dataset mnist --output-folder toy_dataset') when following the 'Getting started' instructions.

Printout from command line for Getting started process:

Collecting package metadata (repodata.json): done
Solving environment: done
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Ran pip subprocess with arguments:
['/opt/Anaconda/envs/deepdiva/bin/python', '-m', 'pip', 'install', '-U', '-r', '/nfs/staff/gusgru/Repositories/DeepDIVA/condaenv.hygei685.requirements.txt']
Pip subprocess output:
Requirement already up-to-date: dask==1.2.2 in /opt/Anaconda/envs/deepdiva/lib/python3.7/site-packages (from -r /nfs/staff/gusgru/Repositories/DeepDIVA/condaenv.hygei685.requirements.txt (line 1)) (1.2.2)
Collecting pydensecrf==1.0rc2 (from -r /nfs/staff/gusgru/Repositories/DeepDIVA/condaenv.hygei685.requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/dd/11/7a79e817078323893f8071bb3fda1f45e34e443ec4a0790f8542158bbced/pydensecrf-1.0rc2.tar.gz
Collecting rarfile==3.0 (from -r /nfs/staff/gusgru/Repositories/DeepDIVA/condaenv.hygei685.requirements.txt (line 3))
Requirement already up-to-date: torch==1.0.1.post2 in /opt/Anaconda/envs/deepdiva/lib/python3.7/site-packages (from -r /nfs/staff/gusgru/Repositories/DeepDIVA/condaenv.hygei685.requirements.txt (line 4)) (1.0.1.post2)
Requirement already up-to-date: wget==3.2 in /opt/Anaconda/envs/deepdiva/lib/python3.7/site-packages (from -r /nfs/staff/gusgru/Repositories/DeepDIVA/condaenv.hygei685.requirements.txt (line 5)) (3.2)
Building wheels for collected packages: pydensecrf
  Building wheel for pydensecrf (setup.py): started
  Building wheel for pydensecrf (setup.py): finished with status 'error'
  Complete output from command /opt/Anaconda/envs/deepdiva/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-xor3fcig/pydensecrf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-w0gp68c6 --python-tag cp37:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/pydensecrf
  copying pydensecrf/test.py -> build/lib.linux-x86_64-3.7/pydensecrf
  copying pydensecrf/utils.py -> build/lib.linux-x86_64-3.7/pydensecrf
  copying pydensecrf/__init__.py -> build/lib.linux-x86_64-3.7/pydensecrf
  copying pydensecrf/test_eigen.py -> build/lib.linux-x86_64-3.7/pydensecrf
  running build_ext
  building 'pydensecrf.eigen' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/pydensecrf
  gcc -pthread -B /opt/Anaconda/envs/deepdiva/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ipydensecrf/densecrf/include -Ipydensecrf -I/opt/Anaconda/envs/deepdiva/include/python3.7m -c pydensecrf/eigen.cpp -o build/temp.linux-x86_64-3.7/pydensecrf/eigen.o
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  In file included from pydensecrf/densecrf/include/Eigen/Core:248:0,
                   from pydensecrf/densecrf/include/Eigen/Dense:1,
                   from pydensecrf/eigen.cpp:299:
  pydensecrf/densecrf/include/Eigen/src/Core/util/Memory.h: In function ‘Index Eigen::internal::first_aligned(const Scalar*, Index)’:
  pydensecrf/densecrf/include/Eigen/src/Core/util/Memory.h:454:48: warning: typedef ‘Packet’ locally defined but not used [-Wunused-local-typedefs]
     typedef typename packet_traits<Scalar>::type Packet;
                                                  ^
  In file included from pydensecrf/densecrf/include/Eigen/Core:324:0,
                   from pydensecrf/densecrf/include/Eigen/Dense:1,
                   from pydensecrf/eigen.cpp:299:
  pydensecrf/densecrf/include/Eigen/src/Core/products/SelfadjointMatrixVector.h: In static member function ‘static void Eigen::internal::selfadjoint_matrix_vector_product<Scalar, Index, StorageOrder, UpLo, ConjugateLhs, ConjugateRhs, Version>::run(Index, const Scalar*, Index, const Scalar*, Index, Scalar*, Scalar)’:
  pydensecrf/densecrf/include/Eigen/src/Core/products/SelfadjointMatrixVector.h:38:44: warning: typedef ‘RealScalar’ locally defined but not used [-Wunused-local-typedefs]
     typedef typename NumTraits<Scalar>::Real RealScalar;
                                              ^
  In file included from pydensecrf/densecrf/include/Eigen/QR:26:0,
                   from pydensecrf/densecrf/include/Eigen/Dense:4,
                   from pydensecrf/eigen.cpp:299:
  pydensecrf/densecrf/include/Eigen/src/QR/HouseholderQR.h: In function ‘void Eigen::internal::householder_qr_inplace_blocked(MatrixQR&, HCoeffs&, typename MatrixQR::Index, typename MatrixQR::Scalar*)’:
  pydensecrf/densecrf/include/Eigen/src/QR/HouseholderQR.h:235:41: warning: typedef ‘RealScalar’ locally defined but not used [-Wunused-local-typedefs]
     typedef typename MatrixQR::RealScalar RealScalar;
                                           ^
  In file included from pydensecrf/densecrf/include/Eigen/Eigenvalues:28:0,
                   from pydensecrf/densecrf/include/Eigen/Dense:7,
                   from pydensecrf/eigen.cpp:299:
  pydensecrf/densecrf/include/Eigen/src/Eigenvalues/Tridiagonalization.h: In function ‘void Eigen::internal::tridiagonalization_inplace(MatrixType&, DiagonalType&, SubDiagonalType&, bool)’:
  pydensecrf/densecrf/include/Eigen/src/Eigenvalues/Tridiagonalization.h:428:38: warning: typedef ‘Index’ locally defined but not used [-Wunused-local-typedefs]
     typedef typename MatrixType::Index Index;
                                        ^
  In file included from pydensecrf/densecrf/include/Eigen/Eigenvalues:34:0,
                   from pydensecrf/densecrf/include/Eigen/Dense:7,
                   from pydensecrf/eigen.cpp:299:
  pydensecrf/densecrf/include/Eigen/src/Eigenvalues/ComplexSchur.h: In static member function ‘static void Eigen::internal::complex_schur_reduce_to_hessenberg<MatrixType, false>::run(Eigen::ComplexSchur<MatrixType>&, const MatrixType&, bool)’:
  pydensecrf/densecrf/include/Eigen/src/Eigenvalues/ComplexSchur.h:312:66: warning: typedef ‘ComplexMatrixType’ locally defined but not used [-Wunused-local-typedefs]
       typedef typename ComplexSchur<MatrixType>::ComplexMatrixType ComplexMatrixType;
                                                                    ^
  pydensecrf/eigen.cpp: In function ‘void __Pyx__ExceptionSave(PyThreadState*, PyObject**, PyObject**, PyObject**)’:
  pydensecrf/eigen.cpp:17032:21: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
       *type = tstate->exc_type;
                       ^
  pydensecrf/eigen.cpp:17033:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
       *value = tstate->exc_value;
                        ^
  pydensecrf/eigen.cpp:17034:19: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
       *tb = tstate->exc_traceback;
                     ^
  pydensecrf/eigen.cpp: In function ‘void __Pyx__ExceptionReset(PyThreadState*, PyObject*, PyObject*, PyObject*)’:
  pydensecrf/eigen.cpp:17041:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                          ^
  pydensecrf/eigen.cpp:17042:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                           ^
  pydensecrf/eigen.cpp:17043:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                        ^
  pydensecrf/eigen.cpp:17044:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
       tstate->exc_type = type;
               ^
  pydensecrf/eigen.cpp:17045:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
       tstate->exc_value = value;
               ^
  pydensecrf/eigen.cpp:17046:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
       tstate->exc_traceback = tb;
               ^
  pydensecrf/eigen.cpp: In function ‘int __Pyx__GetException(PyThreadState*, PyObject**, PyObject**, PyObject**)’:
  pydensecrf/eigen.cpp:17101:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                          ^
  pydensecrf/eigen.cpp:17102:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                           ^
  pydensecrf/eigen.cpp:17103:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                        ^
  pydensecrf/eigen.cpp:17104:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
       tstate->exc_type = local_type;
               ^
  pydensecrf/eigen.cpp:17105:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
       tstate->exc_value = local_value;
               ^
  pydensecrf/eigen.cpp:17106:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
       tstate->exc_traceback = local_tb;
               ^
  pydensecrf/eigen.cpp: In function ‘void __Pyx__ExceptionSwap(PyThreadState*, PyObject**, PyObject**, PyObject**)’:
  pydensecrf/eigen.cpp:17128:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                          ^
  pydensecrf/eigen.cpp:17129:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                           ^
  pydensecrf/eigen.cpp:17130:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                        ^
  pydensecrf/eigen.cpp:17131:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
       tstate->exc_type = *type;
               ^
  pydensecrf/eigen.cpp:17132:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
       tstate->exc_value = *value;
               ^
  pydensecrf/eigen.cpp:17133:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
       tstate->exc_traceback = *tb;
               ^
  In file included from pydensecrf/eigen.cpp:300:0:
  pydensecrf/eigen_impl.cpp: At global scope:
  pydensecrf/eigen_impl.cpp:13:13: warning: ‘void vecf2buf(const VectorXf&, float*)’ defined but not used [-Wunused-function]
   static void vecf2buf(const Eigen::VectorXf& vec, float *mem)
               ^
  pydensecrf/eigen_impl.cpp:27:13: warning: ‘void matf2buf(const MatrixXf&, float*)’ defined but not used [-Wunused-function]
   static void matf2buf(const Eigen::MatrixXf& mat, float *mem)
               ^
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Running setup.py clean for pydensecrf
Failed to build pydensecrf
Installing collected packages: pydensecrf, rarfile
  Running setup.py install for pydensecrf: started
    Running setup.py install for pydensecrf: finished with status 'error'
    Complete output from command /opt/Anaconda/envs/deepdiva/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-xor3fcig/pydensecrf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-6a_kt9ca/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/pydensecrf
    copying pydensecrf/test.py -> build/lib.linux-x86_64-3.7/pydensecrf
    copying pydensecrf/utils.py -> build/lib.linux-x86_64-3.7/pydensecrf
    copying pydensecrf/__init__.py -> build/lib.linux-x86_64-3.7/pydensecrf
    copying pydensecrf/test_eigen.py -> build/lib.linux-x86_64-3.7/pydensecrf
    running build_ext
    building 'pydensecrf.eigen' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/pydensecrf
    gcc -pthread -B /opt/Anaconda/envs/deepdiva/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ipydensecrf/densecrf/include -Ipydensecrf -I/opt/Anaconda/envs/deepdiva/include/python3.7m -c pydensecrf/eigen.cpp -o build/temp.linux-x86_64-3.7/pydensecrf/eigen.o
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    In file included from pydensecrf/densecrf/include/Eigen/Core:248:0,
                     from pydensecrf/densecrf/include/Eigen/Dense:1,
                     from pydensecrf/eigen.cpp:299:
    pydensecrf/densecrf/include/Eigen/src/Core/util/Memory.h: In function ‘Index Eigen::internal::first_aligned(const Scalar*, Index)’:
    pydensecrf/densecrf/include/Eigen/src/Core/util/Memory.h:454:48: warning: typedef ‘Packet’ locally defined but not used [-Wunused-local-typedefs]
       typedef typename packet_traits<Scalar>::type Packet;
                                                    ^
    In file included from pydensecrf/densecrf/include/Eigen/Core:324:0,
                     from pydensecrf/densecrf/include/Eigen/Dense:1,
                     from pydensecrf/eigen.cpp:299:
    pydensecrf/densecrf/include/Eigen/src/Core/products/SelfadjointMatrixVector.h: In static member function ‘static void Eigen::internal::selfadjoint_matrix_vector_product<Scalar, Index, StorageOrder, UpLo, ConjugateLhs, ConjugateRhs, Version>::run(Index, const Scalar*, Index, const Scalar*, Index, Scalar*, Scalar)’:
    pydensecrf/densecrf/include/Eigen/src/Core/products/SelfadjointMatrixVector.h:38:44: warning: typedef ‘RealScalar’ locally defined but not used [-Wunused-local-typedefs]
       typedef typename NumTraits<Scalar>::Real RealScalar;
                                                ^
    In file included from pydensecrf/densecrf/include/Eigen/QR:26:0,
                     from pydensecrf/densecrf/include/Eigen/Dense:4,
                     from pydensecrf/eigen.cpp:299:
    pydensecrf/densecrf/include/Eigen/src/QR/HouseholderQR.h: In function ‘void Eigen::internal::householder_qr_inplace_blocked(MatrixQR&, HCoeffs&, typename MatrixQR::Index, typename MatrixQR::Scalar*)’:
    pydensecrf/densecrf/include/Eigen/src/QR/HouseholderQR.h:235:41: warning: typedef ‘RealScalar’ locally defined but not used [-Wunused-local-typedefs]
       typedef typename MatrixQR::RealScalar RealScalar;
                                             ^
    In file included from pydensecrf/densecrf/include/Eigen/Eigenvalues:28:0,
                     from pydensecrf/densecrf/include/Eigen/Dense:7,
                     from pydensecrf/eigen.cpp:299:
    pydensecrf/densecrf/include/Eigen/src/Eigenvalues/Tridiagonalization.h: In function ‘void Eigen::internal::tridiagonalization_inplace(MatrixType&, DiagonalType&, SubDiagonalType&, bool)’:
    pydensecrf/densecrf/include/Eigen/src/Eigenvalues/Tridiagonalization.h:428:38: warning: typedef ‘Index’ locally defined but not used [-Wunused-local-typedefs]
       typedef typename MatrixType::Index Index;
                                          ^
    In file included from pydensecrf/densecrf/include/Eigen/Eigenvalues:34:0,
                     from pydensecrf/densecrf/include/Eigen/Dense:7,
                     from pydensecrf/eigen.cpp:299:
    pydensecrf/densecrf/include/Eigen/src/Eigenvalues/ComplexSchur.h: In static member function ‘static void Eigen::internal::complex_schur_reduce_to_hessenberg<MatrixType, false>::run(Eigen::ComplexSchur<MatrixType>&, const MatrixType&, bool)’:
    pydensecrf/densecrf/include/Eigen/src/Eigenvalues/ComplexSchur.h:312:66: warning: typedef ‘ComplexMatrixType’ locally defined but not used [-Wunused-local-typedefs]
         typedef typename ComplexSchur<MatrixType>::ComplexMatrixType ComplexMatrixType;
                                                                      ^
    pydensecrf/eigen.cpp: In function ‘void __Pyx__ExceptionSave(PyThreadState*, PyObject**, PyObject**, PyObject**)’:
    pydensecrf/eigen.cpp:17032:21: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
         *type = tstate->exc_type;
                         ^
    pydensecrf/eigen.cpp:17033:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
         *value = tstate->exc_value;
                          ^
    pydensecrf/eigen.cpp:17034:19: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
         *tb = tstate->exc_traceback;
                       ^
    pydensecrf/eigen.cpp: In function ‘void __Pyx__ExceptionReset(PyThreadState*, PyObject*, PyObject*, PyObject*)’:
    pydensecrf/eigen.cpp:17041:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
         tmp_type = tstate->exc_type;
                            ^
    pydensecrf/eigen.cpp:17042:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
         tmp_value = tstate->exc_value;
                             ^
    pydensecrf/eigen.cpp:17043:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
         tmp_tb = tstate->exc_traceback;
                          ^
    pydensecrf/eigen.cpp:17044:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
         tstate->exc_type = type;
                 ^
    pydensecrf/eigen.cpp:17045:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
         tstate->exc_value = value;
                 ^
    pydensecrf/eigen.cpp:17046:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
         tstate->exc_traceback = tb;
                 ^
    pydensecrf/eigen.cpp: In function ‘int __Pyx__GetException(PyThreadState*, PyObject**, PyObject**, PyObject**)’:
    pydensecrf/eigen.cpp:17101:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
         tmp_type = tstate->exc_type;
                            ^
    pydensecrf/eigen.cpp:17102:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’
         tmp_value = tstate->exc_value;
                             ^
    pydensecrf/eigen.cpp:17103:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’
         tmp_tb = tstate->exc_traceback;
                          ^
    pydensecrf/eigen.cpp:17104:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’
         tstate->exc_type = local_type;
Setup completed!
Please run 'source ~/.bashrc' to refresh your environment
You can activate the deepdiva environment with 'source activate deepdiva'
(base) gusgru@B4375:~/Repositories/DeepDIVA$ source ~/.bashrc
(base) gusgru@B4375:~/Repositories/DeepDIVA$ source activate deepdiva
(deepdiva) gusgru@B4375:~/Repositories/DeepDIVA$ python util/data/get_a_dataset.py --dataset mnist --output-folder toy_dataset
Traceback (most recent call last):
  File "util/data/get_a_dataset.py", line 19, in <module>
    import rarfile
ModuleNotFoundError: No module named 'rarfile'

[LindaSt]

We will look into it. In the meanwhile, you can run pip install rarfile (and any other missing packages) to install missing packages and still use DeepDIVA.

[Renthal]

I merged the PR can you see if the problem persists?

[guspih]

The setup works now. Thanks