search

AutodeskAILab / BRepNet

BRepNet: A topological message passing system for solid models
Other
105 stars 31 forks source link

Possible broken dependencies with environment.yml #17

Closed samParry closed 1 year ago

samParry commented 1 year ago

Creating a conda environment from the .yml file seems to install versions of packages that no longer work together with the BRepNet source code. I've followed the installation instructions on several machines and get different error messages on each machine relating to broken dependencies in the site packages installed from the .yml. Can someone verify the installation or update the .yml to specify the package versions that worked correctly when the paper was published?

JoeLambourne commented 1 year ago

Hi @samParry,

Yes. It was working fine, but towards the end of last year we started to have issues with the occwl package. I've been needing to install that with mamba. Let me see what I can do to get BRepNet working again

JoeLambourne commented 1 year ago

Hi @samParry,

Is the problem you are having related to the command 

conda env create -f environment.yml

just hanging and never installing anything? I'm having trouble with this now, but I want to verify you are seeing a similar issue or something different?

samParry commented 1 year ago

Hi @JoeLambourne Yes, I've had issues with very long wait times up to several hours. I also getting site package errors when running the training commands from the Training the model section of the ReadMe. I found that you have to set pytorch-lightning=1.6.4 to get the model to train.

conda install -c conda-forge pytorch-lightning=1.6.4

My current environment works on my MacOS environment with conda 23.1.0 

name: brepnet
channels:
  - conda-forge
  - defaults
  - lambouj
dependencies:
  - python=3.7
  - pythonocc-core=7.5.1
  - pytorch=1.12.1
  - tqdm=4.65.0
  - igl=2.2.1
  - scikit-learn=1.0.2
  - pytorch-lightning=1.6.4
  - xlsxwriter=3.0.9
  - occwl=1.0.0
  - jupyter=1.0.0
  - pythreejs=2.4.1
  - tensorboard=2.11.2

Although, on one of my linux machines I got an OSError

Traceback (most recent call last):
  File "/home/deck/anaconda3/envs/brepnet/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
  File "/home/deck/anaconda3/envs/brepnet/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
  File "/home/deck/anaconda3/envs/brepnet/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 359, in reduce_storage
  File "/home/deck/anaconda3/envs/brepnet/lib/python3.7/multiprocessing/reduction.py", line 194, in DupFd
  File "/home/deck/anaconda3/envs/brepnet/lib/python3.7/multiprocessing/resource_sharer.py", line 48, in __init__
OSError: [Errno 24] Too many open files
JoeLambourne commented 1 year ago

Hi @samParry,

I have a copy of the environment which was working at the time of PR#13. For reference the package versions are here

name: brepnet
channels:
  - pytorch
  - lambouj
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - _pytorch_select=0.1=cpu_0
  - absl-py=1.0.0=pyhd8ed1ab_0
  - aiohttp=3.8.1=py37h540881e_1
  - aiosignal=1.2.0=pyhd8ed1ab_0
  - alsa-lib=1.2.3.2=h166bdaf_0
  - argcomplete=2.0.0=pyhd8ed1ab_0
  - argon2-cffi=21.3.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py37h540881e_2
  - async-timeout=4.0.2=pyhd8ed1ab_0
  - asynctest=0.13.0=py_0
  - atk-1.0=2.36.0=h3371d22_4
  - attr=2.5.1=h166bdaf_0
  - attrs=21.4.0=pyhd8ed1ab_0
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
  - beautifulsoup4=4.11.1=pyha770c72_0
  - blas=1.0=mkl
  - bleach=5.0.0=pyhd8ed1ab_0
  - blinker=1.4=py_1
  - brotli=1.0.9=h166bdaf_7
  - brotli-bin=1.0.9=h166bdaf_7
  - brotlipy=0.7.0=py37h540881e_1004
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.18.1=h7f98852_0
  - ca-certificates=2022.5.18.1=ha878542_0
  - cachetools=5.0.0=pyhd8ed1ab_0
  - cairo=1.16.0=ha61ee94_1011
  - certifi=2022.5.18.1=py37h89c1867_0
  - cffi=1.15.0=py37h036bc23_0
  - charset-normalizer=2.0.12=pyhd8ed1ab_0
  - click=8.1.3=py37h89c1867_0
  - colorama=0.4.4=pyh9f0ad1d_0
  - cryptography=37.0.2=py37h38fbfac_0
  - cudatoolkit=11.0.3=h88f8997_10
  - curl=7.83.1=h7bff187_0
  - cycler=0.11.0=pyhd8ed1ab_0
  - dbus=1.13.6=h5008d03_3
  - debugpy=1.6.0=py37hd23a5d3_0
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - double-conversion=3.1.7=h9c3ff4c_0
  - eigen=3.4.0=h4bd325d_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - expat=2.4.8=h27087fc_0
  - ffmpeg=4.3.2=h37c90e5_3
  - fftw=3.3.10=nompi_h77c792f_102
  - flit-core=3.7.1=pyhd8ed1ab_0
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=hab24e00_0
  - fontconfig=2.14.0=h8e229c2_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.33.3=py37h540881e_0
  - freeimage=3.18.0=hf18588b_8
  - freetype=2.10.4=h0708190_1
  - fribidi=1.0.10=h36c2ea0_0
  - frozenlist=1.3.0=py37h540881e_1
  - fsspec=2022.5.0=pyhd8ed1ab_0
  - future=0.18.2=py37h89c1867_5
  - gdk-pixbuf=2.42.8=hff1cb4f_0
  - gettext=0.19.8.1=h73d1719_1008
  - giflib=5.2.1=h36c2ea0_2
  - gl2ps=1.4.2=h0708190_0
  - glew=2.1.0=h9c3ff4c_2
  - glib=2.70.2=h780b84a_4
  - glib-tools=2.70.2=h780b84a_4
  - gmp=6.2.1=h58526e2_0
  - gnutls=3.6.13=h85f3911_1
  - google-auth=2.6.6=pyh6c4a22f_0
  - google-auth-oauthlib=0.4.6=pyhd8ed1ab_0
  - graphite2=1.3.13=h58526e2_1001
  - grpcio=1.46.3=py37he500948_0
  - gst-plugins-base=1.20.2=hcf0ee16_0
  - gstreamer=1.20.2=hd4edc92_1
  - gtk2=2.24.33=h90689f9_2
  - harfbuzz=4.3.0=hf9f4e7c_0
  - hdf4=4.2.15=h10796ff_3
  - hdf5=1.10.6=nompi_h6a2412b_1114
  - icu=70.1=h27087fc_0
  - idna=3.3=pyhd8ed1ab_0
  - igl=2.2.1=py37h9d0810c_1
  - imath=3.1.5=h6239696_0
  - importlib-metadata=4.11.4=py37h89c1867_0
  - importlib_metadata=4.11.4=hd8ed1ab_0
  - importlib_resources=5.7.1=pyhd8ed1ab_1
  - intel-openmp=2022.0.1=h06a4308_3633
  - ipydatawidgets=4.3.1.post1=pyhc268e32_0
  - ipykernel=6.3.1=py37h6531663_0
  - ipython=7.33.0=py37h89c1867_0
  - ipython_genutils=0.2.0=py_1
  - ipywidgets=7.7.0=pyhd8ed1ab_0
  - jack=1.9.18=hfd4fe87_1001
  - jedi=0.18.1=py37h89c1867_1
  - jinja2=3.1.2=pyhd8ed1ab_0
  - joblib=1.1.0=pyhd8ed1ab_0
  - jpeg=9e=h166bdaf_1
  - json-c=0.15=h98cffda_0
  - jsoncpp=1.9.4=h4bd325d_3
  - jsonschema=4.5.1=pyhd8ed1ab_0
  - jupyter=1.0.0=py37h89c1867_7
  - jupyter_client=7.3.1=pyhd8ed1ab_0
  - jupyter_console=6.4.3=pyhd8ed1ab_0
  - jupyter_core=4.10.0=py37h89c1867_0
  - jupyterlab_pygments=0.2.2=pyhd8ed1ab_0
  - jupyterlab_widgets=1.1.0=pyhd8ed1ab_0
  - jxrlib=1.1=h7f98852_2
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.2=py37h7cecad7_1
  - krb5=1.19.3=h3790be6_0
  - lame=3.100=h7f98852_1001
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=3.0=h9c3ff4c_0
  - libblas=3.9.0=14_linux64_mkl
  - libbrotlicommon=1.0.9=h166bdaf_7
  - libbrotlidec=1.0.9=h166bdaf_7
  - libbrotlienc=1.0.9=h166bdaf_7
  - libcap=2.51=h166bdaf_1
  - libcblas=3.9.0=14_linux64_mkl
  - libclang=14.0.3=default_h2e3cab8_0
  - libclang13=14.0.3=default_h3a83d3e_0
  - libcups=2.3.3=hf5a7f15_1
  - libcurl=7.83.1=h7bff187_0
  - libdb=6.2.32=h9c3ff4c_0
  - libdeflate=1.10=h7f98852_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=h516909a_1
  - libevent=2.1.10=h9b69904_4
  - libffi=3.4.2=h7f98852_5
  - libflac=1.3.4=h27087fc_0
  - libgcc-ng=12.1.0=h8d9b700_16
  - libgfortran-ng=12.1.0=h69a702a_16
  - libgfortran5=12.1.0=hdcd56e2_16
  - libglib=2.70.2=h174f98d_4
  - libglu=9.0.0=he1b5a44_1001
  - libgomp=12.1.0=h8d9b700_16
  - libiconv=1.16=h516909a_0
  - liblapack=3.9.0=14_linux64_mkl
  - libllvm14=14.0.3=he0ac6c6_0
  - libnetcdf=4.8.0=nompi_hcd642e3_103
  - libnghttp2=1.47.0=h727a467_0
  - libnsl=2.0.0=h7f98852_0
  - libogg=1.3.4=h7f98852_1
  - libopus=1.3.1=h7f98852_1
  - libpng=1.6.37=h21135ba_2
  - libpq=14.3=hd77ab85_0
  - libprotobuf=3.20.1=h6239696_0
  - libraw=0.20.2=h10796ff_1
  - libsndfile=1.0.31=h9c3ff4c_1
  - libsodium=1.0.18=h36c2ea0_1
  - libssh2=1.10.0=ha56f1ee_2
  - libstdcxx-ng=12.1.0=ha89aaad_16
  - libtheora=1.1.1=h7f98852_1005
  - libtiff=4.3.0=h0fcbabc_4
  - libtool=2.4.6=h9c3ff4c_1008
  - libuuid=2.32.1=h7f98852_1000
  - libuv=1.43.0=h7f98852_0
  - libvorbis=1.3.7=h9c3ff4c_0
  - libwebp=1.2.2=h3452ae3_0
  - libwebp-base=1.2.2=h7f98852_1
  - libxcb=1.13=h7f98852_1004
  - libxkbcommon=1.0.3=he3ba5ed_0
  - libxml2=2.9.14=h22db469_0
  - libxslt=1.1.33=h8affb1d_4
  - libzip=1.8.0=h4de3113_1
  - libzlib=1.2.11=h166bdaf_1014
  - loguru=0.6.0=py37h89c1867_1
  - lz4-c=1.9.3=h9c3ff4c_1
  - markdown=3.3.7=pyhd8ed1ab_0
  - markupsafe=2.1.1=py37h540881e_1
  - matplotlib=3.5.2=py37h89c1867_0
  - matplotlib-base=3.5.2=py37hc347a89_0
  - matplotlib-inline=0.1.3=pyhd8ed1ab_0
  - mistune=0.8.4=py37h5e8e339_1005
  - mkl=2022.0.1=h06a4308_117
  - multidict=6.0.2=py37h540881e_1
  - munkres=1.1.4=pyh9f0ad1d_0
  - mysql-common=8.0.29=haf5c9bc_1
  - mysql-libs=8.0.29=h28c427c_1
  - nbclient=0.6.3=pyhd8ed1ab_0
  - nbconvert=6.5.0=pyhd8ed1ab_0
  - nbconvert-core=6.5.0=pyhd8ed1ab_0
  - nbconvert-pandoc=6.5.0=pyhd8ed1ab_0
  - nbformat=5.4.0=pyhd8ed1ab_0
  - ncurses=6.3=h27087fc_1
  - nest-asyncio=1.5.5=pyhd8ed1ab_0
  - nettle=3.6=he412f7d_0
  - networkx=2.7.1=pyhd8ed1ab_0
  - ninja=1.11.0=h924138e_0
  - notebook=6.4.11=pyha770c72_0
  - nspr=4.32=h9c3ff4c_1
  - nss=3.78=h2350873_0
  - numpy=1.21.6=py37h976b520_0
  - oauthlib=3.2.0=pyhd8ed1ab_0
  - occt=7.5.1=h7391655_2
  - occwl=1.0.0=py_0
  - openexr=3.1.5=he0ac6c6_0
  - openh264=2.1.1=h780b84a_0
  - openjpeg=2.4.0=hb52868f_1
  - openssl=1.1.1o=h166bdaf_0
  - packaging=21.3=pyhd8ed1ab_0
  - pandas=1.3.5=py37he8f5f7f_0
  - pandoc=2.18=ha770c72_0
  - pandocfilters=1.5.0=pyhd8ed1ab_0
  - pango=1.50.7=hbd2fdc8_0
  - parso=0.8.3=pyhd8ed1ab_0
  - pathlib2=2.3.7.post1=py37h89c1867_1
  - pcre=8.45=h9c3ff4c_0
  - pexpect=4.8.0=pyh9f0ad1d_2
  - pickleshare=0.7.5=py_1003
  - pillow=9.1.1=py37h44f0d7a_0
  - pip=22.1.1=pyhd8ed1ab_0
  - pixman=0.40.0=h36c2ea0_0
  - proj=7.2.0=h277dcde_2
  - prometheus_client=0.14.1=pyhd8ed1ab_0
  - prompt-toolkit=3.0.29=pyha770c72_0
  - prompt_toolkit=3.0.29=hd8ed1ab_0
  - protobuf=3.20.1=py37hd23a5d3_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pugixml=1.11.4=h9c3ff4c_0
  - pulseaudio=14.0=hb166930_4
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.7=py_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pydeprecate=0.3.2=pyhd8ed1ab_0
  - pygments=2.12.0=pyhd8ed1ab_0
  - pyjwt=2.4.0=pyhd8ed1ab_0
  - pyopenssl=22.0.0=pyhd8ed1ab_0
  - pyparsing=3.0.9=pyhd8ed1ab_0
  - pypubsub=4.0.3=py_0
  - pyqt=5.15.4=py37hd23a5d3_0
  - pyqt5-sip=12.9.0=py37hd23a5d3_0
  - pyrsistent=0.18.1=py37h540881e_1
  - pyside2=5.15.4=py37h76b8631_0
  - pysocks=1.7.1=py37h89c1867_5
  - python=3.7.12=hb7a2778_100_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-fastjsonschema=2.15.3=pyhd8ed1ab_0
  - python_abi=3.7=2_cp37m
  - pythonocc-core=7.5.1=py37h8ede140_1
  - pythreejs=2.3.0=pyhd8ed1ab_0
  - pytorch=1.7.1=py3.7_cuda11.0.221_cudnn8.0.5_0
  - pytorch-lightning=1.5.10=pyhd8ed1ab_0
  - pytz=2022.1=pyhd8ed1ab_0
  - pyu2f=0.1.5=pyhd8ed1ab_0
  - pyyaml=6.0=py37h540881e_4
  - pyzmq=23.0.0=py37h0c0c2a8_0
  - qt-main=5.15.3=hf97cb25_1
  - qt-webengine=5.15.4=hcbadb6c_3
  - qtconsole=5.3.0=pyhd8ed1ab_0
  - qtconsole-base=5.3.0=pyha770c72_0
  - qtpy=2.1.0=pyhd8ed1ab_0
  - rapidjson=1.1.0=he1b5a44_1002
  - readline=8.1=h46c0cb4_0
  - requests=2.27.1=pyhd8ed1ab_0
  - requests-oauthlib=1.3.1=pyhd8ed1ab_0
  - rsa=4.8=pyhd8ed1ab_0
  - scikit-learn=1.0.2=py37hf9e9bfc_0
  - scipy=1.7.3=py37hf2a6cf1_0
  - send2trash=1.8.0=pyhd8ed1ab_0
  - setuptools=59.5.0=py37h89c1867_0
  - sip=6.5.1=py37hcd2ae1e_2
  - six=1.16.0=pyh6c4a22f_0
  - sleef=3.5.1=h9b69904_2
  - soupsieve=2.3.1=pyhd8ed1ab_0
  - sqlite=3.38.5=h4ff8645_0
  - tbb=2020.2=h4bd325d_4
  - tbb-devel=2020.2=h4bd325d_4
  - tensorboard=2.9.0=pyhd8ed1ab_0
  - tensorboard-data-server=0.6.0=py37h38fbfac_2
  - tensorboard-plugin-wit=1.8.1=pyhd8ed1ab_0
  - terminado=0.15.0=py37h89c1867_0
  - threadpoolctl=3.1.0=pyh8a188c0_0
  - tinycss2=1.1.1=pyhd8ed1ab_0
  - tk=8.6.12=h27826a3_0
  - toml=0.10.2=pyhd8ed1ab_0
  - torchaudio=0.7.2=py37
  - torchmetrics=0.8.2=pyhd8ed1ab_0
  - torchvision=0.8.2=cpu_py37ha229d99_0
  - tornado=6.1=py37h540881e_3
  - tqdm=4.64.0=pyhd8ed1ab_0
  - traitlets=5.2.1.post0=pyhd8ed1ab_0
  - traittypes=0.2.1=pyh9f0ad1d_2
  - typing-extensions=4.2.0=hd8ed1ab_1
  - typing_extensions=4.2.0=pyha770c72_1
  - unicodedata2=14.0.0=py37h540881e_1
  - urllib3=1.26.9=pyhd8ed1ab_0
  - utfcpp=3.2.1=ha770c72_0
  - vtk=9.0.1=no_osmesa_py37h111716f_107
  - wcwidth=0.2.5=pyh9f0ad1d_2
  - webencodings=0.5.1=py_1
  - werkzeug=2.1.2=pyhd8ed1ab_1
  - wheel=0.37.1=pyhd8ed1ab_0
  - widgetsnbextension=3.6.0=py37h89c1867_0
  - wxpython=4.1.1=py37hecb6014_7
  - x264=1!161.3030=h7f98852_1
  - xlsxwriter=3.0.3=pyhd8ed1ab_0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.0.10=h7f98852_0
  - xorg-libsm=1.2.3=hd9c2040_1000
  - xorg-libx11=1.7.2=h7f98852_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h7f98852_1
  - xorg-libxrender=0.9.10=h7f98852_1003
  - xorg-libxt=1.2.1=h7f98852_2
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h7f98852_1002
  - xorg-xproto=7.0.31=h7f98852_1007
  - xz=5.2.5=h516909a_1
  - yaml=0.2.5=h7f98852_2
  - yarl=1.7.2=py37h540881e_2
  - zeromq=4.3.4=h9c3ff4c_1
  - zipp=3.8.0=pyhd8ed1ab_0
  - zlib=1.2.11=h166bdaf_1014
  - zstd=1.5.2=h8a70e8d_1

I'm trying to use these version to create a locked down environment.yml as you suggested, but because of the long time making new environments it's been hard to make progress.

As part of this process I removed a lot of stuff which I had (accidentally) installed in my base environment. It looks like that helped speed things up

JoeLambourne commented 1 year ago

I am now able to build the environment with the following environment.yml file

name: brepnettest
channels:
  - conda-forge
  - defaults
  - lambouj
dependencies:
  - python=3.7
  - pythonocc-core=7.5.1
  - pytorch=1.7.1
  - tqdm=4.64.0
  - igl=2.2.1
  - scikit-learn=1.0.2
  - pytorch-lightning=1.5.10
  - xlsxwriter=3.0.3
  - occwl=1.0.0
  - jupyter=1.0.0
  - pythreejs=2.3.0
  - tensorboard=2.9.0

I'm now testing this works. The data processing pipeline appears to be working with the segmentation dataset version s2.0.1. This doesn't contain multiple bodies in a single file though.

JoeLambourne commented 1 year ago

@samParry

OSError: [Errno 24] Too many open files

I get that sometimes when using many threads. I have double checked files opened by the dataloader do get closed. The workaround is setting

ulimit -Sn 10000
JoeLambourne commented 1 year ago

In PR#18 I have fixed all the version in environment.yml. Hopefully this should make it possible to reproduce the results in the paper going forward.