search

pschanely / CrossHair

An analysis tool for Python that blurs the line between testing and type systems.
Other
1.03k stars 49 forks source link

Importing spacy fails #172

Closed rasenmaeher92 closed 2 years ago

rasenmaeher92 commented 2 years ago

Expected vs actual behavior In similar vain to #159 I found another package which does not import properly: spacy.

To Reproduce Just including import spacy produces this error for crosshair watch:

Could not import your code:

Traceback (most recent call last):
  File "/path/to/my/conda/env/lib/python3.9/site-packages/crosshair/util.py", line 375, in load_file
    return import_module(module_name)
  File "/path/to/my/conda/env/lib/python3.9/site-packages/crosshair/util.py", line 343, in import_module
    result_module = importlib.import_module(module_name)
  File "/path/to/my/conda/env/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/mnt/macx/data/datasets/plan_extraction.py", line 4, in <module>
    import spacy
  File "/path/to/my/conda/env/lib/python3.9/site-packages/spacy/__init__.py", line 11, in <module>
    from thinc.api import prefer_gpu, require_gpu, require_cpu  # noqa: F401
  File "/path/to/my/conda/env/lib/python3.9/site-packages/thinc/api.py", line 2, in <module>
    from .initializers import normal_init, uniform_init, glorot_uniform_init, zero_init
  File "/path/to/my/conda/env/lib/python3.9/site-packages/thinc/initializers.py", line 4, in <module>
    from .backends import Ops
  File "/path/to/my/conda/env/lib/python3.9/site-packages/thinc/backends/__init__.py", line 7, in <module>
    from .ops import Ops
  File "/path/to/my/conda/env/lib/python3.9/site-packages/thinc/backends/ops.py", line 13, in <module>
    from ..util import get_array_module, is_xp_array, to_numpy
  File "/path/to/my/conda/env/lib/python3.9/site-packages/thinc/util.py", line 48, in <module>
    import tensorflow.experimental.dlpack
  File "/path/to/my/conda/env/lib/python3.9/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/path/to/my/conda/env/lib/python3.9/site-packages/tensorflow/python/__init__.py", line 104, in <module>
    from tensorflow.python.platform import test
  File "/path/to/my/conda/env/lib/python3.9/site-packages/tensorflow/python/platform/test.py", line 20, in <module>
    from tensorflow.python.framework import test_util as _test_util
  File "/path/to/my/conda/env/lib/python3.9/site-packages/tensorflow/python/framework/test_util.py", line 33, in <module>
    from absl.testing import parameterized
  File "/path/to/my/conda/env/lib/python3.9/site-packages/absl/testing/parameterized.py", line 218, in <module>
    from absl.testing import absltest
  File "/path/to/my/conda/env/lib/python3.9/site-packages/absl/testing/absltest.py", line 242, in <module>
    get_default_test_tmpdir(),
  File "/path/to/my/conda/env/lib/python3.9/site-packages/absl/testing/absltest.py", line 180, in get_default_test_tmpdir
    tmpdir = os.path.join(tempfile.gettempdir(), 'absl_testing')
  File "/path/to/my/conda/env/lib/python3.9/tempfile.py", line 287, in gettempdir
    tempdir = _get_default_tempdir()
  File "/path/to/my/conda/env/lib/python3.9/tempfile.py", line 198, in _get_default_tempdir
    fd = _os.open(filename, _bin_openflags, 0o600)
  File "/path/to/my/conda/env/lib/python3.9/site-packages/crosshair/auditwall.py", line 145, in audithook
    handler(event, args)
  File "/path/to/my/conda/env/lib/python3.9/site-packages/crosshair/auditwall.py", line 44, in check_open
    raise SideEffectDetected(
crosshair.auditwall.SideEffectDetected: We've blocked a file writing operation on "/tmp/znr8t36b". CrossHair should not be run on code with side effects
pschanely commented 2 years ago

Thank you for the report!!

I think it's more than reasonable to disable side-effect detection during module import - I will do this. (it has been surprising to me to see what python packages are willing to do at import time!)

NOTE: iit would not surprise me to see tensorflow lazily downloading data at runtime either, which would not addressed by my fix. But such behavior would likely also run afoul of nondeterminism checks.

I'll update this again once I cut 0.0.27 with a fix. Cheers!

rasenmaeher92 commented 2 years ago

Yes, lazy downloads of pre-trained models are a common pattern in machine learning libraries. However, as far as I know, the user usually specifies which model is to be downloaded.

For spacy this would be something like:

nlp = spacy.load("en_core_web_sm")

after installing it with

pip install -U spacy
python -m spacy download en_core_web_sm

Btw, often times I actually do not want to test spacy/pandas/etc. code itself, but other code, e.g. data manipulation. Thus, your import fixes are greatly welcomed!

pschanely commented 2 years ago

Yeah, while there are plenty of sensible limits to put on code under analysis, I'd certainly expect anything and everything to be import-able.

Now released in 0.0.27 - please reopen if it doesn't fix your situation!

rasenmaeher92 commented 2 years ago

Unfortunately it does not fix the problem :(.

If a demo.py file contains

import spacy

and I run python -m crosshair check demo.py

I get this error:

2022-07-13 17:17:54.715318: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-07-13 17:17:54.715356: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/mnt/miniconda3/envs/mlcat6/lib/python3.9/importlib/__init__.py:169: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
  _bootstrap._exec(spec, module)
terminate called after throwing an instance of 'std::runtime_error'
  what():  generic_type: cannot initialize type "GradBucket": an object with that name is already defined
[1]    20175 abort (core dumped)  python -m crosshair check demo.py

I assume you may have done a similar check, so maybe it is related to another dependency. Here is my environment:

name: base
channels:
  - pytorch
  - huggingface
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - abseil-cpp=20210324.2=h2531618_0
  - absl-py=0.15.0=pyhd3eb1b0_0
  - aiohttp=3.8.1=py39h7f8727e_0
  - aiosignal=1.2.0=pyhd3eb1b0_0
  - anyio=2.2.0=py39h06a4308_1
  - appdirs=1.4.4=pyhd3eb1b0_0
  - argon2-cffi=20.1.0=py39h27cfd23_1
  - arrow-cpp=4.0.1=py39he097a17_26_cpu
  - astroid=2.9.0=py39h06a4308_0
  - async-timeout=4.0.1=pyhd3eb1b0_0
  - async_generator=1.10=pyhd3eb1b0_0
  - attrs=21.4.0=pyhd3eb1b0_0
  - aws-c-cal=0.5.11=h95a6274_0
  - aws-c-common=0.6.2=h27cfd23_0
  - aws-c-event-stream=0.2.7=h3541f99_13
  - aws-c-io=0.10.5=hfb6a706_0
  - aws-checksums=0.1.11=ha31a3da_7
  - aws-sdk-cpp=1.8.186=hb4091e7_3
  - babel=2.9.1=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - bcj-cffi=0.5.1=py39h295c915_0
  - beautifulsoup4=4.10.0=pyh06a4308_0
  - bert_score=0.3.9=pyh44b312d_0
  - black=19.10b0=py_0
  - blas=1.0=mkl
  - bleach=4.1.0=pyhd3eb1b0_0
  - blinker=1.4=py39h06a4308_0
  - boost-cpp=1.71.0=h7f8727e_1
  - boto3=1.20.24=pyhd3eb1b0_0
  - botocore=1.23.24=pyhd3eb1b0_0
  - bottleneck=1.3.2=py39hdd57654_1
  - brotli=1.0.9=he6710b0_2
  - brotli-python=1.0.9=py39he80948d_6
  - brotlicffi=1.0.9.2=py39h295c915_0
  - brotlipy=0.7.0=py39h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.18.1=h7f8727e_0
  - ca-certificates=2022.6.15=ha878542_0
  - cachetools=4.2.2=pyhd3eb1b0_0
  - catalogue=2.0.6=py39hf3d152e_2
  - certifi=2022.6.15=py39hf3d152e_0
  - cffi=1.15.0=py39hd667e15_1
  - charset-normalizer=2.0.4=pyhd3eb1b0_0
  - click=8.0.3=pyhd3eb1b0_0
  - click-default-group=1.2.2=pyhd8ed1ab_1
  - cloudpickle=2.0.0=pyhd3eb1b0_0
  - colorama=0.4.4=pyhd3eb1b0_0
  - commonmark=0.9.1=pyhd3eb1b0_0
  - conda=4.12.0=py39h06a4308_0
  - conda-lock=0.13.2=pyhd8ed1ab_0
  - conda-package-handling=1.7.3=py39h27cfd23_1
  - conllu=4.4.1=pyhd3eb1b0_0
  - cryptography=36.0.0=py39h9ce1e76_0
  - cudatoolkit=10.2.89=hfd86e86_1
  - cycler=0.11.0=pyhd3eb1b0_0
  - cymem=2.0.6=py39h5a03fae_3
  - cython=0.29.25=py39hdbfa776_0
  - cython-blis=0.7.8=py39hd257fcd_0
  - cytoolz=0.11.0=py39h27cfd23_0
  - daal4py=2021.5.0=py39h78b71dc_0
  - dal=2021.5.1=h06a4308_803
  - dask-core=2022.2.1=pyhd3eb1b0_0
  - dataclasses=0.8=pyh6d0b6a4_7
  - datasets=1.18.3=py_0
  - dbus=1.13.18=hb2f20db_0
  - debugpy=1.5.1=py39h295c915_0
  - decorator=5.1.1=pyhd3eb1b0_0
  - defusedxml=0.7.1=pyhd3eb1b0_0
  - dill=0.3.4=pyhd3eb1b0_0
  - ensureconda=1.4.1=pyhd8ed1ab_0
  - entrypoints=0.3=py39h06a4308_0
  - et_xmlfile=1.1.0=py39h06a4308_0
  - expat=2.4.1=h2531618_2
  - ffmpeg=4.2.2=h20bf706_0
  - filelock=3.4.2=pyhd3eb1b0_0
  - fontconfig=2.13.1=h6c09931_0
  - fonttools=4.25.0=pyhd3eb1b0_0
  - freetype=2.11.0=h70c0345_0
  - frozenlist=1.2.0=py39h7f8727e_0
  - fsspec=2022.1.0=pyhd3eb1b0_0
  - future=0.18.2=py39h06a4308_1
  - fuzzywuzzy=0.18.0=pyhd8ed1ab_0
  - gdown=4.4.0=pyhd8ed1ab_0
  - gensim=4.1.2=py39h295c915_0
  - gflags=2.2.2=he6710b0_0
  - giflib=5.2.1=h7b6447c_0
  - glib=2.69.1=h4ff587b_1
  - glog=0.5.0=h2531618_0
  - gmp=6.2.1=h2531618_2
  - gnutls=3.6.15=he1e5248_0
  - google-api-core=1.25.1=pyhd3eb1b0_0
  - google-auth=1.33.0=pyhd3eb1b0_0
  - google-auth-oauthlib=0.4.1=py_2
  - google-cloud-core=1.7.1=pyhd3eb1b0_0
  - google-cloud-storage=1.31.0=py_0
  - google-crc32c=1.1.2=py39h27cfd23_0
  - google-resumable-media=1.3.1=pyhd3eb1b0_1
  - googleapis-common-protos=1.53.0=py39h06a4308_0
  - grpc-cpp=1.42.0=ha1441d3_1
  - grpcio=1.42.0=py39hce63b2e_0
  - gst-plugins-base=1.14.0=h8213a91_2
  - gstreamer=1.14.0=h28cd5cc_2
  - h5py=3.6.0=py39ha0f2276_0
  - hdf5=1.10.6=hb1b8bf9_0
  - huggingface_hub=0.2.1=pyhd3eb1b0_0
  - icu=58.2=he6710b0_3
  - idna=3.3=pyhd3eb1b0_0
  - imageio=2.9.0=pyhd3eb1b0_0
  - importlib-metadata=4.8.2=py39h06a4308_0
  - importlib_metadata=4.8.2=hd3eb1b0_0
  - iniconfig=1.1.1=pyhd3eb1b0_0
  - intel-openmp=2021.4.0=h06a4308_3561
  - ipykernel=6.4.1=py39h06a4308_1
  - ipython=7.31.1=py39h06a4308_0
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - ipywidgets=7.6.5=pyhd3eb1b0_1
  - isort=5.9.3=pyhd3eb1b0_0
  - jbig=2.1=hdba287a_0
  - jedi=0.18.1=py39h06a4308_0
  - jinja2=3.0.2=pyhd3eb1b0_0
  - jmespath=0.10.0=pyhd3eb1b0_0
  - joblib=1.1.0=pyhd3eb1b0_0
  - jpeg=9d=h7f8727e_0
  - json5=0.9.6=pyhd3eb1b0_0
  - jsonschema=3.2.0=pyhd3eb1b0_2
  - jupyter=1.0.0=py39h06a4308_7
  - jupyter_client=7.1.2=pyhd3eb1b0_0
  - jupyter_console=6.4.0=pyhd3eb1b0_0
  - jupyter_core=4.9.1=py39h06a4308_0
  - jupyter_server=1.4.1=py39h06a4308_0
  - jupyterlab=3.2.9=pyhd3eb1b0_0
  - jupyterlab_pygments=0.1.2=py_0
  - jupyterlab_server=2.10.2=pyhd3eb1b0_1
  - jupyterlab_widgets=1.0.0=pyhd3eb1b0_1
  - kiwisolver=1.3.1=py39h2531618_0
  - krb5=1.19.2=hac12032_0
  - lame=3.100=h7b6447c_0
  - langcodes=3.3.0=pyhd8ed1ab_0
  - lazy-object-proxy=1.6.0=py39h27cfd23_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - lerc=3.0=h295c915_0
  - libarchive=3.5.2=hccf745f_1
  - libboost=1.71.0=haf77d95_1
  - libbrotlicommon=1.0.9=h7f98852_6
  - libbrotlidec=1.0.9=h7f98852_6
  - libbrotlienc=1.0.9=h7f98852_6
  - libcrc32c=1.1.1=he6710b0_2
  - libcurl=7.81.0=h2574ce0_0
  - libdeflate=1.8=h7f8727e_5
  - libedit=3.1.20210910=h7f8727e_0
  - libev=4.33=h7f8727e_1
  - libevent=2.1.12=h8f2d780_0
  - libffi=3.3=he6710b0_2
  - libgcc-ng=12.1.0=h8d9b700_16
  - libgfortran-ng=7.5.0=ha8ba4b0_17
  - libgfortran4=7.5.0=ha8ba4b0_17
  - libgomp=12.1.0=h8d9b700_16
  - libiconv=1.16=h516909a_0
  - libidn2=2.3.2=h7f8727e_0
  - libmamba=0.22.1=h9c208ef_0
  - libmambapy=0.22.1=py39h3216e65_0
  - libnghttp2=1.46.0=hce63b2e_0
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.37=hbc83047_0
  - libprotobuf=3.19.1=h4ff587b_0
  - libsodium=1.0.18=h7b6447c_0
  - libsolv=0.7.19=h780b84a_5
  - libssh2=1.10.0=ha56f1ee_2
  - libstdcxx-ng=12.1.0=ha89aaad_16
  - libtasn1=4.16.0=h27cfd23_0
  - libthrift=0.15.0=hcc01f38_0
  - libtiff=4.3.0=h6f004c6_2
  - libunistring=0.9.10=h27cfd23_0
  - libutf8proc=2.7.0=h7f98852_0
  - libuuid=1.0.3=h7f8727e_2
  - libuv=1.40.0=h7b6447c_0
  - libvpx=1.7.0=h439df22_0
  - libwebp=1.2.0=h89dd481_0
  - libwebp-base=1.2.0=h27cfd23_0
  - libxcb=1.14=h7b6447c_0
  - libxml2=2.9.12=h03d6c58_0
  - libxslt=1.1.34=hc22bd24_0
  - libzlib=1.2.11=h36c2ea0_1013
  - lmdb=0.9.29=h2531618_0
  - locket=0.2.1=py39h06a4308_2
  - lxml=4.7.1=py39h1f438cf_1
  - lz4-c=1.9.3=h295c915_1
  - lzo=2.10=h7b6447c_2
  - mamba=0.22.1=py39hfa8f2c8_0
  - markdown=3.3.4=py39h06a4308_0
  - markupsafe=2.0.1=py39h27cfd23_0
  - matplotlib=3.5.1=py39h06a4308_0
  - matplotlib-base=3.5.1=py39ha18d171_0
  - matplotlib-inline=0.1.2=pyhd3eb1b0_2
  - mccabe=0.6.1=py39h06a4308_1
  - mistune=0.8.4=py39h27cfd23_1000
  - mkl=2021.4.0=h06a4308_640
  - mkl-service=2.4.0=py39h7f8727e_0
  - mkl_fft=1.3.1=py39hd3c417c_0
  - mkl_random=1.2.2=py39h51133e4_0
  - mpi=1.0=mpich
  - mpich=3.3.2=hc856adb_0
  - msgpack-numpy=0.4.7.1=pyhd3eb1b0_0
  - msgpack-python=1.0.2=py39hff7bd54_1
  - multidict=5.2.0=py39h7f8727e_2
  - multiprocess=0.70.12.2=py39h7f8727e_0
  - multivolumefile=0.2.3=pyhd3eb1b0_0
  - munkres=1.1.4=py_0
  - murmurhash=1.0.7=py39h5a03fae_0
  - mypy=0.910=pyhd3eb1b0_0
  - mypy_extensions=0.4.3=py39h06a4308_1
  - nbclassic=0.2.6=pyhd3eb1b0_0
  - nbclient=0.5.3=pyhd3eb1b0_0
  - nbconvert=6.3.0=py39h06a4308_0
  - nbformat=5.1.3=pyhd3eb1b0_0
  - ncurses=6.3=h7f8727e_2
  - nest-asyncio=1.5.1=pyhd3eb1b0_0
  - nettle=3.7.3=hbbd107a_1
  - networkx=2.7.1=pyhd3eb1b0_0
  - nltk=3.7=pyhd3eb1b0_0
  - notebook=6.4.8=py39h06a4308_0
  - numexpr=2.8.1=py39h6abb31d_0
  - numpy=1.21.2=py39h20f2e39_0
  - numpy-base=1.21.2=py39h79a1101_0
  - oauthlib=3.1.0=py_0
  - olefile=0.46=pyhd3eb1b0_0
  - openh264=2.1.1=h4ff587b_0
  - openjdk=11.0.13=h87a67e3_0
  - openpyxl=3.0.9=pyhd3eb1b0_0
  - openssl=1.1.1q=h166bdaf_0
  - orc=1.7.1=h1be678f_1
  - packaging=21.3=pyhd3eb1b0_0
  - pandas=1.4.1=py39h295c915_0
  - pandocfilters=1.5.0=pyhd3eb1b0_0
  - parso=0.8.3=pyhd3eb1b0_0
  - partd=1.2.0=pyhd3eb1b0_1
  - pathspec=0.7.0=py_0
  - pathy=0.6.2=pyhd8ed1ab_0
  - pcre=8.45=h295c915_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pillow=9.0.1=py39h22f2fdc_0
  - pip=21.2.4=py39h06a4308_0
  - platformdirs=2.4.0=pyhd3eb1b0_0
  - pluggy=1.0.0=py39h06a4308_0
  - preshed=3.0.6=py39h5a03fae_2
  - prometheus_client=0.13.1=pyhd3eb1b0_0
  - prompt-toolkit=3.0.20=pyhd3eb1b0_0
  - prompt_toolkit=3.0.20=hd3eb1b0_0
  - protobuf=3.19.1=py39h295c915_0
  - psutil=5.8.0=py39h27cfd23_1
  - ptyprocess=0.7.0=pyhd3eb1b0_2
  - py=1.10.0=pyhd3eb1b0_0
  - py7zr=0.16.1=pyhd3eb1b0_1
  - pyarrow=4.0.1=py39he0739d4_3
  - pyasn1=0.4.8=pyhd3eb1b0_0
  - pyasn1-modules=0.2.8=py_0
  - pybind11-abi=4=hd8ed1ab_3
  - pycocotools=2.0.4=py39hce5d2b2_0
  - pycosat=0.6.3=py39h27cfd23_0
  - pycparser=2.21=pyhd3eb1b0_0
  - pycryptodomex=3.10.1=py39h27cfd23_1
  - pydantic=1.8.2=py39h3811e60_2
  - pydeprecate=0.3.1=pyhd8ed1ab_0
  - pyemd=0.5.1=py39h9a67853_0
  - pygments=2.11.2=pyhd3eb1b0_0
  - pyjwt=2.3.0=pyhd8ed1ab_1
  - pylint=2.12.2=py39h06a4308_0
  - pyopenssl=22.0.0=pyhd3eb1b0_0
  - pyparsing=3.0.4=pyhd3eb1b0_0
  - pyppmd=0.16.1=py39h295c915_0
  - pyqt=5.9.2=py39h2531618_6
  - pyrsistent=0.18.0=py39heee7806_0
  - pysocks=1.7.1=py39h06a4308_0
  - pytest=6.2.5=py39h06a4308_2
  - python=3.9.7=h12debd9_1
  - python-dateutil=2.8.2=pyhd3eb1b0_0
  - python-levenshtein=0.12.2=py39h27cfd23_0
  - python-lmdb=1.2.1=py39h2531618_1
  - python-xxhash=2.0.2=py39h7f8727e_0
  - python_abi=3.9=1_cp39
  - pytorch=1.10.2=py3.9_cuda10.2_cudnn7.6.5_0
  - pytorch-lightning=1.4.9=pyhd8ed1ab_0
  - pytorch-mutex=1.0=cuda
  - pytz=2021.3=pyhd3eb1b0_0
  - pywavelets=1.3.0=py39h7f8727e_0
  - pyyaml=6.0=py39h7f8727e_1
  - pyzmq=22.3.0=py39h295c915_2
  - pyzstd=0.14.4=py39hb1a5f9f_2
  - qt=5.9.7=h5867ecd_1
  - qtconsole=5.2.2=pyhd3eb1b0_0
  - qtpy=1.11.2=pyhd3eb1b0_0
  - re2=2021.11.01=h9c3ff4c_0
  - readline=8.1.2=h7f8727e_1
  - regex=2021.11.2=py39h7f8727e_0
  - reproc=14.2.3=h7f98852_0
  - reproc-cpp=14.2.3=h9c3ff4c_0
  - requests=2.27.1=pyhd3eb1b0_0
  - requests-oauthlib=1.3.0=py_0
  - rich=10.16.2=pyhd3eb1b0_0
  - rope=0.22.0=pyhd3eb1b0_0
  - rsa=4.7.2=pyhd3eb1b0_1
  - ruamel_yaml=0.15.100=py39h27cfd23_0
  - s2n=1.0.10=h9b69904_0
  - s3transfer=0.5.0=pyhd3eb1b0_0
  - sacremoses=0.0.43=pyhd3eb1b0_0
  - scikit-image=0.16.2=py39ha9443f7_0
  - scikit-learn=1.0.2=py39h51133e4_1
  - scikit-learn-intelex=2021.5.0=py39h06a4308_0
  - scipy=1.7.3=py39hc147768_0
  - seaborn=0.11.2=pyhd3eb1b0_0
  - send2trash=1.8.0=pyhd3eb1b0_1
  - setuptools=58.0.4=py39h06a4308_0
  - shellingham=1.4.0=pyh44b312d_0
  - sip=4.19.13=py39h295c915_0
  - six=1.16.0=pyhd3eb1b0_0
  - snappy=1.1.8=he6710b0_0
  - sniffio=1.2.0=py39h06a4308_1
  - soupsieve=2.3.1=pyhd3eb1b0_0
  - spacy=3.3.1=py39hac2352c_0
  - spacy-legacy=3.0.9=pyhd8ed1ab_0
  - spacy-loggers=1.0.2=pyhd8ed1ab_0
  - sqlite=3.37.2=hc218d9a_0
  - srsly=2.4.3=py39h5a03fae_1
  - tbb=2021.5.0=hd09550d_0
  - tensorboard-data-server=0.6.0=py39hca6d32c_0
  - tensorboard-plugin-wit=1.6.0=py_0
  - tensorboardx=2.2=pyhd3eb1b0_0
  - terminado=0.9.4=py39h06a4308_0
  - testpath=0.5.0=pyhd3eb1b0_0
  - texttable=1.6.4=pyhd3eb1b0_0
  - thinc=8.0.17=py39hac2352c_0
  - threadpoolctl=2.2.0=pyh0d69192_0
  - tk=8.6.11=h1ccaba5_0
  - tokenizers=0.10.3=py39hb317417_1
  - toml=0.10.2=pyhd3eb1b0_0
  - toolz=0.11.2=pyhd3eb1b0_0
  - torchmetrics=0.7.2=pyhd8ed1ab_0
  - torchtext=0.11.2=py39
  - torchvision=0.11.3=py39_cu102
  - tornado=6.1=py39h27cfd23_0
  - tqdm=4.62.3=pyhd3eb1b0_1
  - traitlets=5.1.1=pyhd3eb1b0_0
  - transformers=4.5.1=py_0
  - typed-ast=1.4.3=py39h7f8727e_1
  - typer=0.4.2=pyhd8ed1ab_0
  - typing-extensions=4.1.1=hd3eb1b0_0
  - typing_extensions=4.1.1=pyh06a4308_0
  - tzdata=2021e=hda174b7_0
  - urllib3=1.26.8=pyhd3eb1b0_0
  - wasabi=0.9.1=pyhd8ed1ab_0
  - wcwidth=0.2.5=pyhd3eb1b0_0
  - webencodings=0.5.1=py39h06a4308_1
  - werkzeug=2.0.2=pyhd3eb1b0_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - widgetsnbextension=3.5.1=py39h06a4308_0
  - wrapt=1.13.3=py39h7f8727e_2
  - x264=1!157.20191217=h7b6447c_0
  - xxhash=0.8.0=h7f8727e_3
  - xz=5.2.5=h7b6447c_0
  - yaml=0.2.5=h7b6447c_0
  - yaml-cpp=0.6.3=he1b5a44_4
  - yarl=1.6.3=py39h27cfd23_0
  - zeromq=4.3.4=h2531618_0
  - zipp=3.7.0=pyhd3eb1b0_0
  - zlib=1.2.11=h36c2ea0_1013
  - zstd=1.5.0=ha4553b6_1
  - pip:
    - asttokens==2.0.5
    - astunparse==1.6.3
    - beartype==0.9.1
    - bleurt==0.0.2
    - colored==1.4.3
    - crosshair-tool==0.0.27
    - deepspeed==0.5.10
    - en-core-web-sm==3.3.0
    - fairscale==0.4.5
    - farrow-and-ball==0.0.3
    - flatbuffers==2.0
    - gast==0.5.3
    - glob2==0.7
    - google-pasta==0.2.0
    - hjson==3.0.2
    - humanfriendly==10.0
    - humanize==3.14.0
    - icontract==2.6.1
    - importlib-resources==5.4.0
    - info-nce-pytorch==0.1.4
    - junit-xml==1.8
    - keras==2.8.0
    - keras-preprocessing==1.1.2
    - libclang==13.0.0
    - multimodal==0.0.13
    - mutmut==2.4.0
    - ninja==1.10.2.3
    - opt-einsum==3.3.0
    - plotly==5.6.0
    - pony==0.7.16
    - py-cpuinfo==8.0.0
    - pymongo==4.0.1
    - pysmartdl==1.3.4
    - sentencepiece==0.1.96
    - slurm-gpustat==0.0.12
    - smart-open==5.2.1
    - tables==3.7.0
    - tabulate==0.8.9
    - tenacity==8.0.1
    - tensorboard==2.8.0
    - tensorflow==2.8.0
    - tensorflow-io-gcs-filesystem==0.24.0
    - termcolor==1.1.0
    - tf-estimator-nightly==2.8.0.dev2021122109
    - tf-slim==1.1.0
    - torchray==1.0.0.2
    - triton==1.0.0
    - tueplots==0.0.3
    - typeshed-client==2.0.5
    - typing-inspect==0.7.1
    - vscodedebugvisualizer==0.1.0
    - yacs==0.1.8
    - z3-solver==4.8.9.0
prefix: /path/to/my/env/base
pschanely commented 2 years ago

Thank you for following up! Indeed, with your extra info I was able to reproduce (looks like importing torch is enough to make this crash happen for me).

This problem (and I think it's an additional problem on top of the earlier one) relates to some wacky re-importing tactics I added in #32. Looking at how pytorch uses the TYPE_CHECKING guards, I now believe that the whole original idea of importing with TYPE_CHECKING=1 is fundamentally flawed, and probably needs to be reverted. This means that we may not get all the type annotations that we'd like ... but we'll always match whatever annotations are present at runtime, which is at least easy to explain.

Anyway, this would be a more impactful change than the prior one: I'll take a day or two to try it out on some examples before asking you to try a new version. Thank you again for your patience and persistence!

pschanely commented 2 years ago

Ok - v0.0.28 is released now and rips out the logic that causes trouble in your example. (I was able to reproduce approximately the same crash). But I'm still not in exactly the same environment, and even if fixed, it might just reveal the next issue. I'll really appreciate you attempting it and letting me know how it goes!

pschanely commented 2 years ago

Going to be optimistic and assume my fixed worked for you, and close this. Please reopen though, if it didn't!