search

bertsky / ocrd_detectron2

OCR-D wrapper for detectron2 based segmentation models
16 stars 5 forks source link

ocrd-detectron2-segment does not work with torch from PyPI and segfaults when build and runtime CUDA versions don't match #10

Closed stweil closed 1 year ago

stweil commented 2 years ago

ocrd_detectron2 requires torch>=1.10.1, but fails to run when torch was installed from PyPI:

# Let's try a standard installation with CUDA first.

stweil@ocr-02:~/src/github/OCR-D/ocrd_all$ cd ocrd_detectron2/
stweil@ocr-02:~/src/github/OCR-D/ocrd_all/ocrd_detectron2$ python3 -m venv venv
stweil@ocr-02:~/src/github/OCR-D/ocrd_all/ocrd_detectron2$ source venv/bin/activate
(venv) stweil@ocr-02:~/src/github/OCR-D/ocrd_all/ocrd_detectron2$ make install CUDA_VERSION=11.3
if test -n "$CUDA_VERSION"; then :; \
elif test -s /usr/local/cuda/version.txt; then \
    CUDA_VERSION=$(sed 's/^.* //;s/\([0-9]\+[.][0-9]\).*/\1/' /usr/local/cuda/version.txt); \
elif command -v nvcc &>/dev/null; then \
    CUDA_VERSION=$(nvcc --version | sed -n '/^Cuda/{s/.* release //;s/,.*//;p}'); \
elif command -v nvidia-smi &>/dev/null; then \
    CUDA_VERSION=$(nvidia-smi | sed -n '/CUDA Version/{s/.*CUDA Version: //;s/ .*//;p}'); \
elif command -v pkg-config &>/dev/null; then \
    CUDA_VERSION=$(pkg-config --list-all | sed -n '/^cudart/{s/cudart-//;s/ .*//;p;q}'); \
fi && \
if test "$CUDA_VERSION" = 10.0 -o "$CUDA_VERSION" = 11.0; then \
    echo "Detected CUDA version $CUDA_VERSION, which is not supported by Detectron2 - falling back to CPU-only"; CUDA_VERSION=CPU; \
elif test -z "$CUDA_VERSION"; then \
    echo "Cannot find CUDA runtime library, assuming CPU-only"; CUDA_VERSION=CPU; \
fi && echo "Detected CUDA version: $CUDA_VERSION" && \
if test "$CUDA_VERSION" = CPU; then CUDA=cpu; \
else IFS=. CUDA=($CUDA_VERSION) && CUDA=cu${CUDA[0]}${CUDA[1]}; \
fi && pip3 install -r requirements.txt \
-f "https://dl.fbaipublicfiles.com/detectron2/wheels/$CUDA/torch1.10/index.html" \
-f "https://download.pytorch.org/whl/$CUDA/torch_stable.html"
Detected CUDA version: 11.3
Looking in links: https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html, https://download.pytorch.org/whl/cu113/torch_stable.html
Collecting ocrd>=2.30
  Downloading ocrd-2.32.0-py3-none-any.whl (64 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.3/64.3 KB 4.2 MB/s eta 0:00:00
Collecting click>=7.0
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting scipy
  Downloading scipy-1.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.6/41.6 MB 65.5 MB/s eta 0:00:00
Collecting numpy>=1.17.0
  Using cached numpy-1.22.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Collecting pillow>=7.1.2
  Using cached Pillow-9.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
Collecting scikit-image>=0.17.2
  Downloading scikit_image-0.19.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 81.9 MB/s eta 0:00:00
Collecting torch>=1.10.1
  Downloading https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp38-cp38-linux_x86_64.whl (1637.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 GB 5.7 MB/s eta 0:00:00
Collecting torchvision>=0.11.2
  Downloading https://download.pytorch.org/whl/cu113/torchvision-0.12.0%2Bcu113-cp38-cp38-linux_x86_64.whl (22.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 22.3/22.3 MB 92.4 MB/s eta 0:00:00
Collecting detectron2>=0.6
  Using cached https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/detectron2-0.6%2Bcu113-cp38-cp38-linux_x86_64.whl (7.0 MB)
Collecting ocrd-validators==2.32.0
  Downloading ocrd_validators-2.32.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.9/58.9 KB 6.4 MB/s eta 0:00:00
Collecting bagit-profile>=1.3.0
  Using cached bagit_profile-1.3.1-py3-none-any.whl (14 kB)
Collecting Deprecated==1.2.0
  Using cached Deprecated-1.2.0-py2.py3-none-any.whl (7.8 kB)
Collecting ocrd-models==2.32.0
  Downloading ocrd_models-2.32.0-py3-none-any.whl (102 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.0/102.0 KB 10.4 MB/s eta 0:00:00
Collecting bagit>=1.7.0
  Using cached bagit-1.8.1-py2.py3-none-any.whl (35 kB)
Collecting jsonschema
  Using cached jsonschema-4.4.0-py3-none-any.whl (72 kB)
Collecting ocrd-modelfactory==2.32.0
  Downloading ocrd_modelfactory-2.32.0-py3-none-any.whl (2.6 kB)
Collecting requests
  Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting Flask
  Downloading Flask-2.1.2-py3-none-any.whl (95 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.2/95.2 KB 10.2 MB/s eta 0:00:00
Collecting ocrd-utils==2.32.0
  Downloading ocrd_utils-2.32.0-py3-none-any.whl (19 kB)
Collecting lxml
  Downloading lxml-4.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/6.9 MB 90.4 MB/s eta 0:00:00
Collecting pyyaml
  Using cached PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)
Collecting opencv-python-headless
  Using cached opencv_python_headless-4.5.5.64-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (47.8 MB)
Collecting wrapt<2,>=1
  Downloading wrapt-1.14.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.0/81.0 KB 9.7 MB/s eta 0:00:00
Collecting atomicwrites>=1.3.0
  Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting shapely
  Downloading Shapely-1.8.1.post1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 83.6 MB/s eta 0:00:00
Collecting packaging>=20.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting networkx>=2.2
  Downloading networkx-2.8-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 74.5 MB/s eta 0:00:00
Collecting imageio>=2.4.1
  Using cached imageio-2.18.0-py3-none-any.whl (3.4 MB)
Collecting tifffile>=2019.7.26
  Downloading tifffile-2022.4.28-py3-none-any.whl (193 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 193.9/193.9 KB 24.1 MB/s eta 0:00:00
Collecting PyWavelets>=1.1.1
  Downloading PyWavelets-1.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/6.9 MB 92.2 MB/s eta 0:00:00
Collecting typing-extensions
  Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting termcolor>=1.1
  Using cached termcolor-1.1.0.tar.gz (3.9 kB)
  Preparing metadata (setup.py) ... done
Collecting pycocotools>=2.0.2
  Using cached pycocotools-2.0.4-cp38-cp38-linux_x86_64.whl
Collecting hydra-core>=1.1
  Using cached hydra_core-1.1.2-py3-none-any.whl (147 kB)
Collecting pydot
  Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB)
Collecting omegaconf>=2.1
  Using cached omegaconf-2.1.2-py3-none-any.whl (74 kB)
Collecting cloudpickle
  Using cached cloudpickle-2.0.0-py3-none-any.whl (25 kB)
Collecting yacs>=0.1.8
  Using cached yacs-0.1.8-py3-none-any.whl (14 kB)
Collecting future
  Using cached future-0.18.2.tar.gz (829 kB)
  Preparing metadata (setup.py) ... done
Collecting iopath<0.1.10,>=0.1.7
  Using cached iopath-0.1.9-py3-none-any.whl (27 kB)
Collecting tensorboard
  Using cached tensorboard-2.8.0-py3-none-any.whl (5.8 MB)
Collecting tqdm>4.29.0
  Using cached tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
Collecting matplotlib
  Using cached matplotlib-3.5.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.3 MB)
Collecting fvcore<0.1.6,>=0.1.5
  Using cached fvcore-0.1.5.post20220414.tar.gz (50 kB)
  Preparing metadata (setup.py) ... done
Collecting tabulate
  Using cached tabulate-0.8.9-py3-none-any.whl (25 kB)
Collecting black==21.4b2
  Using cached black-21.4b2-py3-none-any.whl (130 kB)
Collecting pathspec<1,>=0.8.1
  Using cached pathspec-0.9.0-py2.py3-none-any.whl (31 kB)
Collecting appdirs
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting mypy-extensions>=0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting regex>=2020.1.8
  Using cached regex-2022.4.24-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (764 kB)
Collecting toml>=0.10.1
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting antlr4-python3-runtime==4.8
  Using cached antlr4-python3-runtime-4.8.tar.gz (112 kB)
  Preparing metadata (setup.py) ... done
Collecting importlib-resources<5.3
  Using cached importlib_resources-5.2.3-py3-none-any.whl (27 kB)
Collecting portalocker
  Using cached portalocker-2.4.0-py2.py3-none-any.whl (16 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.8-py3-none-any.whl (98 kB)
Collecting python-dateutil>=2.7
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.4.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting fonttools>=4.22.0
  Using cached fonttools-4.33.3-py3-none-any.whl (930 kB)
Collecting Werkzeug>=2.0
  Using cached Werkzeug-2.1.2-py3-none-any.whl (224 kB)
Collecting importlib-metadata>=3.6.0
  Using cached importlib_metadata-4.11.3-py3-none-any.whl (18 kB)
Collecting Jinja2>=3.0
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 KB 13.6 MB/s eta 0:00:00
Collecting itsdangerous>=2.0
  Using cached itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting attrs>=17.4.0
  Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Downloading pyrsistent-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (119 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 119.8/119.8 KB 12.1 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting grpcio>=1.24.3
  Using cached grpcio-1.44.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
Collecting protobuf>=3.6.0
  Using cached protobuf-3.20.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
Collecting markdown>=2.6.8
  Using cached Markdown-3.3.6-py3-none-any.whl (97 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Using cached tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
Collecting tensorboard-plugin-wit>=1.6.0
  Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
Collecting wheel>=0.26
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting absl-py>=0.4
  Using cached absl_py-1.0.0-py3-none-any.whl (126 kB)
Collecting google-auth<3,>=1.6.3
  Using cached google_auth-2.6.6-py2.py3-none-any.whl (156 kB)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->-r requirements.txt (line 9)) (56.0.0)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting rsa<5,>=3.1.4
  Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting cachetools<6.0,>=2.0.0
  Using cached cachetools-5.0.0-py3-none-any.whl (9.1 kB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting requests-oauthlib>=0.7.0
  Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting zipp>=0.5
  Using cached zipp-3.8.0-py3-none-any.whl (5.4 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Using legacy 'setup.py install' for fvcore, since package 'wheel' is not installed.
Using legacy 'setup.py install' for antlr4-python3-runtime, since package 'wheel' is not installed.
Using legacy 'setup.py install' for termcolor, since package 'wheel' is not installed.
Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
Installing collected packages: termcolor, tensorboard-plugin-wit, tabulate, pyasn1, mypy-extensions, certifi, bagit, appdirs, antlr4-python3-runtime, zipp, wrapt, wheel, Werkzeug, urllib3, typing-extensions, tqdm, toml, tensorboard-data-server, six, shapely, rsa, regex, pyyaml, pyrsistent, pyparsing, pyasn1-modules, protobuf, portalocker, pillow, pathspec, oauthlib, numpy, networkx, MarkupSafe, lxml, kiwisolver, itsdangerous, idna, future, fonttools, cycler, cloudpickle, click, charset-normalizer, cachetools, attrs, atomicwrites, yacs, torch, tifffile, scipy, requests, PyWavelets, python-dateutil, pydot, packaging, opencv-python-headless, omegaconf, ocrd-utils, Jinja2, iopath, importlib-resources, importlib-metadata, imageio, grpcio, google-auth, Deprecated, black, absl-py, torchvision, scikit-image, requests-oauthlib, ocrd-models, matplotlib, markdown, jsonschema, hydra-core, fvcore, Flask, bagit-profile, pycocotools, ocrd-modelfactory, google-auth-oauthlib, tensorboard, ocrd-validators, ocrd, detectron2
  Running setup.py install for termcolor ... done
  Running setup.py install for antlr4-python3-runtime ... done
  Running setup.py install for future ... done
  Running setup.py install for fvcore ... done
Successfully installed Deprecated-1.2.0 Flask-2.1.2 Jinja2-3.1.2 MarkupSafe-2.1.1 PyWavelets-1.3.0 Werkzeug-2.1.2 absl-py-1.0.0 antlr4-python3-runtime-4.8 appdirs-1.4.4 atomicwrites-1.4.0 attrs-21.4.0 bagit-1.8.1 bagit-profile-1.3.1 black-21.4b2 cachetools-5.0.0 certifi-2021.10.8 charset-normalizer-2.0.12 click-8.1.3 cloudpickle-2.0.0 cycler-0.11.0 detectron2-0.6+cu113 fonttools-4.33.3 future-0.18.2 fvcore-0.1.5.post20220414 google-auth-2.6.6 google-auth-oauthlib-0.4.6 grpcio-1.44.0 hydra-core-1.1.2 idna-3.3 imageio-2.18.0 importlib-metadata-4.11.3 importlib-resources-5.2.3 iopath-0.1.9 itsdangerous-2.1.2 jsonschema-4.4.0 kiwisolver-1.4.2 lxml-4.8.0 markdown-3.3.6 matplotlib-3.5.1 mypy-extensions-0.4.3 networkx-2.8 numpy-1.22.3 oauthlib-3.2.0 ocrd-2.32.0 ocrd-modelfactory-2.32.0 ocrd-models-2.32.0 ocrd-utils-2.32.0 ocrd-validators-2.32.0 omegaconf-2.1.2 opencv-python-headless-4.5.5.64 packaging-21.3 pathspec-0.9.0 pillow-9.1.0 portalocker-2.4.0 protobuf-3.20.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycocotools-2.0.4 pydot-1.4.2 pyparsing-3.0.8 pyrsistent-0.18.1 python-dateutil-2.8.2 pyyaml-6.0 regex-2022.4.24 requests-2.27.1 requests-oauthlib-1.3.1 rsa-4.8 scikit-image-0.19.2 scipy-1.8.0 shapely-1.8.1.post1 six-1.16.0 tabulate-0.8.9 tensorboard-2.8.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 termcolor-1.1.0 tifffile-2022.4.28 toml-0.10.2 torch-1.11.0+cu113 torchvision-0.12.0+cu113 tqdm-4.64.0 typing-extensions-4.2.0 urllib3-1.26.9 wheel-0.37.1 wrapt-1.14.0 yacs-0.1.8 zipp-3.8.0
pip3 install .
Processing /home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2
  Preparing metadata (setup.py) ... done
Requirement already satisfied: ocrd>=2.30 in ./venv/lib/python3.8/site-packages (from ocrd-detectron2==0.1.1) (2.32.0)
Requirement already satisfied: click>=7.0 in ./venv/lib/python3.8/site-packages (from ocrd-detectron2==0.1.1) (8.1.3)
Requirement already satisfied: scipy in ./venv/lib/python3.8/site-packages (from ocrd-detectron2==0.1.1) (1.8.0)
Requirement already satisfied: numpy>=1.17.0 in ./venv/lib/python3.8/site-packages (from ocrd-detectron2==0.1.1) (1.22.3)
Requirement already satisfied: pillow>=7.1.2 in ./venv/lib/python3.8/site-packages (from ocrd-detectron2==0.1.1) (9.1.0)
Requirement already satisfied: scikit-image>=0.17.2 in ./venv/lib/python3.8/site-packages (from ocrd-detectron2==0.1.1) (0.19.2)
Requirement already satisfied: torch>=1.10.1 in ./venv/lib/python3.8/site-packages (from ocrd-detectron2==0.1.1) (1.11.0+cu113)
Requirement already satisfied: torchvision>=0.11.2 in ./venv/lib/python3.8/site-packages (from ocrd-detectron2==0.1.1) (0.12.0+cu113)
Requirement already satisfied: detectron2>=0.6 in ./venv/lib/python3.8/site-packages (from ocrd-detectron2==0.1.1) (0.6+cu113)
Requirement already satisfied: tabulate in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (0.8.9)
Requirement already satisfied: pydot in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (1.4.2)
Requirement already satisfied: matplotlib in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (3.5.1)
Requirement already satisfied: tensorboard in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (2.8.0)
Requirement already satisfied: omegaconf>=2.1 in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (2.1.2)
Requirement already satisfied: tqdm>4.29.0 in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (4.64.0)
Requirement already satisfied: fvcore<0.1.6,>=0.1.5 in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (0.1.5.post20220414)
Requirement already satisfied: black==21.4b2 in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (21.4b2)
Requirement already satisfied: termcolor>=1.1 in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (1.1.0)
Requirement already satisfied: future in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (0.18.2)
Requirement already satisfied: iopath<0.1.10,>=0.1.7 in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (0.1.9)
Requirement already satisfied: pycocotools>=2.0.2 in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (2.0.4)
Requirement already satisfied: yacs>=0.1.8 in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (0.1.8)
Requirement already satisfied: cloudpickle in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (2.0.0)
Requirement already satisfied: hydra-core>=1.1 in ./venv/lib/python3.8/site-packages (from detectron2>=0.6->ocrd-detectron2==0.1.1) (1.1.2)
Requirement already satisfied: regex>=2020.1.8 in ./venv/lib/python3.8/site-packages (from black==21.4b2->detectron2>=0.6->ocrd-detectron2==0.1.1) (2022.4.24)
Requirement already satisfied: mypy-extensions>=0.4.3 in ./venv/lib/python3.8/site-packages (from black==21.4b2->detectron2>=0.6->ocrd-detectron2==0.1.1) (0.4.3)
Requirement already satisfied: pathspec<1,>=0.8.1 in ./venv/lib/python3.8/site-packages (from black==21.4b2->detectron2>=0.6->ocrd-detectron2==0.1.1) (0.9.0)
Requirement already satisfied: appdirs in ./venv/lib/python3.8/site-packages (from black==21.4b2->detectron2>=0.6->ocrd-detectron2==0.1.1) (1.4.4)
Requirement already satisfied: toml>=0.10.1 in ./venv/lib/python3.8/site-packages (from black==21.4b2->detectron2>=0.6->ocrd-detectron2==0.1.1) (0.10.2)
Requirement already satisfied: bagit>=1.7.0 in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (1.8.1)
Requirement already satisfied: opencv-python-headless in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (4.5.5.64)
Requirement already satisfied: Deprecated==1.2.0 in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (1.2.0)
Requirement already satisfied: ocrd-utils==2.32.0 in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (2.32.0)
Requirement already satisfied: requests in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (2.27.1)
Requirement already satisfied: pyyaml in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (6.0)
Requirement already satisfied: ocrd-modelfactory==2.32.0 in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (2.32.0)
Requirement already satisfied: bagit-profile>=1.3.0 in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (1.3.1)
Requirement already satisfied: ocrd-validators==2.32.0 in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (2.32.0)
Requirement already satisfied: ocrd-models==2.32.0 in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (2.32.0)
Requirement already satisfied: Flask in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (2.1.2)
Requirement already satisfied: lxml in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (4.8.0)
Requirement already satisfied: jsonschema in ./venv/lib/python3.8/site-packages (from ocrd>=2.30->ocrd-detectron2==0.1.1) (4.4.0)
Requirement already satisfied: wrapt<2,>=1 in ./venv/lib/python3.8/site-packages (from Deprecated==1.2.0->ocrd>=2.30->ocrd-detectron2==0.1.1) (1.14.0)
Requirement already satisfied: atomicwrites>=1.3.0 in ./venv/lib/python3.8/site-packages (from ocrd-utils==2.32.0->ocrd>=2.30->ocrd-detectron2==0.1.1) (1.4.0)
Requirement already satisfied: shapely in ./venv/lib/python3.8/site-packages (from ocrd-validators==2.32.0->ocrd>=2.30->ocrd-detectron2==0.1.1) (1.8.1.post1)
Requirement already satisfied: PyWavelets>=1.1.1 in ./venv/lib/python3.8/site-packages (from scikit-image>=0.17.2->ocrd-detectron2==0.1.1) (1.3.0)
Requirement already satisfied: packaging>=20.0 in ./venv/lib/python3.8/site-packages (from scikit-image>=0.17.2->ocrd-detectron2==0.1.1) (21.3)
Requirement already satisfied: networkx>=2.2 in ./venv/lib/python3.8/site-packages (from scikit-image>=0.17.2->ocrd-detectron2==0.1.1) (2.8)
Requirement already satisfied: imageio>=2.4.1 in ./venv/lib/python3.8/site-packages (from scikit-image>=0.17.2->ocrd-detectron2==0.1.1) (2.18.0)
Requirement already satisfied: tifffile>=2019.7.26 in ./venv/lib/python3.8/site-packages (from scikit-image>=0.17.2->ocrd-detectron2==0.1.1) (2022.4.28)
Requirement already satisfied: typing-extensions in ./venv/lib/python3.8/site-packages (from torch>=1.10.1->ocrd-detectron2==0.1.1) (4.2.0)
Requirement already satisfied: importlib-resources<5.3 in ./venv/lib/python3.8/site-packages (from hydra-core>=1.1->detectron2>=0.6->ocrd-detectron2==0.1.1) (5.2.3)
Requirement already satisfied: antlr4-python3-runtime==4.8 in ./venv/lib/python3.8/site-packages (from hydra-core>=1.1->detectron2>=0.6->ocrd-detectron2==0.1.1) (4.8)
Requirement already satisfied: portalocker in ./venv/lib/python3.8/site-packages (from iopath<0.1.10,>=0.1.7->detectron2>=0.6->ocrd-detectron2==0.1.1) (2.4.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./venv/lib/python3.8/site-packages (from packaging>=20.0->scikit-image>=0.17.2->ocrd-detectron2==0.1.1) (3.0.8)
Requirement already satisfied: fonttools>=4.22.0 in ./venv/lib/python3.8/site-packages (from matplotlib->detectron2>=0.6->ocrd-detectron2==0.1.1) (4.33.3)
Requirement already satisfied: cycler>=0.10 in ./venv/lib/python3.8/site-packages (from matplotlib->detectron2>=0.6->ocrd-detectron2==0.1.1) (0.11.0)
Requirement already satisfied: python-dateutil>=2.7 in ./venv/lib/python3.8/site-packages (from matplotlib->detectron2>=0.6->ocrd-detectron2==0.1.1) (2.8.2)
Requirement already satisfied: kiwisolver>=1.0.1 in ./venv/lib/python3.8/site-packages (from matplotlib->detectron2>=0.6->ocrd-detectron2==0.1.1) (1.4.2)
Requirement already satisfied: importlib-metadata>=3.6.0 in ./venv/lib/python3.8/site-packages (from Flask->ocrd>=2.30->ocrd-detectron2==0.1.1) (4.11.3)
Requirement already satisfied: itsdangerous>=2.0 in ./venv/lib/python3.8/site-packages (from Flask->ocrd>=2.30->ocrd-detectron2==0.1.1) (2.1.2)
Requirement already satisfied: Jinja2>=3.0 in ./venv/lib/python3.8/site-packages (from Flask->ocrd>=2.30->ocrd-detectron2==0.1.1) (3.1.2)
Requirement already satisfied: Werkzeug>=2.0 in ./venv/lib/python3.8/site-packages (from Flask->ocrd>=2.30->ocrd-detectron2==0.1.1) (2.1.2)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in ./venv/lib/python3.8/site-packages (from jsonschema->ocrd>=2.30->ocrd-detectron2==0.1.1) (0.18.1)
Requirement already satisfied: attrs>=17.4.0 in ./venv/lib/python3.8/site-packages (from jsonschema->ocrd>=2.30->ocrd-detectron2==0.1.1) (21.4.0)
Requirement already satisfied: charset-normalizer~=2.0.0 in ./venv/lib/python3.8/site-packages (from requests->ocrd>=2.30->ocrd-detectron2==0.1.1) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.8/site-packages (from requests->ocrd>=2.30->ocrd-detectron2==0.1.1) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.8/site-packages (from requests->ocrd>=2.30->ocrd-detectron2==0.1.1) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./venv/lib/python3.8/site-packages (from requests->ocrd>=2.30->ocrd-detectron2==0.1.1) (1.26.9)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (0.4.6)
Requirement already satisfied: markdown>=2.6.8 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (3.3.6)
Requirement already satisfied: protobuf>=3.6.0 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (3.20.1)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (0.6.1)
Requirement already satisfied: google-auth<3,>=1.6.3 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (2.6.6)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (56.0.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (1.8.1)
Requirement already satisfied: absl-py>=0.4 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (1.0.0)
Requirement already satisfied: wheel>=0.26 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (0.37.1)
Requirement already satisfied: grpcio>=1.24.3 in ./venv/lib/python3.8/site-packages (from tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (1.44.0)
Requirement already satisfied: six in ./venv/lib/python3.8/site-packages (from absl-py>=0.4->tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (1.16.0)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in ./venv/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (5.0.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in ./venv/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in ./venv/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (4.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in ./venv/lib/python3.8/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (1.3.1)
Requirement already satisfied: zipp>=0.5 in ./venv/lib/python3.8/site-packages (from importlib-metadata>=3.6.0->Flask->ocrd>=2.30->ocrd-detectron2==0.1.1) (3.8.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./venv/lib/python3.8/site-packages (from Jinja2>=3.0->Flask->ocrd>=2.30->ocrd-detectron2==0.1.1) (2.1.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in ./venv/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in ./venv/lib/python3.8/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard->detectron2>=0.6->ocrd-detectron2==0.1.1) (3.2.0)
Building wheels for collected packages: ocrd-detectron2
  Building wheel for ocrd-detectron2 (setup.py) ... done
  Created wheel for ocrd-detectron2: filename=ocrd_detectron2-0.1.1-py3-none-any.whl size=15941 sha256=d3edd8ab97d39ee1eda75d936b0b801caf5f23dd6b677726cf4372e32a553aac
  Stored in directory: /tmp/pip-ephem-wheel-cache-2xkc4fqt/wheels/33/78/30/7c8e61e8c59abc8d3260ebeaaa189ebd04184ecff7da32e3fa
Successfully built ocrd-detectron2
Installing collected packages: ocrd-detectron2
Successfully installed ocrd-detectron2-0.1.1

# ocrd-detectron2-segment "works" (well, at least somehow).

(venv) stweil@ocr-02:~/src/github/OCR-D/ocrd_all/ocrd_detectron2$ ocrd-detectron2-segment 
Segmentation fault

# Now install torch from PyPI.

(venv) stweil@ocr-02:~/src/github/OCR-D/ocrd_all/ocrd_detectron2$ pip install torch==1.10.2 torchvision
Requirement already satisfied: torch==1.10.2 in ./venv/lib/python3.8/site-packages (1.10.2)
Requirement already satisfied: torchvision in ./venv/lib/python3.8/site-packages (0.12.0+cu113)
Requirement already satisfied: typing-extensions in ./venv/lib/python3.8/site-packages (from torch==1.10.2) (4.2.0)
Requirement already satisfied: numpy in ./venv/lib/python3.8/site-packages (from torchvision) (1.22.3)
Collecting torchvision
  Downloading torchvision-0.12.0-cp38-cp38-manylinux1_x86_64.whl (21.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 78.6 MB/s eta 0:00:00
  Downloading torchvision-0.11.3-cp38-cp38-manylinux1_x86_64.whl (23.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.2/23.2 MB 71.8 MB/s eta 0:00:00
Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in ./venv/lib/python3.8/site-packages (from torchvision) (9.1.0)
Installing collected packages: torchvision
  Attempting uninstall: torchvision
    Found existing installation: torchvision 0.12.0+cu113
    Uninstalling torchvision-0.12.0+cu113:
      Successfully uninstalled torchvision-0.12.0+cu113
Successfully installed torchvision-0.11.3

# All requirements are still fine.

(venv) stweil@ocr-02:~/src/github/OCR-D/ocrd_all/ocrd_detectron2$ pip check
No broken requirements found.

# But ocrd-detectron2-segment no longer works.

(venv) stweil@ocr-02:~/src/github/OCR-D/ocrd_all/ocrd_detectron2$ ocrd-detectron2-segment 
Traceback (most recent call last):
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/bin/ocrd-detectron2-segment", line 5, in <module>
    from ocrd_detectron2.cli import ocrd_detectron2_segment
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/ocrd_detectron2/cli.py", line 4, in <module>
    from .segment import Detectron2Segment
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/ocrd_detectron2/segment.py", line 18, in <module>
    from detectron2.engine import DefaultPredictor
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/engine/__init__.py", line 11, in <module>
    from .hooks import *
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/engine/hooks.py", line 22, in <module>
    from detectron2.evaluation.testing import flatten_results_dict
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/evaluation/__init__.py", line 2, in <module>
    from .cityscapes_evaluation import CityscapesInstanceEvaluator, CityscapesSemSegEvaluator
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/evaluation/cityscapes_evaluation.py", line 11, in <module>
    from detectron2.data import MetadataCatalog
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/data/__init__.py", line 4, in <module>
    from .build import (
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/data/build.py", line 13, in <module>
    from detectron2.structures import BoxMode
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/structures/__init__.py", line 3, in <module>
    from .image_list import ImageList
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/structures/image_list.py", line 8, in <module>
    from detectron2.layers.wrappers import shapes_to_tensor
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/layers/__init__.py", line 3, in <module>
    from .deform_conv import DeformConv, ModulatedDeformConv
  File "/home/stweil/src/github/OCR-D/ocrd_all/ocrd_detectron2/venv/lib/python3.8/site-packages/detectron2/layers/deform_conv.py", line 11, in <module>
    from detectron2 import _C
ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
bertsky commented 2 years ago

ocrd-detectron2-segment "works" (well, at least somehow).

(venv) stweil@ocr-02:~/src/github/OCR-D/ocrd_all/ocrd_detectron2$ ocrd-detectron2-segment Segmentation fault

I would not call that "works".

Are you sure your system has CUDA 11.3?

Now install torch from PyPI.

(venv) stweil@ocr-02:~/src/github/OCR-D/ocrd_all/ocrd_detectron2$ pip install torch==1.10.2 torchvision

That obviously cannot work. Detectron2 was built against another Pytorch (torch-1.11.0+cu113) above. Now you take that away. Use another venv if you need a specific Pytorch other than the one picked by ocrd_detectron2.

stweil commented 2 years ago

I am sure that my Debian system has CUDA 11.2. Therefore "works" here just means that ocrd-detectron2-segment does not fail because of build issues. Of course it is not nice that the program crashes. It should at least terminate with a reasonable error message or ideally run with "older" CUDA versions, too. Will each update of CUDA require updates of ocrd_detectron2? Then ocrd-detectron2-segment should make a version check!

stweil commented 2 years ago

That obviously cannot work. Detectron2 was built against another Pytorch (torch-1.11.0+cu113) above. Now you take that away. Use another venv if you need a specific Pytorch other than the one picked by ocrd_detectron2.

Yes, that's why I said that I don't think this is an issue for ocrd_detectron2 and why that OCR-D processor must currently use a separate venv because other processors in the main venv use the PyPI torch.

bertsky commented 2 years ago

I am sure that my Debian system has CUDA 11.2. Therefore "works" here just means that ocrd-detectron2-segment does not fail because of build issues. Of course it is not nice that the program crashes. It should at least terminate with a reasonable error message or ideally run with "older" CUDA versions, too.

There is not much application developers can do to achieve that, unfortunately. If the CUDA libs don't match, you only get to know when importing crashes.

Please state CUDA_VERSION=11.3 or let it auto-detect, and try again (after pulling master).

Will each update of CUDA require updates of ocrd_detectron2? Then ocrd-detectron2-segment should make a version check!

No, not at all. The requirements are very lax here: detectron2>=0.6

bertsky commented 2 years ago

That obviously cannot work. Detectron2 was built against another Pytorch (torch-1.11.0+cu113) above. Now you take that away. Use another venv if you need a specific Pytorch other than the one picked by ocrd_detectron2.

Yes, that's why I said that I don't think this is an issue for ocrd_detectron2 and why that OCR-D processor must currently use a separate venv because other processors in the main venv use the PyPI torch.

I see. But that could be true for any such combination of modules in a single venv.

I guess we would "just" have to enforce an ordering of ocrd_typegroups_modifier before ocrd_detectron2.

stweil commented 2 years ago

I guess we would "just" have to enforce an ordering of ocrd_typegroups_modifier before ocrd_detectron2.

That does not help. My host has CUDA 11.2, and therefore make -C ocrd_detectron2 install installs the incompatible torch from PyPI.

bertsky commented 2 years ago

I guess we would "just" have to enforce an ordering of ocrd_typegroups_modifier before ocrd_detectron2.

That does not help. My host has CUDA 11.2, and therefore make -C ocrd_detectron2 install installs the incompatible torch from PyPI.

That's unrelated, though. Detectron2 prebuilt wheels were missing for 11.2, and at the time I had no CPU fallback for that, so you installed for 11.3, which you probably still have in your wheel cache – see other issue.