`pip install -U farm-haystack` does not trigger schema generation

[ZanSara]

Describe the bug

• pip install -U farm-haystack does not trigger schema generation

Error message

• The file will be silently not generated
• At the first attempt to load a pipeline, it triggers a FileNotFoundError: [Errno 2] No such file or directory: '/path/to/haystack/haystack/json-schemas/haystack-pipeline-main.schema.json'

Workaround

• Install Haystack without the -U option.

Expected behavior

• The generation should occur.

Additional context

• This might be an issue with hatch-autorun. I'll reach out to them for help.
• https://github.com/deepset-ai/haystack/pull/3485/commits/43b20a039d186e4df340656de10ad94886f8f80c

To Reproduce

• pip install -U ., then try to load any pipeline.

FAQ Check

• [X] Have you had a look at our new FAQ page?

System:

• OS: Linux (Ubuntu)
• Haystack version (commit or version number): main
• Python 3.10.6

[ofek]

lmk if I can help

[ZanSara]

Hey @ofek thank you for jumping in! I didn't try to debug this one yet. However I'm curious, is there any "obvious" reason why the flag -U might not trigger the execution of the hook? I'm not very knowledgeable of how Hatch works, but at a first glance I could find no mention to an issue like this in the hatch-autorun docs or repo.

This is how pip install . -vvv looks like. After this, the autorun code is definitely executed when I invoke the Python interpreter (the script logs).

pip log

``` → pip install . -vvv WARNING:hatch_autorun:Haystack is generating the YAML schema for Pipelines validation. This only happens once, after installing the package. /usr/lib/python3.10/distutils/command/install.py:13: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead from distutils.sysconfig import get_config_vars, is_virtual_environment Using pip 22.0.2 from /home/sara/work/haystack/venv/lib/python3.10/site-packages/pip (python 3.10) Non-user install because user site-packages disabled /home/sara/work/haystack/venv/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release warnings.warn( /home/sara/work/haystack/venv/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release warnings.warn( Created temporary directory: /tmp/pip-ephem-wheel-cache-oh40qiyf Created temporary directory: /tmp/pip-req-tracker-j9ex1r28 Initialized build tracking at /tmp/pip-req-tracker-j9ex1r28 Created build tracker: /tmp/pip-req-tracker-j9ex1r28 Entered build tracker: /tmp/pip-req-tracker-j9ex1r28 Created temporary directory: /tmp/pip-install-bbh84x2z Processing /home/sara/work/haystack Added file:///home/sara/work/haystack to build tracker '/tmp/pip-req-tracker-j9ex1r28' Created temporary directory: /tmp/pip-build-env-29hn01we Created temporary directory: /tmp/pip-standalone-pip-fhbd0wpe Running command pip subprocess to install build dependencies Using pip 22.0.2 from /tmp/pip-standalone-pip-fhbd0wpe/__env_pip__.zip/pip (python 3.10) Collecting hatchling>=1.8.0 Using cached hatchling-1.11.1-py3-none-any.whl (73 kB) Collecting tomli>=1.2.2 Using cached tomli-2.0.1-py3-none-any.whl (12 kB) Collecting pluggy>=1.0.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting editables>=0.3 Using cached editables-0.3-py3-none-any.whl (4.7 kB) Collecting pathspec>=0.10.1 Using cached pathspec-0.10.1-py3-none-any.whl (27 kB) Collecting packaging>=21.3 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Installing collected packages: tomli, pyparsing, pluggy, pathspec, editables, packaging, hatchling Creating /tmp/pip-build-env-29hn01we/overlay/bin changing mode of /tmp/pip-build-env-29hn01we/overlay/bin/hatchling to 775 Successfully installed editables-0.3 hatchling-1.11.1 packaging-21.3 pathspec-0.10.1 pluggy-1.0.0 pyparsing-3.0.9 tomli-2.0.1 Installing build dependencies ... done Running command Getting requirements to build wheel Getting requirements to build wheel ... done Created temporary directory: /tmp/pip-standalone-pip-vu5vpeyo Running command pip subprocess to install backend dependencies Using pip 22.0.2 from /tmp/pip-standalone-pip-vu5vpeyo/__env_pip__.zip/pip (python 3.10) Collecting hatch-autorun Using cached hatch_autorun-1.1.0-py3-none-any.whl (5.3 kB) Collecting hatchling>=1.6.0 Using cached hatchling-1.11.1-py3-none-any.whl (73 kB) Collecting packaging>=21.3 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting tomli>=1.2.2 Using cached tomli-2.0.1-py3-none-any.whl (12 kB) Collecting pathspec>=0.10.1 Using cached pathspec-0.10.1-py3-none-any.whl (27 kB) Collecting pluggy>=1.0.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting editables>=0.3 Using cached editables-0.3-py3-none-any.whl (4.7 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Installing collected packages: tomli, pyparsing, pluggy, pathspec, editables, packaging, hatchling, hatch-autorun Creating /tmp/pip-build-env-29hn01we/normal/bin changing mode of /tmp/pip-build-env-29hn01we/normal/bin/hatchling to 775 Successfully installed editables-0.3 hatch-autorun-1.1.0 hatchling-1.11.1 packaging-21.3 pathspec-0.10.1 pluggy-1.0.0 pyparsing-3.0.9 tomli-2.0.1 Installing backend dependencies ... done Created temporary directory: /tmp/pip-modern-metadata-nn4pr31o Running command Preparing metadata (pyproject.toml) Preparing metadata (pyproject.toml) ... done Source in /home/sara/work/haystack has version 1.11.0rc0, which satisfies requirement farm-haystack==1.11.0rc0 from file:///home/sara/work/haystack Removed farm-haystack==1.11.0rc0 from file:///home/sara/work/haystack from build tracker '/tmp/pip-req-tracker-j9ex1r28' Requirement already satisfied: scikit-learn>=1.0.0 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.1.2) Requirement already satisfied: sentence-transformers>=2.2.0 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.2.2) Requirement already satisfied: tika in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.24) Requirement already satisfied: huggingface-hub>=0.5.0 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (0.10.1) Requirement already satisfied: pydantic in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.10.2) Requirement already satisfied: elasticsearch<8,>=7.7 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (7.9.1) Requirement already satisfied: more-itertools in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (9.0.0) Requirement already satisfied: langdetect in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.0.9) Requirement already satisfied: networkx in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.8.7) Requirement already satisfied: mmh3 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (3.0.0) Requirement already satisfied: transformers==4.21.2 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (4.21.2) Requirement already satisfied: pandas in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.5.1) Requirement already satisfied: scipy>=1.3.2 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.9.3) Requirement already satisfied: jsonschema in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (4.16.0) Requirement already satisfied: azure-ai-formrecognizer>=3.2.0b2 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (3.2.0) Requirement already satisfied: dill in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (0.3.5.1) Requirement already satisfied: quantulum3 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (0.7.11) Requirement already satisfied: nltk in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (3.7) Requirement already satisfied: torch<1.13,>1.9 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.12.1) Requirement already satisfied: rapidfuzz<2.8.0,>=2.0.15 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.7.0) Requirement already satisfied: tqdm in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (4.64.1) Requirement already satisfied: posthog in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.1.2) Requirement already satisfied: mlflow in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.30.0) Requirement already satisfied: python-docx in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (0.8.11) Requirement already satisfied: requests in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.27.1) Requirement already satisfied: seqeval in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.2.2) Requirement already satisfied: regex!=2019.12.17 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (2022.9.13) Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (0.12.1) Requirement already satisfied: pyyaml>=5.1 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (5.4.1) Requirement already satisfied: numpy>=1.17 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (1.23.4) Requirement already satisfied: filelock in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (3.8.0) Requirement already satisfied: packaging>=20.0 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (21.3) Requirement already satisfied: typing-extensions>=4.0.1 in ./venv/lib/python3.10/site-packages (from azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (4.4.0) Requirement already satisfied: msrest>=0.6.21 in ./venv/lib/python3.10/site-packages (from azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (0.7.1) Requirement already satisfied: azure-common~=1.1 in ./venv/lib/python3.10/site-packages (from azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (1.1.28) Requirement already satisfied: azure-core<2.0.0,>=1.23.0 in ./venv/lib/python3.10/site-packages (from azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (1.26.0) Requirement already satisfied: urllib3>=1.21.1 in ./venv/lib/python3.10/site-packages (from elasticsearch<8,>=7.7->farm-haystack==1.11.0rc0) (1.26.12) Requirement already satisfied: certifi in ./venv/lib/python3.10/site-packages (from elasticsearch<8,>=7.7->farm-haystack==1.11.0rc0) (2022.9.24) Requirement already satisfied: jarowinkler<2.0.0,>=1.2.0 in ./venv/lib/python3.10/site-packages (from rapidfuzz<2.8.0,>=2.0.15->farm-haystack==1.11.0rc0) (1.2.3) Requirement already satisfied: joblib>=1.0.0 in ./venv/lib/python3.10/site-packages (from scikit-learn>=1.0.0->farm-haystack==1.11.0rc0) (1.2.0) Requirement already satisfied: threadpoolctl>=2.0.0 in ./venv/lib/python3.10/site-packages (from scikit-learn>=1.0.0->farm-haystack==1.11.0rc0) (3.1.0) Requirement already satisfied: sentencepiece in ./venv/lib/python3.10/site-packages (from sentence-transformers>=2.2.0->farm-haystack==1.11.0rc0) (0.1.97) Requirement already satisfied: torchvision in ./venv/lib/python3.10/site-packages (from sentence-transformers>=2.2.0->farm-haystack==1.11.0rc0) (0.13.1) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in ./venv/lib/python3.10/site-packages (from jsonschema->farm-haystack==1.11.0rc0) (0.18.1) Requirement already satisfied: attrs>=17.4.0 in ./venv/lib/python3.10/site-packages (from jsonschema->farm-haystack==1.11.0rc0) (22.1.0) Requirement already satisfied: six in ./venv/lib/python3.10/site-packages (from langdetect->farm-haystack==1.11.0rc0) (1.16.0) Requirement already satisfied: click<9,>=7.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (8.0.4) Requirement already satisfied: pytz<2023 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (2022.5) Requirement already satisfied: cloudpickle<3 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (2.2.0) Requirement already satisfied: alembic<2 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (1.8.1) Requirement already satisfied: gitpython<4,>=2.1.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (3.1.29) Requirement already satisfied: entrypoints<1 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (0.4) Requirement already satisfied: protobuf<5,>=3.12.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (3.20.1) Requirement already satisfied: importlib-metadata!=4.7.0,<6,>=3.7.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (4.13.0) Requirement already satisfied: gunicorn<21 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (20.1.0) Requirement already satisfied: Flask<3 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (2.2.2) Requirement already satisfied: sqlparse<1,>=0.4.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (0.4.3) Requirement already satisfied: sqlalchemy<2,>=1.4.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (1.4.42) Requirement already satisfied: prometheus-flask-exporter<1 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (0.20.3) Requirement already satisfied: querystring-parser<2 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (1.2.4) Requirement already satisfied: databricks-cli<1,>=0.8.7 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (0.17.3) Requirement already satisfied: docker<7,>=4.0.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (6.0.0) Requirement already satisfied: python-dateutil>=2.8.1 in ./venv/lib/python3.10/site-packages (from pandas->farm-haystack==1.11.0rc0) (2.8.2) Requirement already satisfied: charset-normalizer~=2.0.0 in ./venv/lib/python3.10/site-packages (from requests->farm-haystack==1.11.0rc0) (2.0.12) Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.10/site-packages (from requests->farm-haystack==1.11.0rc0) (3.4) Requirement already satisfied: monotonic>=1.5 in ./venv/lib/python3.10/site-packages (from posthog->farm-haystack==1.11.0rc0) (1.6) Requirement already satisfied: backoff<2.0.0,>=1.10.0 in ./venv/lib/python3.10/site-packages (from posthog->farm-haystack==1.11.0rc0) (1.11.1) Requirement already satisfied: lxml>=2.3.2 in ./venv/lib/python3.10/site-packages (from python-docx->farm-haystack==1.11.0rc0) (4.9.1) Requirement already satisfied: num2words in ./venv/lib/python3.10/site-packages (from quantulum3->farm-haystack==1.11.0rc0) (0.5.12) Requirement already satisfied: inflect in ./venv/lib/python3.10/site-packages (from quantulum3->farm-haystack==1.11.0rc0) (6.0.2) Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from tika->farm-haystack==1.11.0rc0) (59.6.0) Requirement already satisfied: Mako in ./venv/lib/python3.10/site-packages (from alembic<2->mlflow->farm-haystack==1.11.0rc0) (1.2.3) Requirement already satisfied: tabulate>=0.7.7 in ./venv/lib/python3.10/site-packages (from databricks-cli<1,>=0.8.7->mlflow->farm-haystack==1.11.0rc0) (0.9.0) Requirement already satisfied: oauthlib>=3.1.0 in ./venv/lib/python3.10/site-packages (from databricks-cli<1,>=0.8.7->mlflow->farm-haystack==1.11.0rc0) (3.2.2) Requirement already satisfied: pyjwt>=1.7.0 in ./venv/lib/python3.10/site-packages (from databricks-cli<1,>=0.8.7->mlflow->farm-haystack==1.11.0rc0) (2.6.0) Requirement already satisfied: websocket-client>=0.32.0 in ./venv/lib/python3.10/site-packages (from docker<7,>=4.0.0->mlflow->farm-haystack==1.11.0rc0) (1.4.1) Requirement already satisfied: itsdangerous>=2.0 in ./venv/lib/python3.10/site-packages (from Flask<3->mlflow->farm-haystack==1.11.0rc0) (2.1.2) Requirement already satisfied: Jinja2>=3.0 in ./venv/lib/python3.10/site-packages (from Flask<3->mlflow->farm-haystack==1.11.0rc0) (3.1.2) Requirement already satisfied: Werkzeug>=2.2.2 in ./venv/lib/python3.10/site-packages (from Flask<3->mlflow->farm-haystack==1.11.0rc0) (2.2.2) Requirement already satisfied: gitdb<5,>=4.0.1 in ./venv/lib/python3.10/site-packages (from gitpython<4,>=2.1.0->mlflow->farm-haystack==1.11.0rc0) (4.0.9) Requirement already satisfied: zipp>=0.5 in ./venv/lib/python3.10/site-packages (from importlib-metadata!=4.7.0,<6,>=3.7.0->mlflow->farm-haystack==1.11.0rc0) (3.10.0) Requirement already satisfied: isodate>=0.6.0 in ./venv/lib/python3.10/site-packages (from msrest>=0.6.21->azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (0.6.1) Requirement already satisfied: requests-oauthlib>=0.5.0 in ./venv/lib/python3.10/site-packages (from msrest>=0.6.21->azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (1.3.1) Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./venv/lib/python3.10/site-packages (from packaging>=20.0->transformers==4.21.2->farm-haystack==1.11.0rc0) (3.0.9) Requirement already satisfied: prometheus-client in ./venv/lib/python3.10/site-packages (from prometheus-flask-exporter<1->mlflow->farm-haystack==1.11.0rc0) (0.15.0) Requirement already satisfied: greenlet!=0.4.17 in ./venv/lib/python3.10/site-packages (from sqlalchemy<2,>=1.4.0->mlflow->farm-haystack==1.11.0rc0) (1.1.3.post0) Requirement already satisfied: docopt>=0.6.2 in ./venv/lib/python3.10/site-packages (from num2words->quantulum3->farm-haystack==1.11.0rc0) (0.6.2) Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in ./venv/lib/python3.10/site-packages (from torchvision->sentence-transformers>=2.2.0->farm-haystack==1.11.0rc0) (9.2.0) Requirement already satisfied: smmap<6,>=3.0.1 in ./venv/lib/python3.10/site-packages (from gitdb<5,>=4.0.1->gitpython<4,>=2.1.0->mlflow->farm-haystack==1.11.0rc0) (5.0.0) Requirement already satisfied: MarkupSafe>=2.0 in ./venv/lib/python3.10/site-packages (from Jinja2>=3.0->Flask<3->mlflow->farm-haystack==1.11.0rc0) (2.1.1) Created temporary directory: /tmp/pip-unpack-adou0xlf Building wheels for collected packages: farm-haystack Created temporary directory: /tmp/pip-wheel-ne72esnw Destination directory: /tmp/pip-wheel-ne72esnw Running command Building wheel for farm-haystack (pyproject.toml) Building wheel for farm-haystack (pyproject.toml) ... done Created wheel for farm-haystack: filename=farm_haystack-1.11.0rc0-py3-none-any.whl size=596136 sha256=a45c2eeede973de49996fd3161163ff50fd06f77ceb81462aa372904433bfcc1 Stored in directory: /tmp/pip-ephem-wheel-cache-oh40qiyf/wheels/42/18/4a/ae81f7002e0ea1eaeb5ec6e501efd7d9ec1dec198c9a8dfd32 Successfully built farm-haystack Installing collected packages: farm-haystack Attempting uninstall: farm-haystack Found existing installation: farm-haystack 1.11.0rc0 Uninstalling farm-haystack-1.11.0rc0: Created temporary directory: /home/sara/work/haystack/venv/lib/python3.10/site-packages/~arm_haystack-1.11.0rc0.dist-info Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/farm_haystack-1.11.0rc0.dist-info/ Created temporary directory: /tmp/pip-uninstall-xa450nef Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/__init__.py Created temporary directory: /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/~_pycache__ Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/__pycache__/ Created temporary directory: /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/~ocument_stores Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/document_stores/ Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/environment.py Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/errors.py Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/json-schemas/.gitignore Created temporary directory: /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/json-schemas/~_pycache__ Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/json-schemas/__pycache__/ Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/json-schemas/generate_schema.py Created temporary directory: /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/~odeling Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/modeling/ Created temporary directory: /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/~odes Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/nodes/ Created temporary directory: /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/~ipelines Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/pipelines/ Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/schema.py Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/telemetry.py Created temporary directory: /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/~tils Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/haystack/utils/ Successfully uninstalled farm-haystack-1.11.0rc0 /home/sara/work/haystack/venv/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release warnings.warn( Successfully installed farm-haystack-1.11.0rc0 Removed build tracker: '/tmp/pip-req-tracker-j9ex1r28' → python WARNING:hatch_autorun:Haystack is generating the YAML schema for Pipelines validation. This only happens once, after installing the package. Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> ```

This is how pip install -U . -vvv looks like instead. The autorun code is not executed.

pip log

``` → pip install -U . -vvv Using pip 22.0.2 from /home/sara/work/haystack/venv/lib/python3.10/site-packages/pip (python 3.10) Non-user install because user site-packages disabled Created temporary directory: /tmp/pip-ephem-wheel-cache-47egcqm8 Created temporary directory: /tmp/pip-req-tracker-_4kdilm0 Initialized build tracking at /tmp/pip-req-tracker-_4kdilm0 Created build tracker: /tmp/pip-req-tracker-_4kdilm0 Entered build tracker: /tmp/pip-req-tracker-_4kdilm0 Created temporary directory: /tmp/pip-install-ovlqp6bp Processing /home/sara/work/haystack Added file:///home/sara/work/haystack to build tracker '/tmp/pip-req-tracker-_4kdilm0' Created temporary directory: /tmp/pip-build-env-u6hnh62f Created temporary directory: /tmp/pip-standalone-pip-holsksgc Running command pip subprocess to install build dependencies Using pip 22.0.2 from /tmp/pip-standalone-pip-holsksgc/__env_pip__.zip/pip (python 3.10) Collecting hatchling>=1.8.0 Using cached hatchling-1.11.1-py3-none-any.whl (73 kB) Collecting editables>=0.3 Using cached editables-0.3-py3-none-any.whl (4.7 kB) Collecting packaging>=21.3 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting pathspec>=0.10.1 Using cached pathspec-0.10.1-py3-none-any.whl (27 kB) Collecting tomli>=1.2.2 Using cached tomli-2.0.1-py3-none-any.whl (12 kB) Collecting pluggy>=1.0.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Installing collected packages: tomli, pyparsing, pluggy, pathspec, editables, packaging, hatchling Creating /tmp/pip-build-env-u6hnh62f/overlay/bin changing mode of /tmp/pip-build-env-u6hnh62f/overlay/bin/hatchling to 775 Successfully installed editables-0.3 hatchling-1.11.1 packaging-21.3 pathspec-0.10.1 pluggy-1.0.0 pyparsing-3.0.9 tomli-2.0.1 Installing build dependencies ... done Running command Getting requirements to build wheel Getting requirements to build wheel ... done Created temporary directory: /tmp/pip-standalone-pip-v3fi6dz2 Running command pip subprocess to install backend dependencies Using pip 22.0.2 from /tmp/pip-standalone-pip-v3fi6dz2/__env_pip__.zip/pip (python 3.10) Collecting hatch-autorun Using cached hatch_autorun-1.1.0-py3-none-any.whl (5.3 kB) Collecting hatchling>=1.6.0 Using cached hatchling-1.11.1-py3-none-any.whl (73 kB) Collecting editables>=0.3 Using cached editables-0.3-py3-none-any.whl (4.7 kB) Collecting pathspec>=0.10.1 Using cached pathspec-0.10.1-py3-none-any.whl (27 kB) Collecting pluggy>=1.0.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting tomli>=1.2.2 Using cached tomli-2.0.1-py3-none-any.whl (12 kB) Collecting packaging>=21.3 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Installing collected packages: tomli, pyparsing, pluggy, pathspec, editables, packaging, hatchling, hatch-autorun Creating /tmp/pip-build-env-u6hnh62f/normal/bin changing mode of /tmp/pip-build-env-u6hnh62f/normal/bin/hatchling to 775 Successfully installed editables-0.3 hatch-autorun-1.1.0 hatchling-1.11.1 packaging-21.3 pathspec-0.10.1 pluggy-1.0.0 pyparsing-3.0.9 tomli-2.0.1 Installing backend dependencies ... done Created temporary directory: /tmp/pip-modern-metadata-8__vnsvf Running command Preparing metadata (pyproject.toml) Preparing metadata (pyproject.toml) ... done Source in /home/sara/work/haystack has version 1.11.0rc0, which satisfies requirement farm-haystack==1.11.0rc0 from file:///home/sara/work/haystack Removed farm-haystack==1.11.0rc0 from file:///home/sara/work/haystack from build tracker '/tmp/pip-req-tracker-_4kdilm0' Requirement already satisfied: pydantic in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.10.2) Requirement already satisfied: huggingface-hub>=0.5.0 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (0.10.1) Requirement already satisfied: networkx in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.8.7) Requirement already satisfied: nltk in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (3.7) Requirement already satisfied: torch<1.13,>1.9 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.12.1) Requirement already satisfied: jsonschema in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (4.16.0) Requirement already satisfied: python-docx in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (0.8.11) Requirement already satisfied: mmh3 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (3.0.0) Requirement already satisfied: pandas in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.5.1) Requirement already satisfied: scipy>=1.3.2 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.9.3) Requirement already satisfied: elasticsearch<8,>=7.7 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (7.9.1) Requirement already satisfied: dill in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (0.3.5.1) Requirement already satisfied: transformers==4.21.2 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (4.21.2) Requirement already satisfied: mlflow in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.30.0) Requirement already satisfied: rapidfuzz<2.8.0,>=2.0.15 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.7.0) Requirement already satisfied: sentence-transformers>=2.2.0 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.2.2) Requirement already satisfied: seqeval in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.2.2) Requirement already satisfied: tqdm in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (4.64.1) Requirement already satisfied: tika in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.24) Requirement already satisfied: langdetect in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.0.9) Requirement already satisfied: azure-ai-formrecognizer>=3.2.0b2 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (3.2.0) Requirement already satisfied: requests in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.27.1) Requirement already satisfied: posthog in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (2.1.2) Requirement already satisfied: quantulum3 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (0.7.11) Requirement already satisfied: scikit-learn>=1.0.0 in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (1.1.2) Requirement already satisfied: more-itertools in ./venv/lib/python3.10/site-packages (from farm-haystack==1.11.0rc0) (9.0.0) Requirement already satisfied: pyyaml>=5.1 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (5.4.1) Requirement already satisfied: packaging>=20.0 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (21.3) Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (0.12.1) Requirement already satisfied: numpy>=1.17 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (1.23.4) Requirement already satisfied: regex!=2019.12.17 in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (2022.9.13) Requirement already satisfied: filelock in ./venv/lib/python3.10/site-packages (from transformers==4.21.2->farm-haystack==1.11.0rc0) (3.8.0) Requirement already satisfied: azure-core<2.0.0,>=1.23.0 in ./venv/lib/python3.10/site-packages (from azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (1.26.0) Requirement already satisfied: typing-extensions>=4.0.1 in ./venv/lib/python3.10/site-packages (from azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (4.4.0) Requirement already satisfied: azure-common~=1.1 in ./venv/lib/python3.10/site-packages (from azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (1.1.28) Requirement already satisfied: msrest>=0.6.21 in ./venv/lib/python3.10/site-packages (from azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (0.7.1) Requirement already satisfied: urllib3>=1.21.1 in ./venv/lib/python3.10/site-packages (from elasticsearch<8,>=7.7->farm-haystack==1.11.0rc0) (1.26.12) Requirement already satisfied: certifi in ./venv/lib/python3.10/site-packages (from elasticsearch<8,>=7.7->farm-haystack==1.11.0rc0) (2022.9.24) Requirement already satisfied: jarowinkler<2.0.0,>=1.2.0 in ./venv/lib/python3.10/site-packages (from rapidfuzz<2.8.0,>=2.0.15->farm-haystack==1.11.0rc0) (1.2.3) Requirement already satisfied: threadpoolctl>=2.0.0 in ./venv/lib/python3.10/site-packages (from scikit-learn>=1.0.0->farm-haystack==1.11.0rc0) (3.1.0) Requirement already satisfied: joblib>=1.0.0 in ./venv/lib/python3.10/site-packages (from scikit-learn>=1.0.0->farm-haystack==1.11.0rc0) (1.2.0) Requirement already satisfied: torchvision in ./venv/lib/python3.10/site-packages (from sentence-transformers>=2.2.0->farm-haystack==1.11.0rc0) (0.13.1) Requirement already satisfied: sentencepiece in ./venv/lib/python3.10/site-packages (from sentence-transformers>=2.2.0->farm-haystack==1.11.0rc0) (0.1.97) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in ./venv/lib/python3.10/site-packages (from jsonschema->farm-haystack==1.11.0rc0) (0.18.1) Requirement already satisfied: attrs>=17.4.0 in ./venv/lib/python3.10/site-packages (from jsonschema->farm-haystack==1.11.0rc0) (22.1.0) Requirement already satisfied: six in ./venv/lib/python3.10/site-packages (from langdetect->farm-haystack==1.11.0rc0) (1.16.0) Requirement already satisfied: gitpython<4,>=2.1.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (3.1.29) Requirement already satisfied: querystring-parser<2 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (1.2.4) Requirement already satisfied: sqlalchemy<2,>=1.4.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (1.4.42) Requirement already satisfied: docker<7,>=4.0.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (6.0.0) Requirement already satisfied: prometheus-flask-exporter<1 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (0.20.3) Requirement already satisfied: gunicorn<21 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (20.1.0) Requirement already satisfied: pytz<2023 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (2022.5) Requirement already satisfied: protobuf<5,>=3.12.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (3.20.1) Requirement already satisfied: cloudpickle<3 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (2.2.0) Requirement already satisfied: importlib-metadata!=4.7.0,<6,>=3.7.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (4.13.0) Requirement already satisfied: Flask<3 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (2.2.2) Requirement already satisfied: click<9,>=7.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (8.0.4) Requirement already satisfied: alembic<2 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (1.8.1) Requirement already satisfied: sqlparse<1,>=0.4.0 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (0.4.3) Requirement already satisfied: databricks-cli<1,>=0.8.7 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (0.17.3) Requirement already satisfied: entrypoints<1 in ./venv/lib/python3.10/site-packages (from mlflow->farm-haystack==1.11.0rc0) (0.4) Requirement already satisfied: python-dateutil>=2.8.1 in ./venv/lib/python3.10/site-packages (from pandas->farm-haystack==1.11.0rc0) (2.8.2) Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.10/site-packages (from requests->farm-haystack==1.11.0rc0) (3.4) Requirement already satisfied: charset-normalizer~=2.0.0 in ./venv/lib/python3.10/site-packages (from requests->farm-haystack==1.11.0rc0) (2.0.12) Requirement already satisfied: backoff<2.0.0,>=1.10.0 in ./venv/lib/python3.10/site-packages (from posthog->farm-haystack==1.11.0rc0) (1.11.1) Requirement already satisfied: monotonic>=1.5 in ./venv/lib/python3.10/site-packages (from posthog->farm-haystack==1.11.0rc0) (1.6) Requirement already satisfied: lxml>=2.3.2 in ./venv/lib/python3.10/site-packages (from python-docx->farm-haystack==1.11.0rc0) (4.9.1) Requirement already satisfied: num2words in ./venv/lib/python3.10/site-packages (from quantulum3->farm-haystack==1.11.0rc0) (0.5.12) Requirement already satisfied: inflect in ./venv/lib/python3.10/site-packages (from quantulum3->farm-haystack==1.11.0rc0) (6.0.2) Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from tika->farm-haystack==1.11.0rc0) (59.6.0) Requirement already satisfied: Mako in ./venv/lib/python3.10/site-packages (from alembic<2->mlflow->farm-haystack==1.11.0rc0) (1.2.3) Requirement already satisfied: tabulate>=0.7.7 in ./venv/lib/python3.10/site-packages (from databricks-cli<1,>=0.8.7->mlflow->farm-haystack==1.11.0rc0) (0.9.0) Requirement already satisfied: pyjwt>=1.7.0 in ./venv/lib/python3.10/site-packages (from databricks-cli<1,>=0.8.7->mlflow->farm-haystack==1.11.0rc0) (2.6.0) Requirement already satisfied: oauthlib>=3.1.0 in ./venv/lib/python3.10/site-packages (from databricks-cli<1,>=0.8.7->mlflow->farm-haystack==1.11.0rc0) (3.2.2) Requirement already satisfied: websocket-client>=0.32.0 in ./venv/lib/python3.10/site-packages (from docker<7,>=4.0.0->mlflow->farm-haystack==1.11.0rc0) (1.4.1) Requirement already satisfied: Werkzeug>=2.2.2 in ./venv/lib/python3.10/site-packages (from Flask<3->mlflow->farm-haystack==1.11.0rc0) (2.2.2) Requirement already satisfied: itsdangerous>=2.0 in ./venv/lib/python3.10/site-packages (from Flask<3->mlflow->farm-haystack==1.11.0rc0) (2.1.2) Requirement already satisfied: Jinja2>=3.0 in ./venv/lib/python3.10/site-packages (from Flask<3->mlflow->farm-haystack==1.11.0rc0) (3.1.2) Requirement already satisfied: gitdb<5,>=4.0.1 in ./venv/lib/python3.10/site-packages (from gitpython<4,>=2.1.0->mlflow->farm-haystack==1.11.0rc0) (4.0.9) Requirement already satisfied: zipp>=0.5 in ./venv/lib/python3.10/site-packages (from importlib-metadata!=4.7.0,<6,>=3.7.0->mlflow->farm-haystack==1.11.0rc0) (3.10.0) Requirement already satisfied: isodate>=0.6.0 in ./venv/lib/python3.10/site-packages (from msrest>=0.6.21->azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (0.6.1) Requirement already satisfied: requests-oauthlib>=0.5.0 in ./venv/lib/python3.10/site-packages (from msrest>=0.6.21->azure-ai-formrecognizer>=3.2.0b2->farm-haystack==1.11.0rc0) (1.3.1) Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./venv/lib/python3.10/site-packages (from packaging>=20.0->transformers==4.21.2->farm-haystack==1.11.0rc0) (3.0.9) Requirement already satisfied: prometheus-client in ./venv/lib/python3.10/site-packages (from prometheus-flask-exporter<1->mlflow->farm-haystack==1.11.0rc0) (0.15.0) Requirement already satisfied: greenlet!=0.4.17 in ./venv/lib/python3.10/site-packages (from sqlalchemy<2,>=1.4.0->mlflow->farm-haystack==1.11.0rc0) (1.1.3.post0) Requirement already satisfied: docopt>=0.6.2 in ./venv/lib/python3.10/site-packages (from num2words->quantulum3->farm-haystack==1.11.0rc0) (0.6.2) Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in ./venv/lib/python3.10/site-packages (from torchvision->sentence-transformers>=2.2.0->farm-haystack==1.11.0rc0) (9.2.0) Requirement already satisfied: smmap<6,>=3.0.1 in ./venv/lib/python3.10/site-packages (from gitdb<5,>=4.0.1->gitpython<4,>=2.1.0->mlflow->farm-haystack==1.11.0rc0) (5.0.0) Requirement already satisfied: MarkupSafe>=2.0 in ./venv/lib/python3.10/site-packages (from Jinja2>=3.0->Flask<3->mlflow->farm-haystack==1.11.0rc0) (2.1.1) Created temporary directory: /tmp/pip-unpack-yqkh0f6_ Building wheels for collected packages: farm-haystack Created temporary directory: /tmp/pip-wheel-v5tlzf0t Destination directory: /tmp/pip-wheel-v5tlzf0t Running command Building wheel for farm-haystack (pyproject.toml) Building wheel for farm-haystack (pyproject.toml) ... done Created wheel for farm-haystack: filename=farm_haystack-1.11.0rc0-py3-none-any.whl size=596136 sha256=a45c2eeede973de49996fd3161163ff50fd06f77ceb81462aa372904433bfcc1 Stored in directory: /tmp/pip-ephem-wheel-cache-47egcqm8/wheels/42/18/4a/ae81f7002e0ea1eaeb5ec6e501efd7d9ec1dec198c9a8dfd32 Successfully built farm-haystack Installing collected packages: farm-haystack Attempting uninstall: farm-haystack Found existing installation: farm-haystack 1.11.0rc0 Uninstalling farm-haystack-1.11.0rc0: Created temporary directory: /home/sara/work/haystack/venv/lib/python3.10/site-packages/~arm_haystack-1.11.0rc0.dist-info Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/farm_haystack-1.11.0rc0.dist-info/ Created temporary directory: /tmp/pip-uninstall-sfzzg4g4 Removing file or directory /home/sara/work/haystack/venv/lib/python3.10/site-packages/farm_haystack.pth Successfully uninstalled farm-haystack-1.11.0rc0 Successfully installed farm-haystack-1.11.0rc0 Removed build tracker: '/tmp/pip-req-tracker-_4kdilm0' → python Python 3.10.6 (main, Aug 10 2022, 11:35:20) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> ```

The package seems to be re-installed in both cases, so -U doesn't seem to be preventing the re-installation of Haystack. But I might have missed something. Leads welcome :slightly_smiling_face:

[masci]

With https://github.com/deepset-ai/haystack/pull/3533 we changed strategy, generating the schema lazily instead of at startup, closing as not valid.