Got dependency issue with spacy and markupsafe

[xwang233]

After this PR https://github.com/pytorch/benchmark/pull/1261 change https://github.com/pytorch/benchmark/commit/94078d9288ff99ef50557a3bd87970badbef0458#diff-fcf1c48cd82709b9bd6dc7881e2a425d15502a75925097be18583a7c6a0ffdafR5, while doing python install.py, I got

root@f3b64780b5e8:/workspace/torch-benchmark/benchmark# python install.py
checking packages torch, torchvision, torchtext, torchaudio are installed...OK
checking out Git LFS files...OK
decompressing input tarball: tacotron2-minimal.tar.gz...OK
decompressing input tarball: LearningToPaint_inputs.tar.gz...OK
decompressing input tarball: Background_Matting_inputs.tar.gz...OK
decompressing input tarball: Super_SloMo_inputs.tar.gz...OK
decompressing input tarball: pytorch_stargan_inputs.tar.gz...OK
decompressing input tarball: coco2017-minimal.tar.gz...OK
decompressing input tarball: pytorch_CycleGAN_and_pix2pix_inputs.tar.gz...OK
decompressing input tarball: coco128.tar.gz...OK
decompressing input tarball: multi30k.tar.gz...OK
decompressing input tarball: speech_transformer_inputs.tar.gz...OK
running setup for /opt/pytorch/benchmark/torchbenchmark/models/BERT_pytorch...OK
running setup for /opt/pytorch/benchmark/torchbenchmark/models/Background_Matting...OK
running setup for /opt/pytorch/benchmark/torchbenchmark/models/DALLE2_pytorch...OK
running setup for /opt/pytorch/benchmark/torchbenchmark/models/LearningToPaint...OK
running setup for /opt/pytorch/benchmark/torchbenchmark/models/Super_SloMo...OK
running setup for /opt/pytorch/benchmark/torchbenchmark/models/alexnet...OK
running setup for /opt/pytorch/benchmark/torchbenchmark/models/attention_is_all_you_need_pytorch...FAIL
Error for /opt/pytorch/benchmark/torchbenchmark/models/attention_is_all_you_need_pytorch:
---------------------------------------------------------------------------
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/conda/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/opt/conda/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/opt/conda/lib/python3.8/site-packages/spacy/__init__.py", line 15, in <module>
    from .cli.info import info  # noqa: F401
  File "/opt/conda/lib/python3.8/site-packages/spacy/cli/__init__.py", line 17, in <module>
    from .debug_diff import debug_diff  # noqa: F401
  File "/opt/conda/lib/python3.8/site-packages/spacy/cli/debug_diff.py", line 10, in <module>
    from .init_config import init_config, Optimizations
  File "/opt/conda/lib/python3.8/site-packages/spacy/cli/init_config.py", line 8, in <module>
    from jinja2 import Template
  File "/opt/conda/lib/python3.8/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/opt/conda/lib/python3.8/site-packages/jinja2/environment.py", line 25, in <module>
    from .defaults import BLOCK_END_STRING
  File "/opt/conda/lib/python3.8/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
  File "/opt/conda/lib/python3.8/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/opt/conda/lib/python3.8/site-packages/markupsafe/__init__.py)
Traceback (most recent call last):
  File "install.py", line 22, in <module>
    spacy_download('en_core_web_sm')
  File "install.py", line 10, in spacy_download
    subprocess.check_call([sys.executable, '-m', 'spacy', 'download', language])
  File "/opt/conda/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-m', 'spacy', 'download', 'en_core_web_sm']' returned non-zero exit status 1.

---------------------------------------------------------------------------

Traceback (most recent call last):
  File "install.py", line 134, in <module>
    raise RuntimeError("Failed to complete setup")
RuntimeError: Failed to complete setup

There is a similar issue https://github.com/aws/aws-sam-cli/issues/3661#issuecomment-1043965261.

My temporary workaround is to pin the package version markupsafe==2.0.1 like this https://github.com/xwang233/benchmark/commit/3dcf675debbe5a6d55e23e15c95be9618fbf620c

cc @xuzhao9

[xuzhao9]

Strictly speaking, this is not dependency issue with spacy/markupsafe, but jinja2 and markupsafe.

I am okay to pin the markupsafe package version to 2.0.1. It is weird that this error escaped our CI test though:

• Python 3.10: https://github.com/pytorch/benchmark/actions/runs/3341400057/jobs/5532585292
• Python 3.8: https://app.circleci.com/pipelines/github/pytorch/benchmark/5525/workflows/5eab99f4-92f2-4e78-ae5d-8a64ba0c4dbc/jobs/5675

[xuzhao9]

@xwang233 Is it because the jinja2 version in your local docker is old? The latest 3.1.x version doesn't import soft_unicode from markupsafe: https://github.com/pallets/jinja/blob/main/src/jinja2/filters.py#L13

[xwang233]

It could be the reason. Our CI is using jinja2==2.11.3. It could be pinned be another conda package in our environment.

[xuzhao9]

@xwang233 imo we should pin jinja2 version here instead of markupsafe version. I can review a PR on that.

[xuzhao9]

@xwang233 just want to confirm, are you going to submit a PR?

[xwang233]

Hi @xuzhao9 , I won't submit a PR about this since it's not an upstream issue but actually an issue in our environment. I'll fix our environment instead.

[xuzhao9]

Closed as this is downstream environment issue.