Notebook execution fails because required env variable isn't defined even though it is defined

ptitzler commented 3 years ago

1.2.1

Scenario:

import the content of the attached archive
In JL open the hello_world directory
- Open the hello_world-notebook.pipeline. It comprises of one node, which requires an environment variable named DATASET_URL. Note that the variable is defined.

"nodes": [
        {
          "id": "8ce8e5de-239b-42bb-a99d-32bf868ebe77",
          "type": "execution_node",
          "op": "execute-notebook-node",
          "app_data": {
            "filename": "load_data.ipynb",
            "runtime_image": "amancevice/pandas:1.1.1",
            "env_vars": [
              "DATASET_URL=https://dax-cdn.cdn.appdomain.cloud/dax-noaa-weather-data-jfk-airport/1.1.4/noaa-weather-data-jfk-airport.tar.gz"
            ],

The notebook fails to execute locally in code cell 2 because the variable is not defined. This issue was originally by another user. I have been unable to reproduce it using the attached artifacts and have requested additional information.

hello_world.tar.gz

ptitzler commented 3 years ago

pip freeze output:

aiohttp==3.6.2
ansiwrap==0.8.4
appdirs==1.4.4
async-generator==1.10
async-timeout==3.0.1
atpublic==1.0
attrs==19.3.0
autopep8==1.5.4
backcall==0.1.0
black==20.8b1
bleach==3.1.5
bump2version==1.0.0
bumpversion==0.6.0
cachetools==4.1.1
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
click==7.1.2
cloudpickle==1.6.0
colorama==0.4.3
configobj==5.0.6
configparser==5.0.0
coverage==5.3
cryptography==3.1.1
cycler==0.10.0
dataclasses==0.7
decorator==4.4.2
defusedxml==0.6.0
Deprecated==1.2.10
distlib==0.3.0
distro==1.5.0
dnf==0.0.1
docutils==0.16
dpath==2.0.1
dvc==1.0.0
elyra==1.2.1
entrypoints==0.3
filelock==3.0.12
flake8==3.8.3
flatten-json==0.1.7
flufl.lock==3.2
funcy==1.14
future==0.18.2
gitdb==4.0.5
GitPython==3.1.3
google-api-core==1.22.2
google-auth==1.22.0
google-cloud-core==1.4.1
google-cloud-storage==1.31.2
google-crc32c==1.0.0
google-resumable-media==1.0.0
googleapis-common-protos==1.52.0
grandalf==0.6
humanize==2.4.0
idna==2.9
idna-ssl==1.1.0
importlib-metadata==1.6.0
importlib-resources==1.5.0
inflect==3.0.2
ipykernel==5.3.0
ipython==7.15.0
ipython-genutils==0.2.0
jedi==0.17.0
jeepney==0.4.3
Jinja2==2.11.2
json5==0.9.5
jsonpath-ng==1.5.1
jsonschema==3.2.0
jupyter-client==6.1.3
jupyter-core==4.6.3
jupyter-kernel-gateway==2.4.0
jupyterlab==2.2.8
jupyterlab-git==0.21.1
jupyterlab-server==1.2.0
keyring==21.4.0
kfp==1.0.0
kfp-notebook==0.12.0
kfp-server-api==1.0.1
kiwisolver==1.2.0
kubernetes==11.0.0
MarkupSafe==1.1.1
matplotlib==3.3.2
mccabe==0.6.1
minio==6.0.0
mistune==0.8.4
multidict==4.7.6
mypy-extensions==0.4.3
nanotime==0.5.2
nbclient==0.5.0
nbconvert==5.6.1
nbdime==2.0.0
nbformat==5.0.6
nest-asyncio==1.4.1
networkx==2.3
notebook==6.0.3
numpy==1.18.5
oauthlib==3.1.0
packaging==20.4
panda==0.3.1
pandas==1.0.4
pandocfilters==1.4.2
papermill==2.2.0
parso==0.7.0
pathspec==0.8.0
pathtools==0.1.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.1.2
pkginfo==1.5.0.1
pluggy==0.13.1
ply==3.11
prometheus-client==0.8.0
prompt-toolkit==3.0.5
protobuf==3.13.0
ptyprocess==0.6.0
py==1.9.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.6.0
pycparser==2.20
pydot==1.4.1
pyflakes==2.2.0
PyGithub==1.53
Pygments==2.6.1
pygtrie==2.3.2
PyJWT==1.7.1
pyparsing==2.4.7
pyrsistent==0.16.0
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
pyzmq==19.0.1
readme-renderer==26.0
regex==2020.9.27
requests==2.24.0
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
rfc3986==1.4.0
rfc3986-validator==0.1.1
rsa==4.6
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.0
scipy==1.5.2
seaborn==0.11.0
SecretStorage==3.1.2
Send2Trash==1.5.0
shortuuid==1.0.1
shtab==1.0.2
six==1.15.0
smmap==3.0.4
strip-hints==0.1.9
svgwrite==1.4
tabulate==0.8.7
tenacity==6.2.0
terminado==0.8.3
testpath==0.4.4
texttable==1.6.2
textwrap3==0.9.2
toml==0.10.1
tornado==6.0.4
tox==3.20.0
tqdm==4.46.1
traitlets==4.3.3
Tree==0.2.4
treelib==1.6.1
twine==3.2.0
typed-ast==1.4.1
typing-extensions==3.7.4.3
urllib3==1.25.9
virtualenv==20.0.21
voluptuous==0.11.7
watchdog==0.10.3
wcwidth==0.2.2
webencodings==0.5.1
websocket-client==0.57.0
wrapt==1.12.1
yarl==1.6.0
zc.lockfile==2.0
zipp==3.1.0

(py36-venv) [root@featuretest test001]# jupyter serverextension list
config dir: /root/pydev/py36-venv/etc/jupyter
    elyra  enabled 
    - Validating...
      elyra 1.2.1 OK
    jupyterlab  enabled 
    - Validating...
      jupyterlab 2.2.8 OK
    jupyterlab_git  enabled 
    - Validating...
      jupyterlab_git 0.21.1 OK
    nbdime  enabled 
    - Validating...
      nbdime 2.0.0 OK

(py36-venv) [root@featuretest test001]# jupyter labextension list 
JupyterLab v2.2.8
Known labextensions:
   app dir: /root/pydev/py36-venv/share/jupyter/lab
        @elyra/code-snippet-extension v1.2.1  enabled  OK
        @elyra/metadata-extension v1.2.1  enabled  OK
        @elyra/pipeline-editor-extension v1.2.1  enabled  OK
        @elyra/python-editor-extension v1.2.1  enabled  OK
        @elyra/theme-extension v1.2.1  enabled  OK
        @jupyterlab/git v0.21.1  enabled  OK
        @jupyterlab/toc v4.0.0  enabled  OK
        nbdime-jupyterlab v2.0.0  enabled  OK
Disabled core extensions:
    @jupyterlab/launcher-extension

ptitzler commented 3 years ago

The log output with debug enabled:

Note the payload, which contains the env variable definition.

JSON payload: {'doc_type': 'pipeline', 'version': '3.0', 'json_schema': 'http://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json', 'id': 'adaedb50-80c9-45db-b90f-33de2e9c02f0', 'primary_pipeline': '65e5daf7-6ecb-4f45-9e23-e469cb701a38', 'pipelines': [{'id': '65e5daf7-6ecb-4f45-9e23-e469cb701a38', 'nodes': [{'id': '89911417-fb9e-4298-a39b-8599e034b723', 'type': 'execution_node', 'op': 'execute-notebook-node', 'app_data': {'filename': 'examples/pipelines/hello_world/load_data.ipynb', 'runtime_image': 'amancevice/pandas:1.1.1', 'env_vars': ['DATASET_URL=https://dax-cdn.cdn.appdomain.cloud/dax-noaa-weather-data-jfk-airport/1.1.4/noaa-weather-data-jfk-airport.tar.gz'], 'include_subdirectories': False, 'outputs': ['data/noaa-weather-data-jfk-airport/jfk_weather.csv'], 'dependencies': [], 'invalidNodeError': None, 'ui_data': {'label': 'load_data.ipynb', 'image': 'data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20width%3D%2216%22%20viewBox%3D%220%200%2022%2022%22%3E%0A%20%20%3Cg%20class%3D%22jp-icon-warn0%20jp-icon-selectable%22%20fill%3D%22%23EF6C00%22%3E%0A%20%20%20%20%3Cpath%20d%3D%22M18.7%203.3v15.4H3.3V3.3h15.4m1.5-1.5H1.8v18.3h18.3l.1-18.3z%22%2F%3E%0A%20%20%20%20%3Cpath%20d%3D%22M16.5%2016.5l-5.4-4.3-5.6%204.3v-11h11z%22%2F%3E%0A%20%20%3C%2Fg%3E%0A%3C%2Fsvg%3E%0A', 'x_pos': 192, 'y_pos': 115, 'description': 'Notebook file'}}, 'inputs': [{'id': 'inPort', 'app_data': {'ui_data': {'cardinality': {'min': 0, 'max': 1}, 'label': 'Input Port'}}}], 'outputs': [{'id': 'outPort', 'app_data': {'ui_data': {'cardinality': {'min': 0, 'max': -1}, 'label': 'Output Port'}}}]}], 'app_data': {'ui_data': {'comments': []}, 'version': 3, 'name': 'test', 'runtime': 'local', 'runtime-config': 'local'}, 'runtime_ref': ''}], 'schemas': []}

[D 16:05:35.593 LabApp] 200 POST /git/status?1601672735511 (127.0.0.1) 79.09ms
[D 16:05:36.523 LabApp] Pipeline SchedulerHandler now executing post request
[D 16:05:36.523 LabApp] JSON payload: {'doc_type': 'pipeline', 'version': '3.0', 'json_schema': 'http://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json', 'id': 'adaedb50-80c9-45db-b90f-33de2e9c02f0', 'primary_pipeline': '65e5daf7-6ecb-4f45-9e23-e469cb701a38', 'pipelines': [{'id': '65e5daf7-6ecb-4f45-9e23-e469cb701a38', 'nodes': [{'id': '89911417-fb9e-4298-a39b-8599e034b723', 'type': 'execution_node', 'op': 'execute-notebook-node', 'app_data': {'filename': 'examples/pipelines/hello_world/load_data.ipynb', 'runtime_image': 'amancevice/pandas:1.1.1', 'env_vars': ['DATASET_URL=https://dax-cdn.cdn.appdomain.cloud/dax-noaa-weather-data-jfk-airport/1.1.4/noaa-weather-data-jfk-airport.tar.gz'], 'include_subdirectories': False, 'outputs': ['data/noaa-weather-data-jfk-airport/jfk_weather.csv'], 'dependencies': [], 'invalidNodeError': None, 'ui_data': {'label': 'load_data.ipynb', 'image': 'data:image/svg+xml;utf8,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20width%3D%2216%22%20viewBox%3D%220%200%2022%2022%22%3E%0A%20%20%3Cg%20class%3D%22jp-icon-warn0%20jp-icon-selectable%22%20fill%3D%22%23EF6C00%22%3E%0A%20%20%20%20%3Cpath%20d%3D%22M18.7%203.3v15.4H3.3V3.3h15.4m1.5-1.5H1.8v18.3h18.3l.1-18.3z%22%2F%3E%0A%20%20%20%20%3Cpath%20d%3D%22M16.5%2016.5l-5.4-4.3-5.6%204.3v-11h11z%22%2F%3E%0A%20%20%3C%2Fg%3E%0A%3C%2Fsvg%3E%0A', 'x_pos': 192, 'y_pos': 115, 'description': 'Notebook file'}}, 'inputs': [{'id': 'inPort', 'app_data': {'ui_data': {'cardinality': {'min': 0, 'max': 1}, 'label': 'Input Port'}}}], 'outputs': [{'id': 'outPort', 'app_data': {'ui_data': {'cardinality': {'min': 0, 'max': -1}, 'label': 'Output Port'}}}]}], 'app_data': {'ui_data': {'comments': []}, 'version': 3, 'name': 'test', 'runtime': 'local', 'runtime-config': 'local'}, 'runtime_ref': ''}], 'schemas': []}
[D 16:05:36.525 LabApp] Adding operation for 'examples/pipelines/hello_world/load_data.ipynb' to pipeline: test
[I 16:05:36.525 LabApp] Processing Pipeline : test
[D 16:05:36.526 LabApp] Processing: /root/pydev/py36-venv/bin/examples/pipelines/hello_world/load_data.ipynb
Input Notebook:  /root/pydev/py36-venv/bin/examples/pipelines/hello_world/load_data.ipynb
Output Notebook: /root/pydev/py36-venv/bin/examples/pipelines/hello_world/load_data.ipynb
Working directory: /root/pydev/py36-venv/bin/examples/pipelines/hello_world
Executing:   0%|                                                                                                                                                                          | 0/11 [00:00<?, ?cell/s][D 16:05:38.157 LabApp] 200 GET /metrics?1601672738136 (127.0.0.1) 16.31ms
[D 16:05:38.682 LabApp.jupyterlab_git] Execute ['git', 'for-each-ref', '--format=%(refname:short)%09%(objectname)%09%(upstream:short)%09%(HEAD)', 'refs/heads/'] in /root/pydev/py36-venv/bin/examples.
[D 16:05:38.699 LabApp.jupyterlab_git] Code: 0
    Output: master  d2253ecc2f5a555b0f8735716cd5b7be1ef27375    origin/master   *

    Error: 
[D 16:05:38.699 LabApp.jupyterlab_git] Execute ['git', 'for-each-ref', '--format=%(refname:short)%09%(objectname)', 'refs/remotes/'] in /root/pydev/py36-venv/bin/examples.
[D 16:05:38.716 LabApp.jupyterlab_git] Code: 0
    Output: origin/HEAD d2253ecc2f5a555b0f8735716cd5b7be1ef27375
    origin/master   d2253ecc2f5a555b0f8735716cd5b7be1ef27375

    Error: 
[D 16:05:38.717 LabApp] 200 POST /git/branch?1601672738677 (127.0.0.1) 36.96ms
[D 16:05:38.735 LabApp.jupyterlab_git] Execute ['git', 'status', '--porcelain', '-u', '-z'] in /root/pydev/py36-venv/bin/examples.
[D 16:05:38.754 LabApp.jupyterlab_git] Code: 0
    Output:  M .gitattributes M .github/pull_request_template.md M .gitignore M CONTRIBUTING.md M LICENSE M README.md M binder/getting-started/code-snippets/inspect_df.json M binder/getting-started/code-snippets/mit_license_text.json M binder/getting-started/getting_started.md M binder/getting-started/getting_started_notebook.ipynb M binder/getting-started/getting_started_python.py M binder/getting-started/images/consume_code_snippet.png M binder/getting-started/images/create_code_snippet.png M...
    Error: 
[D 16:05:38.755 LabApp.jupyterlab_git] Execute ['git', 'diff', '--numstat', '-z', '--cached', '4b825dc642cb6eb9a060e54bf8d69288fbee4904'] in /root/pydev/py36-venv/bin/examples.
[D 16:05:38.816 LabApp.jupyterlab_git] Code: 0
    Output: 29  0   .gitattributes27    0   .github/pull_request_template.md93  0   .gitignore26    0   CONTRIBUTING.md242  0   LICENSE31   0   README.md21 0   binder/getting-started/code-snippets/inspect_df.json13  0   binder/getting-started/code-snippets/mit_license_text.json115   0   binder/getting-started/getting_started.md144    0   binder/getting-started/getting_started_notebook.ipynb52 0   binder/getting-started/getting_started_python.py-   -   binder/getting-started/images/consume_code_snippet.png- -   binder/getting-started/images...
    Error: 
[D 16:05:38.819 LabApp] 200 POST /git/status?1601672738730 (127.0.0.1) 86.33ms
Executing notebook with kernel: python3
Executing:  18%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d                                                                                                                                    | 2/11 [00:01<00:09,  1.05s/cell][D 16:05:40.268 LabApp] 200 GET /api/terminals?1601672740263 (127.0.0.1) 2.20ms
Executing: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 11/11 [00:03<00:00,  3.25cell/s]
Traceback (most recent call last):
  File "/root/pydev/py36-venv/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/papermill/cli.py", line 256, in papermill
    execution_timeout=execution_timeout,
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/papermill/execute.py", line 106, in execute_notebook
    **engine_kwargs
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/papermill/engines.py", line 343, in execute_notebook
    cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/papermill/engines.py", line 402, in execute_managed_notebook
    return PapermillNotebookClient(nb_man, **final_kwargs).execute()
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/papermill/clientwrap.py", line 48, in execute
    self.set_widgets_metadata()
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/nbclient/client.py", line 439, in setup_kernel
    self._cleanup_kernel()
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/root/pydev/py36-venv/lib64/python3.6/site-packages/nbclient/client.py", line 358, in _async_cleanup_kernel
    await ensure_async(self.km.cleanup_resources())
AttributeError: 'AsyncKernelManager' object has no attribute 'cleanup_resources'
[E 16:05:41.459 LabApp] Internal error executing /root/pydev/py36-venv/bin/examples/pipelines/hello_world/load_data.ipynb
[E 16:05:41.460 LabApp] Uncaught exception POST /elyra/pipeline/schedule?1601672736518 (127.0.0.1)
    HTTPServerRequest(protocol='http', host='localhost:8963', method='POST', uri='/elyra/pipeline/schedule?1601672736518', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/root/pydev/py36-venv/lib64/python3.6/site-packages/elyra/pipeline/processor_local.py", line 151, in process
        subprocess.run(argv, cwd=file_dir, env=envs, check=True)
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/subprocess.py", line 438, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['papermill', '/root/pydev/py36-venv/bin/examples/pipelines/hello_world/load_data.ipynb', '/root/pydev/py36-venv/bin/examples/pipelines/hello_world/load_data.ipynb', '--cwd', '/root/pydev/py36-venv/bin/examples/pipelines/hello_world']' returned non-zero exit status 1.

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):

ptitzler commented 3 years ago

Not sure why, but in this output the failure was caused by AttributeError: 'AsyncKernelManager' object has no attribute 'cleanup_resources', which hints at another broken case of dependency mismatch @lresende @kevin-bates

fyi @ibuleje

kevin-bates commented 3 years ago

Please upgrade juptyer_client: pip install --upgrade jupyter_client

ptitzler commented 3 years ago

Confirmed that after the upgrade the pipeline works as expected with Jupyter debug enabled and disabled 🥳

lresende commented 3 years ago

Let me start by confessing that I only saw the logs available as plain text in the issue. How did we come about to say this was related to environment variables? Were there suggestions about it in the tar.gz?

kevin-bates commented 3 years ago

How did we come about to say this was related to environment variables?

Patrick had come to me previously about this and the original failure was because the notebook code was enforcing the constraint that DATASETURL exist has an env per the code in the cell. We went through the normal approach - make sure the env variable is configured on the node properties, etc. That all checked out. So I noticed that when DEBUG is enabled, we log the pipeline JSON (and configured envs), so I asked that the server be restarted with DEBUG logging enabled - and it was.

It was at that time that we found a) the JSON did indeed reflect that DATASET_URL was configured on the pipeline. However, it was at that time that we ran into the papermill->nbclient->jupyter_client dependency issue - where I requested that jupyter_client be upgraded. Once upgraded, the original issue didn't reproduce. Since that was with DEBUG, I suggested restoring the logging settings to no DEBUG, and even then, the issue did not reproduce.

This, IMO, implies that there's something in papermill that will prevent the correct environment from getting instantiated, yet, bypass (ignore) issues related to ModuleNotFound kinds of things. ... I just tried reproducing this by disabling DEBUG, downgrading jupyter_client==6.1.3 and running a pipeline that was missing an env, and can only reproduce the transitive dependency issue relative to jupyter_client - so I don't know if it just so happened that that issue got introduced (unknowingly) but it might be good to determine if there's a hidden issue until DEBUG is enabled.

elyra-ai / elyra

Notebook execution fails because required env variable isn't defined even though it is defined #954