Closed grantav closed 3 years ago
DonJayamanne
commented
3 years ago Thanks for filling this issue. Please could you let me know what metadata is not being saved. Can you give some details and screenshots.
Please could you upload the ipynb as well.
grantav
commented
3 years ago It was kernel spec. I will screenshot the difference between the 2 files
Sent from my iPhone
On 17 Apr 2021, at 00:16, Don Jayamanne @.***> wrote:
Thanks for filling this issue. Please could you let me know what metadata is not being saved. Can you give some details and screenshots.
Please could you upload the ipynb as well.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.
grantav
commented
3 years ago Please see attached documents.
The metadata section of the notebooks and a diff between the 2 saved in jupyter and in vs code.
On Sun, 18 Apr 2021 at 13:54, Grant Vorenberg @.***> wrote:
It was kernel spec. I will screenshot the difference between the 2 files
Sent from my iPhone
On 17 Apr 2021, at 00:16, Don Jayamanne @.***> wrote:
Thanks for filling this issue. Please could you let me know what metadata is not being saved. Can you give some details and screenshots.
Please could you upload the ipynb as well.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-jupyter/issues/5524#issuecomment-821568258, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADNN3ZJZNLAFETT5BKWLDMLTJCSLLANCNFSM426Z6IBA .
"metadata": { "celltoolbar": "Tags", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }
"display_name": "Python 3.8.3 64-bit ('base': conda)", 286c286 < "name": "python3"
"name": "python383jvsc74a57bd0fc2f79212b9aaab49a9398475eca83543810af022be1929a600f8cb43b6e382b" 303c303 < }
} \ No newline at end of file
"metadata": { "celltoolbar": "Tags", "kernelspec": { "display_name": "Python 3.8.3 64-bit ('base': conda)", "language": "python", "name": "python383jvsc74a57bd0fc2f79212b9aaab49a9398475eca83543810af022be1929a600f8cb43b6e382b" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 4 }
DonJayamanne
commented
3 years ago Thanks for the information. I'd like to clarify my understanding of the issue. The problem is as follows:
"display_name": "Python 3", but now its "Python 3.8.3 64-bit ('base': conda)"
Python 3.5.3 64-bit (base conda) environment. Jupyter doesn't natively support Conda Environments without the use of an extension. And if you did install that, then you'd see similar changes in the display name.python3 and now its python383jvsc74a57bd0fc2f79212b9aaab49a9398475eca83543810af022be1929a600f8cb43b6e382b
I.e. metadata is not missing, but it's different. Please could you confirm this.
grantav
commented
3 years ago I will review and revert.
Sent from my iPhone
On 20 Apr 2021, at 19:05, Don Jayamanne @.***> wrote:
Thanks for the information. I'd like to clarify my understanding of the issue. The problem is as follows:
Display name used to be "display_name": "Python 3", but now its "Python 3.8.3 64-bit ('base': conda)" This is because when you ran a cell against this notebook, you selected Python 3.5.3 64-bit (base conda) environment. Jupyter doesn't natively support Conda Environments without the use of an extension. And if you did install that, then you'd see similar changes in the display name. Name used to be python3 and now its python383jvsc74a57bd0fc2f79212b9aaab49a9398475eca83543810af022be1929a600f8cb43b6e382b Again, this is similar to the previous response. However we're looking at not changing this if you use the current default interpreter in VS Code. That being said if the kernel seleced is not the default interpreter, it will get updated. This is because Jupyter doesn't natively support all interpreters on your machine (e.g. Conda, pipenv, etc) it only supports kernels. To ensure users opening a notebook in VS Code can run against the same kernel, we need to save the name of the interpreter (kernel), just as Jupyter does. Hence the metadata is updated. A new line is missing Duplicate of #5428 I.e. metadata is not missing, but it's different. Please could you confirm this.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.
grantav
commented
3 years ago Ok, I think I have wrapped my head around the problem. You are correct, it is not missing metadate. Never would've thought I would say this but, the kernel spec "is too specific" for widespread use (in our use case).
The code runs in different environments/infrastructure according to locale, cost centres, responsibility and most importantly functional and system requirements and dependencies. These systems are all compatible but they obviously differ by minor versions based on their respective requirements and needs. The idea is to have a consolidated image for the containers but some requirements are mutually exclusive at this point in time.
Being too specific when saving the exact kernel of the dev platform, which then goes into the PR, ultimately breaks the workflows and pipelines on the various systems when the code is deployed. I have implemented a sed script to fix this issue but it is a hack. I need something more robust.
I am sure there are many ways to "skin the cat" but I need to take into account the change in the way my team works. I need to find a way to configure the local vscode environment to allow for the specification of a generic kernel (as before) or maybe a kernel range for saving notebooks.
I have pioneered and promoted the use of vscode so that we could all have a visual "point and clickly" collaboration experience, especially for those that are new to cloud infrastructure and devops i.e.: lowering the barrier to entry with a familair interface and feel. If I cannot, I will need to consider migrating the entire team back to jupyterhub and cli git. This would be sub-optimal.
On Tue, 20 Apr 2021 at 23:20, Grant Vorenberg @.***> wrote:
I will review and revert.
Sent from my iPhone
On 20 Apr 2021, at 19:05, Don Jayamanne @.***> wrote:
Thanks for the information. I'd like to clarify my understanding of the issue. The problem is as follows:
- Display name used to be "display_name": "Python 3", but now its "Python 3.8.3 64-bit ('base': conda)"
- This is because when you ran a cell against this notebook, you selected Python 3.5.3 64-bit (base conda) environment. Jupyter doesn't natively support Conda Environments without the use of an extension. And if you did install that, then you'd see similar changes in the display name.
- Name used to be python3 and now its python383jvsc74a57bd0fc2f79212b9aaab49a9398475eca83543810af022be1929a600f8cb43b6e382b
- Again, this is similar to the previous response.
- However we're looking at not changing this if you use the current default interpreter in VS Code.
- That being said if the kernel seleced is not the default interpreter, it will get updated.
- This is because Jupyter doesn't natively support all interpreters on your machine (e.g. Conda, pipenv, etc) it only supports kernels. To ensure users opening a notebook in VS Code can run against the same kernel, we need to save the name of the interpreter (kernel), just as Jupyter does. Hence the metadata is updated.
- A new line is missing
- Duplicate of #5428 https://github.com/microsoft/vscode-jupyter/issues/5428
I.e. metadata is not missing, but it's different. Please could you confirm this.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-jupyter/issues/5524#issuecomment-823404332, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADNN3ZOONCXIGB7CUKRGWHDTJWQ5PANCNFSM426Z6IBA .
DonJayamanne
commented
3 years ago Thanks I understand the problem and we'll try to fix this Related to https://github.com/microsoft/vscode-jupyter/issues/5612
DonJayamanne
commented
3 years ago @grantav We've discussed this internally & we'll fix this.
Basically we'll leave the kernel spec as is, and not udpate the name to contain the long name python383jvsc74a57bd0fc2f79212b9aaab49a9398475eca83543810af022be1929a600f8cb43b6e382b.
We'll instead add some custo metadata into the notebook document.
DonJayamanne
commented
3 years ago Closing this issue in favor of #5612
Issue Type: Bug
Hi VSCode Team,
It seems that certain metadata from the jupyter notebooks I am working with (python3.X) is not being saved inside the notebook.
Opening and saving the notebook using jupyter from the cli "jupyter notebook /path/to/project/" fixes the issue.
This started after I upgraded to 1.55.2-1618307277. Downgrading to 1.55.1-1617808414 does not fix the issue.
Please see error below:
[2021-04-15 06:16:55,928] {{bash_operator.py:157}} INFO - Input Notebook: /usr/local/code/Path/To/Notebooks/20210414-SA_Monthly_ProductQuality_Report.ipynb [2021-04-15 06:16:55,928] {{bash_operator.py:157}} INFO - Output Notebook: /usr/local/code/Path/To/Notebooks/data/logs/20210308-20210414-SA_Monthly_ProductQuality_Report-runlog.ipynb [2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - Traceback (most recent call last): [2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/ipython_genutils/ipstruct.py", line 132, in getattr [2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - result = self[key] 21-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - KeyError: 'language' [2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - [2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - During handling of the above exception, another exception occurred: [2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - [2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - Traceback (most recent call last): [2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - File "/usr/local/bin/papermill", line 8, in
[2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - sys.exit(papermill())
[2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in call
[2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - return self.main(args, kwargs)
[2021-04-15 06:16:55,956] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - rv = self.invoke(ctx)
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - return ctx.invoke(self.callback, ctx.params)
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - return callback(args, *kwargs)
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - return f(get_current_context(), args, **kwargs)
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/papermill/cli.py", line 256, in papermill
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - execution_timeout=execution_timeout,
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/papermill/execute.py", line 91, in execute_notebook
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - nb = parameterize_notebook(nb, parameters, report_mode)
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/papermill/parameterize.py", line 77, in parameterize_notebook
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - language = nb.metadata.kernelspec.language
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - File "/usr/local/lib/python3.7/site-packages/ipython_genutils/ipstruct.py", line 134, in getattr
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - raise AttributeError(key)
[2021-04-15 06:16:55,957] {{bash_operator.py:157}} INFO - AttributeError: language
[2021-04-15 06:16:56,032] {{bash_operator.py:161}} INFO - Command exited with return code 1
[2021-04-15 06:16:56,039] {{taskinstance.py:1150}} ERROR - Bash command failed
Please note that I am using a remote system (azure virtual machine) and executing in a container: Python 3.7.9 Package Version
adal 1.2.6 alembic 1.5.5 amqp 2.6.1 ansiwrap 0.8.4 apache-airflow 1.10.14 apispec 1.3.3 appdirs 1.4.4 argcomplete 1.12.2 argon2-cffi 20.1.0 async-generator 1.10 attrs 20.3.0 azure-common 1.1.26 azure-core 1.11.0 azure-cosmos 3.2.0 azure-datalake-store 0.0.51 azure-identity 1.5.0 azure-keyvault 4.1.0 azure-keyvault-certificates 4.2.1 azure-keyvault-keys 4.3.1 azure-keyvault-secrets 4.2.0 azure-mgmt-containerinstance 1.5.0 azure-mgmt-core 1.2.2 azure-mgmt-datalake-nspkg 3.0.1 azure-mgmt-datalake-store 0.5.0 azure-mgmt-nspkg 3.0.2 azure-mgmt-resource 16.0.0 azure-nspkg 3.0.2 azure-storage 0.36.0 azure-storage-blob 12.7.1 azure-storage-common 2.1.0 Babel 2.9.0 backcall 0.2.0 bcrypt 3.2.0 billiard 3.6.3.0 black 20.8b1 bleach 3.3.0 cached-property 1.5.2 cattrs 1.0.0 celery 4.4.7 certifi 2020.12.5 cffi 1.14.5 chardet 3.0.4 click 7.1.2 colorama 0.4.4 colorlog 4.0.2 configparser 3.5.3 connections 0.2 croniter 0.3.37 cryptography 3.4.6 cycler 0.10.0 decorator 4.4.2 defusedxml 0.6.0 dill 0.3.3 dnspython 2.1.0 docker 3.7.3 docker-pycreds 0.4.0 docutils 0.16 email-validator 1.1.2 entrypoints 0.3 fastparquet 0.5.0 feather-format 0.4.1 findspark 1.4.2 Flask 1.1.2 Flask-Admin 1.5.4 Flask-AppBuilder 2.3.4 Flask-Babel 1.0.0 Flask-Bcrypt 0.7.1 Flask-Caching 1.3.3 Flask-JWT-Extended 3.25.1 Flask-Login 0.4.1 Flask-OpenID 1.2.5 Flask-SQLAlchemy 2.4.4 flask-swagger 0.2.14 Flask-WTF 0.14.3 flower 0.9.7 funcsigs 1.0.2 future 0.18.2 graphviz 0.16 gunicorn 20.0.4 hmsclient 0.1.1 humanize 3.2.0 idna 2.10 importlib-metadata 2.1.1 importlib-resources 1.5.0 ipykernel 5.5.0 ipympl 0.6.3 ipython 7.21.0 ipython-genutils 0.2.0 ipython-sql 0.4.0 ipywidgets 7.6.3 iso8601 0.1.14 isodate 0.6.0 itsdangerous 1.1.0 JayDeBeApi 1.2.3 jedi 0.18.0 Jinja2 2.11.3 joblib 1.0.1 JPype1 0.7.1 json-merge-patch 0.2 jsonschema 3.2.0 jupyter 1.0.0 jupyter-client 6.1.11 jupyter-console 6.2.0 jupyter-core 4.7.1 jupyterlab-pygments 0.1.2 jupyterlab-widgets 1.0.0 kiwisolver 1.3.1 kombu 4.6.11 lazy-object-proxy 1.4.3 llvmlite 0.35.0 lockfile 0.12.2 Mako 1.1.4 Markdown 2.6.11 MarkupSafe 1.1.1 marshmallow 2.21.0 marshmallow-enum 1.5.1 marshmallow-sqlalchemy 0.23.1 matplotlib 3.3.4 mistune 0.8.4 msal 1.9.0 msal-extensions 0.3.0 msrest 0.6.21 msrestazure 0.6.4 mypy-extensions 0.4.3 mysqlclient 1.3.14 natsort 7.1.1 nbclient 0.5.3 nbconvert 6.0.7 nbformat 5.1.2 ndg-httpsclient 0.5.1 nest-asyncio 1.5.1 notebook 6.2.0 numba 0.52.0 numpy 1.20.1 oauthlib 3.1.0 packaging 20.9 pandas 1.2.2 pandocfilters 1.4.3 papermill 2.3.2 paramiko 2.7.2 parso 0.8.1 pathspec 0.8.1 patsy 0.5.1 pendulum 1.4.4 pexpect 4.8.0 pickleshare 0.7.5 Pillow 8.1.0 pip 21.0.1 plotly 4.0.0 portalocker 1.7.1 prettytable 0.7.2 prison 0.1.3 prometheus-client 0.8.0 prompt-toolkit 3.0.16 psutil 5.8.0 psycopg2-binary 2.8.6 ptyprocess 0.7.0 pyarrow 3.0.0 pyasn1 0.4.8 pycparser 2.20 Pygments 2.8.0 PyHive 0.6.3 PyJWT 1.7.1 pymssql 2.1.5 pymsteams 0.1.14 PyNaCl 1.4.0 pyodbc 4.0.30 pyOpenSSL 20.0.1 pyparsing 2.4.7 pyrsistent 0.17.3 pysftp 0.2.9 python-daemon 2.3.0 python-dateutil 2.8.1 python-dotenv 0.15.0 python-editor 1.0.4 python-http-client 3.3.2 python-nvd3 0.15.0 python-slugify 4.0.1 python3-openid 3.2.0 pytz 2021.1 pytzdata 2020.1 PyYAML 5.4.1 pyzmq 22.0.3 qtconsole 5.0.2 QtPy 1.9.0 redis 3.2.0 regex 2020.11.13 requests 2.23.0 requests-oauthlib 1.3.0 retrying 1.3.3 sasl 0.2.1 scikit-learn 0.24.1 scipy 1.6.1 seaborn 0.11.1 Send2Trash 1.5.0 sendgrid 6.6.0 setproctitle 1.2.2 setuptools 54.0.0 six 1.15.0 sklearn 0.0 sql-magic 0.0.4 SQLAlchemy 1.3.23 SQLAlchemy-JSONField 0.9.0 SQLAlchemy-Utils 0.36.8 sqlparse 0.4.1 sshtunnel 0.1.5 starkbank-ecdsa 1.1.0 statsmodels 0.12.2 tabulate 0.8.9 tenacity 4.12.0 terminado 0.9.2 testpath 0.4.4 text-unidecode 1.3 textwrap3 0.9.2 threadpoolctl 2.1.0 thrift 0.13.0 thrift-sasl 0.4.2 toml 0.10.2 tornado 5.1.1 tqdm 4.58.0 traitlets 5.0.5 typed-ast 1.4.2 typing-extensions 3.7.4.3 tzlocal 1.5.1 unicodecsv 0.14.1 urllib3 1.25.11 vine 1.3.0 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 0.57.0 Werkzeug 0.16.1 wheel 0.36.2 widgetsnbextension 3.5.1 WTForms 2.3.3 XlsxWriter 1.3.7 zipp 3.4.0 zope.deprecation 4.4.0
Extension version: 2021.5.745244803 VS Code version: Code 1.55.1 (08a217c4d27a02a5bcde898fd7981bda5b49391b, 2021-04-07T15:01:24.315Z) OS version: Linux x64 5.4.0-71-generic Fetching remote diagnostics for 'ssh-remote+staging' timed out.
System Info
|Item|Value| |---|---| |CPUs|Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz (12 x 800)| |GPU Status|2d_canvas: enabledgpu_compositing: enabled
multiple_raster_threads: enabled_on
oop_rasterization: disabled_off
opengl: enabled_on
protected_video_decode: unavailable_off
rasterization: disabled_software
skia_renderer: enabled_on
video_decode: disabled_software
vulkan: disabled_off
webgl: enabled
webgl2: enabled| |Load (avg)|1, 1, 0| |Memory (System)|31.04GB (25.61GB free)| |Process Argv|--no-sandbox --unity-launch --crash-reporter-id c87e831b-665f-45c7-b64b-938508a361e1| |Screen Reader|no| |VM|0%| |DESKTOP_SESSION|pantheon| |XDG_CURRENT_DESKTOP|Unity| |XDG_SESSION_DESKTOP|pantheon| |XDG_SESSION_TYPE|x11| Fetching remote diagnostics for 'ssh-remote+staging' timed out.
A/B Experiments
``` vsliv368cf:30146710 vsreu685:30147344 python383:30185418 vspor879:30202332 vspor708:30202333 vspor363:30204092 vstry244:30276681 pythonvsdeb440:30248342 pythonvsded773:30248341 pythonvspyt875:30259475 pythonvspyt639:30281910 pythontb:30283811 wslfolderdoc:30282074 vspre833:30267464 pythonptprofiler:30281270 vscnewfiletext:30288035 vshan820cf:30276953 pythondataviewer:30285071 vscus158:30286553 ```