[BUG] Jupyter notebook execution behaving randomly

kingshukb commented 1 year ago

SynapseML version

using docker installation mentioned here https://microsoft.github.io/SynapseML/docs/getting_started/installation/#docker

System information

Language version (e.g. python 3.8, scala 2.12):
Spark Version (e.g. 3.2.3):
Spark Platform (e.g. Synapse, Databricks):

using the docker setup mentioened here: https://microsoft.github.io/SynapseML/docs/getting_started/installation/#docker

Note: using Macbook Pro (Apple M1 Pro chip)

Describe the problem

sometimes the same code runs sometimes it takes too much time(i feel it hangs execution), attaching screenshot of same:

Screenshot 2023-07-06 at 8 40 06 PM Although it stucks in different part of notebook if separate attempts. even simple commands like echo fails.

attaching logs as well in below section: (Note: i had to restart jupyter kernal many time to start execution from begining, it randomaly stops further execution. i had to start jupyter in debug mode to check)

Code to reproduce issue

NA

Other info / logs

demo-notebook-1 | [D 15:06:15.332 NotebookApp] 304 GET /static/components/MathJax/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full,Safe&delayStartupUntil=configured (172.20.0.1) 118.460000ms demo-notebook-1 | [D 15:06:15.359 NotebookApp] 304 GET /custom/custom.css (172.20.0.1) 89.880000ms demo-notebook-1 | [D 15:06:15.375 NotebookApp] 304 GET /static/components/MathJax/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full,Safe&delayStartupUntil=configured (172.20.0.1) 8.050000ms demo-notebook-1 | [D 15:06:15.796 NotebookApp] 304 GET /static/components/MathJax/config/TeX-AMS-MML_HTMLorMML-full.js?V=2.7.9 (172.20.0.1) 16.000000ms demo-notebook-1 | [D 15:06:15.956 NotebookApp] 304 GET /custom/custom-preload.js?v=20230706121415 (172.20.0.1) 18.320000ms demo-notebook-1 | [D 15:06:15.967 NotebookApp] 304 GET /static/components/MathJax/config/TeX-AMS-MMLHTMLorMML-full.js?V=2.7.9 (172.20.0.1) 5.560000ms demo-notebook-1 | [D 15:06:15.980 NotebookApp] 304 GET /custom/custom-preload.js?v=20230706121415 (172.20.0.1) 5.570000ms demo-notebook-1 | [D 15:06:16.026 NotebookApp] 304 GET /static/components/MathJax/config/Safe.js?V=2.7.9 (172.20.0.1) 8.610000ms demo-notebook-1 | [D 15:06:16.255 NotebookApp] Paths used for configuration of notebook: demo-notebook-1 | /etc/jupyter/nbconfig/notebook.json demo-notebook-1 | [D 15:06:16.295 NotebookApp] Paths used for configuration of notebook: demo-notebook-1 | /usr/local/etc/jupyter/nbconfig/notebook.d/widgetsnbextension.json demo-notebook-1 | /usr/local/etc/jupyter/nbconfig/notebook.json demo-notebook-1 | [D 15:06:16.313 NotebookApp] Paths used for configuration of notebook: demo-notebook-1 | /root/.local/etc/jupyter/nbconfig/notebook.json demo-notebook-1 | [D 15:06:16.332 NotebookApp] Paths used for configuration of notebook: demo-notebook-1 | /root/.jupyter/nbconfig/notebook.json demo-notebook-1 | [D 15:06:16.349 NotebookApp] 200 GET /api/config/notebook?=1688655975811 (172.20.0.1) 298.760000ms demo-notebook-1 | [D 15:06:16.361 NotebookApp] Paths used for configuration of common: demo-notebook-1 | /etc/jupyter/nbconfig/common.json demo-notebook-1 | [D 15:06:16.366 NotebookApp] Paths used for configuration of common: demo-notebook-1 | /usr/local/etc/jupyter/nbconfig/common.json demo-notebook-1 | [D 15:06:16.372 NotebookApp] Paths used for configuration of common: demo-notebook-1 | /root/.local/etc/jupyter/nbconfig/common.json demo-notebook-1 | [D 15:06:16.379 NotebookApp] Paths used for configuration of common: demo-notebook-1 | /root/.jupyter/nbconfig/common.json demo-notebook-1 | [D 15:06:16.385 NotebookApp] 200 GET /api/config/common?_=1688655975812 (172.20.0.1) 31.180000ms demo-notebook-1 | [D 15:06:16.440 NotebookApp] Found kernel python3 in /usr/local/share/jupyter/kernels demo-notebook-1 | [D 15:06:16.474 NotebookApp] 304 GET /api/kernelspecs (172.20.0.1) 65.320000ms demo-notebook-1 | [D 15:06:16.501 NotebookApp] 304 GET /static/components/MathJax/config/Safe.js?V=2.7.9 (172.20.0.1) 86.230000ms demo-notebook-1 | [D 15:06:16.508 NotebookApp] 304 GET /static/components/MathJax/extensions/TeX/begingroup.js?V=2.7.9 (172.20.0.1) 90.120000ms demo-notebook-1 | [W 15:06:16.612 NotebookApp] Notebook test_docker_compose/synapses_if.ipynb is not trusted demo-notebook-1 | [D 15:06:16.636 NotebookApp] 200 GET /api/contents/test_docker_compose/synapsesif.ipynb?type=notebook&=1688655975813 (172.20.0.1) 151.380000ms demo-notebook-1 | [D 15:06:16.640 NotebookApp] 304 GET /custom/custom.js?v=20230706121415 (172.20.0.1) 150.970000ms demo-notebook-1 | [D 15:06:16.643 NotebookApp] 304 GET /static/components/MathJax/extensions/TeX/newcommand.js?V=2.7.9 (172.20.0.1) 145.860000ms demo-notebook-1 | [D 15:06:16.650 NotebookApp] 304 GET /static/components/MathJax/extensions/MathML/content-mathml.js?V=2.7.9 (172.20.0.1) 137.110000ms demo-notebook-1 | [D 15:06:16.689 NotebookApp] 304 GET /static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js?V=2.7.9 (172.20.0.1) 19.670000ms demo-notebook-1 | [D 15:06:16.697 NotebookApp] 304 GET /static/components/MathJax/extensions/TeX/begingroup.js?V=2.7.9 (172.20.0.1) 24.910000ms demo-notebook-1 | [D 15:06:16.700 NotebookApp] 304 GET /custom/custom.js?v=20230706121415 (172.20.0.1) 25.490000ms demo-notebook-1 | [D 15:06:16.704 NotebookApp] 304 GET /static/components/MathJax/extensions/Safe.js?V=2.7.9 (172.20.0.1) 25.040000ms demo-notebook-1 | [D 15:06:16.714 NotebookApp] 304 GET /nbextensions/jupyter-js-widgets/extension.js?v=20230706121415 (172.20.0.1) 22.470000ms demo-notebook-1 | [D 15:06:16.731 NotebookApp] 304 GET /static/components/MathJax/extensions/TeX/newcommand.js?V=2.7.9 (172.20.0.1) 12.760000ms demo-notebook-1 | [D 15:06:16.736 NotebookApp] 304 GET /static/components/MathJax/extensions/MathML/content-mathml.js?V=2.7.9 (172.20.0.1) 11.290000ms demo-notebook-1 | [D 15:06:16.770 NotebookApp] 200 GET /nbextensions/jupyter-js-widgets/extension.js?v=20230706121415 (172.20.0.1) 28.960000ms demo-notebook-1 | [D 15:06:16.793 NotebookApp] 200 GET /static/components/MathJax/extensions/Safe.js?V=2.7.9 (172.20.0.1) 48.530000ms demo-notebook-1 | [D 15:06:16.800 NotebookApp] 304 GET /static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js?V=2.7.9 (172.20.0.1) 51.410000ms demo-notebook-1 | [D 15:06:16.845 NotebookApp] 200 GET /api/contents/test_docker_compose/synapsesif.ipynb/checkpoints?=1688655975814 (172.20.0.1) 21.340000ms demo-notebook-1 | [D 15:06:16.874 NotebookApp] 201 POST /api/sessions (172.20.0.1) 43.300000ms demo-notebook-1 | [D 15:06:17.003 NotebookApp] Initializing websocket connection /api/kernels/3295e79c-6f1d-4e50-8732-1eedf72d2109/channels demo-notebook-1 | [D 15:06:17.045 NotebookApp] 101 GET /api/kernels/3295e79c-6f1d-4e50-8732-1eedf72d2109/channels?session_id=277e9fc2dd9c4e3db74ad8e64f847349 (172.20.0.1) 46.530000ms demo-notebook-1 | [D 15:06:17.047 NotebookApp] Opening websocket /api/kernels/3295e79c-6f1d-4e50-8732-1eedf72d2109/channels demo-notebook-1 | [D 15:06:17.048 NotebookApp] Getting buffer for 3295e79c-6f1d-4e50-8732-1eedf72d2109 demo-notebook-1 | [D 15:06:17.052 NotebookApp] Connecting to: tcp://127.0.0.1:42545 demo-notebook-1 | [D 15:06:17.079 NotebookApp] Connecting to: tcp://127.0.0.1:46619 demo-notebook-1 | [D 15:06:17.084 NotebookApp] Connecting to: tcp://127.0.0.1:39049 demo-notebook-1 | [D 15:06:17.088 NotebookApp] Connecting to: tcp://127.0.0.1:36609 demo-notebook-1 | [D 15:06:17.092 NotebookApp] Nudge: not nudging busy kernel 3295e79c-6f1d-4e50-8732-1eedf72d2109 demo-notebook-1 | [D 15:06:17.984 NotebookApp] 304 GET /static/base/images/favicon-busy-1.ico (172.20.0.1) 3.630000ms demo-notebook-1 | [D 15:08:17.074 NotebookApp] 200 GET /api/contents/test_docker_compose/synapsesif.ipynb?content=0&=1688655975815 (172.20.0.1) 61.970000ms demo-notebook-1 | [I 15:08:17.113 NotebookApp] Saving file at /test_docker_compose/synapses_if.ipynb demo-notebook-1 | [D 15:08:17.115 NotebookApp] Saving /notebooks/features/test_docker_compose/synapses_if.ipynb demo-notebook-1 | [W 15:08:17.123 NotebookApp] Notebook test_docker_compose/synapses_if.ipynb is not trusted demo-notebook-1 | [D 15:08:17.286 NotebookApp] 200 PUT /api/contents/test_docker_compose/synapses_if.ipynb (172.20.0.1) 185.620000m

What component(s) does this bug affect?

[ ] area/cognitive: Cognitive project
[X] area/core: Core project
[X] area/deep-learning: DeepLearning project
[ ] area/lightgbm: Lightgbm project
[ ] area/opencv: Opencv project
[ ] area/vw: VW project
[ ] area/website: Website
[X] area/build: Project build system
[ ] area/notebooks: Samples under notebooks folder
[X] area/docker: Docker usage
[ ] area/models: models related issue

What language(s) does this bug affect?

[ ] language/scala: Scala source code
[ ] language/python: Pyspark APIs
[ ] language/r: R APIs
[ ] language/csharp: .NET APIs
[ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

[ ] integrations/synapse: Azure Synapse integrations
[ ] integrations/azureml: Azure ML integrations
[ ] integrations/databricks: Databricks integrations

github-actions[bot] commented 1 year ago

Hey @kingshukb :wave:! Thank you so much for reporting the issue/feature request :rotating_light:. Someone from SynapseML Team will be looking to triage this issue soon. We appreciate your patience.

mhamilton723 commented 1 year ago

Thanks @kingshukb what notebook are you trying to run with docker? One thing you can do is pass docker setting to increase the ammount of RAM your container can use. Also for quick dev our binder setup might also help you

kingshukb commented 1 year ago

I somehow made it work using a custom dockerfile installing all the components and loading jars. now its behaving properly. i am able to execute the code, but getting stuck while using the TabularSHAP, getting the below error on displaying results as :

`shap = TabularSHAP( inputCols=inputCols, outputCol="shapValues", model=model, targetCol="outlierScore", backgroundData=F.broadcast(df_test), )

shaps_local = shaps.toPandas() shaps_local ` as per the example https://microsoft.github.io/SynapseML/docs/features/isolation_forest/IsolationForest%20-%20Multivariate%20Anomaly%20Detection/

i am getting following errors:

Py4JJavaError: An error occurred while calling o1837.collectToPython. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 103.0 failed 1 times, most recent failure: Lost task 0.0 in stage 103.0 (TID 2555) (904ba53d7cc2 executor driver): java.lang.AssertionError: assertion failed

Any Idea, please help. Thanks.

memoryz commented 1 month ago

I was able to run the notebook a few times without any issue, with one change to do the docker image.

The change I had to make is this: tools/docker/demo/Dockerfile, line 30:

RUN curl -sSL https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh \
    && bash /tmp/miniconda.sh -bfp /usr/local \
    && rm -rf /tmp/miniconda.sh \
    && conda update conda \
-    && conda install -y python=3 jupyter pyspark \
+    && conda install -y python=3.9 jupyter pyspark \
    && conda clean --all --yes

Then run the followings from repo root:

docker build . -f tools/docker/demo/Dockerfile -t synapseml
docker run -it -p 8888:8888 --rm -e ACCEPT_EULA=yes synapseml:latest

The reason for the change is the dependencies in the IsolationForest notebook is not compatible with the latest version of python3 (3.12).

memoryz commented 1 month ago

@mhamilton723, I suggest we fix the python version in the conda install step whenever we update the Spark version.

microsoft / SynapseML