search

googlecolab / jupyter_http_over_ws

Apache License 2.0
266 stars 56 forks source link

Sporadically (but often) getting 404 GET error "/api/contents/var/colab/app.log (127.0.0.1): No such file or directory: var/colab/app.log" #27

Open piojanu opened 2 years ago

piojanu commented 2 years ago

Hi!

I have a problem, my Colab notebook keeps crashing when connected to the local runtime (over two SSH tunnels, if it makes the difference) with the error 404 GET /api/contents/var/colab/app.log (127.0.0.1): No such file or directory: var/colab/app.log. I pasted the cmd output below. Sometimes it happens immediately when I try to run the first cell in the notebook after successful connection to the local runtime and sometimes after some time mid-training (using TF v2). My OS is Ubuntu 18.04.

Do you know this problem? Any ideas on how to fix it?

[I 09:13:30.945 NotebookApp] Jupyter Notebook 6.4.6 is running at:
[I 09:13:30.945 NotebookApp] http://localhost:8888/?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa
[I 09:13:30.945 NotebookApp]  or http://127.0.0.1:8888/?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa
[I 09:13:30.945 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 09:13:30.948 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/pjanuszewski/.local/share/jupyter/runtime/nbserver-8792-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa
     or http://127.0.0.1:8888/?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa
[I 09:13:50.656 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.910000ms
[I 09:13:50.865 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.880000ms
[I 09:13:50.905 NotebookApp] Kernel started: f4a78d66-276f-4b34-a18a-88722e9d992e, name: python3
[I 09:13:50.952 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.930000ms
[I 09:13:50.963 NotebookApp] proxying WebSocket connection to: ws://localhost:8888/api/kernels/f4a78d66-276f-4b34-a18a-88722e9d992e/channels?session_id=cad2755a9fd5414ec4dd128190228efc&jupyter_http_over_ws_auth_url=http%3A%2F%2Flocalhost%3A8888%2F%3Ftoken%3De848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa
[I 09:13:54.878 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.900000ms
[I 09:13:56.594 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.840000ms
[I 09:13:57.777 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.810000ms
[I 09:14:11.905 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
WARNING:root:kernel f4a78d66-276f-4b34-a18a-88722e9d992e restarted
[I 09:14:12.374 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 1.430000ms
[I 09:14:12.394 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.490000ms
[W 09:14:12.401 NotebookApp] 404 GET /api/contents/var/colab/app.log (127.0.0.1): No such file or directory: var/colab/app.log
[W 09:14:12.401 NotebookApp] No such file or directory: var/colab/app.log
[W 09:14:12.401 NotebookApp] 404 GET /api/contents/var/colab/app.log (127.0.0.1) 0.840000ms referer=None
[I 09:14:12.406 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.350000ms
[W 09:14:12.412 NotebookApp] 404 GET /api/contents/var/colab/ooms (127.0.0.1): No such file or directory: var/colab/ooms
[W 09:14:12.412 NotebookApp] No such file or directory: var/colab/ooms
[W 09:14:12.412 NotebookApp] 404 GET /api/contents/var/colab/ooms (127.0.0.1) 0.590000ms referer=None
[I 09:14:15.560 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.800000ms
[W 09:14:15.573 NotebookApp] 404 GET /api/contents/var/colab/app.log (127.0.0.1): No such file or directory: var/colab/app.log
[W 09:14:15.574 NotebookApp] No such file or directory: var/colab/app.log
[W 09:14:15.574 NotebookApp] 404 GET /api/contents/var/colab/app.log (127.0.0.1) 1.180000ms referer=None
[I 09:15:18.702 NotebookApp] 302 GET /?token=e848e3b8ff73d61e99bbc4ea2e575ff4157442406978cbfa (127.0.0.1) 0.800000ms
[I 09:15:23.921 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports

Thanks! Piotr

blois commented 2 years ago

The GET request to /api/contents/var/colab/app.log should be occurring after the crash, I'd be surprised if it's the cause of it. When Colab sees an unexpected runtime restart it attempts to fetch the logs to see if there's a known issue logged so it can display a workaround.

piojanu commented 2 years ago

Okay, so how can I see what's the problem?

onderpolat-glean commented 1 year ago

adding

import logging import sys

logging.basicConfig(format='%(asctime)s | %(levelname)s : %(message)s', level=logging.INFO, stream=sys.stdout)

to the beginning of your notebook solves this problem.

convertthinking commented 1 year ago

I have a similar problem, my OS is Ubuntu 22.10.

Colab failed to connect local Docker runtime till I downgraded to jupyterlab==3.0.0 tornado==6.1, like others do.

Then it's crashes due to unknown error on importing parts of a package inside a notebook.

root@1a404a587288:/home# jupyter notebook --notebook-dir=/ --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com' --port=8889 --NotebookApp.port_retries=0

| | | |_ | | | | __ | || | ' \/ / _ | / -) _/| ./_,_,|_\| |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

jupyter_http_over_ws extension initialized. Listening on /http_over_websocket [W 2023-05-14 00:40:04.873 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-05-14 00:40:04.873 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-05-14 00:40:04.873 LabApp] 'allow_origin' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-05-14 00:40:04.873 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-05-14 00:40:04.874 LabApp] 'port_retries' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-05-14 00:40:04.874 LabApp] 'port_retries' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-05-14 00:40:04.874 LabApp] 'allow_root' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [W 2023-05-14 00:40:04.874 LabApp] 'allow_root' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. [I 2023-05-14 00:40:04.879 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.8/dist-packages/jupyterlab [I 2023-05-14 00:40:04.879 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab [I 00:40:04.886 NotebookApp] Serving notebooks from local directory: / [I 00:40:04.886 NotebookApp] Jupyter Notebook 6.5.4 is running at: [I 00:40:04.886 NotebookApp] http://1a404a587288:8889/?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c [I 00:40:04.886 NotebookApp] or http://127.0.0.1:8889/?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c [I 00:40:04.886 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 00:40:04.891 NotebookApp] 

To access the notebook, open this file in a browser:
    file:///root/.local/share/jupyter/runtime/nbserver-2272-open.html
Or copy and paste one of these URLs:
    http://1a404a587288:8889/?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c
 or http://127.0.0.1:8889/?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c

[I 00:40:21.840 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.670000ms [I 00:40:22.194 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.610000ms [I 00:40:22.256 NotebookApp] Kernel started: bf27e1d2-c9bd-48ea-b71d-a9a4a0866ab6, name: python3 [I 00:40:22.283 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 1.020000ms [I 00:40:22.290 NotebookApp] proxying WebSocket connection to: ws://localhost:8889/api/kernels/bf27e1d2-c9bd-48ea-b71d-a9a4a0866ab6/channels?session_id=9f26beaa-7f19-462c-8237-bd62fb15b2ec&jupyter_http_over_ws_auth_url=http%3A%2F%2Flocalhost%3A8889%2F%3Ftoken%3D2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c [I 00:41:04.245 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports WARNING:root:kernel bf27e1d2-c9bd-48ea-b71d-a9a4a0866ab6 restarted [I 00:41:05.348 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.800000ms [I 00:41:05.364 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.510000ms [W 00:41:05.373 NotebookApp] 404 GET /api/contents/var/colab/events (127.0.0.1): file or directory does not exist: 'var/colab/events' [W 00:41:05.373 NotebookApp] file or directory does not exist: 'var/colab/events' [W 00:41:05.374 NotebookApp] 404 GET /api/contents/var/colab/events (127.0.0.1) 1.520000ms referer=None [I 00:41:05.378 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.560000ms [W 00:41:05.399 NotebookApp] 404 GET /api/contents/var/colab/ooms (127.0.0.1): file or directory does not exist: 'var/colab/ooms' [W 00:41:05.401 NotebookApp] file or directory does not exist: 'var/colab/ooms' [W 00:41:05.401 NotebookApp] 404 GET /api/contents/var/colab/ooms (127.0.0.1) 7.620000ms referer=None [I 00:41:05.409 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.720000ms [I 00:41:05.429 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.540000ms [W 00:41:05.437 NotebookApp] 404 GET /api/contents/var/colab/events (127.0.0.1): file or directory does not exist: 'var/colab/events' [W 00:41:05.437 NotebookApp] file or directory does not exist: 'var/colab/events' [W 00:41:05.438 NotebookApp] 404 GET /api/contents/var/colab/events (127.0.0.1) 1.240000ms referer=None [I 00:41:05.442 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.550000ms [W 00:41:05.451 NotebookApp] 404 GET /api/contents/var/colab/ooms (127.0.0.1): file or directory does not exist: 'var/colab/ooms' [W 00:41:05.451 NotebookApp] file or directory does not exist: 'var/colab/ooms' [W 00:41:05.451 NotebookApp] 404 GET /api/contents/var/colab/ooms (127.0.0.1) 1.340000ms referer=None [I 00:41:05.455 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.610000ms [W 00:41:05.464 NotebookApp] Blocking request with no referer [W 00:41:05.464 NotebookApp] 403 GET /files/var/colab/app.log (127.0.0.1): Blocking request from unknown origin [W 00:41:05.475 NotebookApp] 403 GET /files/var/colab/app.log (127.0.0.1) 12.060000ms referer=None [I 00:41:05.480 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.560000ms [W 00:41:05.489 NotebookApp] Blocking request with no referer [W 00:41:05.489 NotebookApp] 403 GET /files/var/colab/app.log (127.0.0.1): Blocking request from unknown origin [W 00:41:05.491 NotebookApp] 403 GET /files/var/colab/app.log (127.0.0.1) 2.040000ms referer=None [I 00:48:28.272 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports WARNING:root:kernel bf27e1d2-c9bd-48ea-b71d-a9a4a0866ab6 restarted [I 00:48:29.275 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.900000ms [I 00:48:29.302 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.600000ms [W 00:48:29.311 NotebookApp] 404 GET /api/contents/var/colab/events (127.0.0.1): file or directory does not exist: 'var/colab/events' [W 00:48:29.312 NotebookApp] file or directory does not exist: 'var/colab/events' [W 00:48:29.312 NotebookApp] 404 GET /api/contents/var/colab/events (127.0.0.1) 1.320000ms referer=None [I 00:48:29.316 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.540000ms [W 00:48:29.325 NotebookApp] 404 GET /api/contents/var/colab/ooms (127.0.0.1): file or directory does not exist: 'var/colab/ooms' [W 00:48:29.325 NotebookApp] file or directory does not exist: 'var/colab/ooms' [W 00:48:29.326 NotebookApp] 404 GET /api/contents/var/colab/ooms (127.0.0.1) 1.320000ms referer=None [I 00:48:29.337 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.610000ms [W 00:48:29.345 NotebookApp] Blocking request with no referer [W 00:48:29.345 NotebookApp] 403 GET /files/var/colab/app.log (127.0.0.1): Blocking request from unknown origin [W 00:48:29.347 NotebookApp] 403 GET /files/var/colab/app.log (127.0.0.1) 1.920000ms referer=None [I 00:48:30.904 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.790000ms [I 00:48:31.019 NotebookApp] 302 GET /?token=2e59f48ed4c5bb7e3355ce39c3ab50a2512d1e6e39e64f0c (127.0.0.1) 0.740000ms [W 00:48:31.029 NotebookApp] Blocking request with no referer [W 00:48:31.029 NotebookApp] 403 GET /files/var/colab/app.log (127.0.0.1): Blocking request from unknown origin [W 00:48:31.031 NotebookApp] 403 GET /files/var/colab/app.log (127.0.0.1) 2.190000ms referer=None [I 00:49:04.299 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports WARNING:root:kernel bf27e1d2-c9bd-48ea-b71d-a9a4a0866ab6 restarted

with messages like

Your session crashed. Automatically restarting.

Adding

import logging
import sys

logging.basicConfig(format='%(asctime)s | %(levelname)s : %(message)s', level=logging.INFO, stream=sys.stdout)

to the beginning of your notebook didn't solve this problem for me. I even can't see any logging present.

I created the dir /var/colab but it didn't change anything.

On trying get log in the colab notebook in my browser it opens popup window with message

Could not fetch /var/colab/app.log from backend
Could not fetch resource at : 404 Not Found 
tb@https://ssl.gstatic.com/colaboratory-static/common/85c0f731b1fe913211f742b41f99dc8e/external_polymer_binary.js:89:517
HB@https://ssl.gstatic.com/colaboratory-static/common/85c0f731b1fe913211f742b41f99dc8e/external_polymer_binary.js:1507:2524
pF@https://ssl.gstatic.com/colaboratory-static/common/85c0f731b1fe913211f742b41f99dc8e/external_polymer_binary.js:1924:186
g.get/<@https://ssl.gstatic.com/colaboratory-static/common/85c0f731b1fe913211f742b41f99dc8e/external_polymer_binary.js:6724:133
xa@https://ssl.gstatic.com/colaboratory-static/common/85c0f731b1fe913211f742b41f99dc8e/external_polymer_binary.js:21:57
ua.prototype.next_@https://ssl.gstatic.com/colaboratory-static/common/85c0f731b1fe913211f742b41f99dc8e/external_polymer_binary.js:19:195
laa/this.next@https://ssl.gstatic.com/colaboratory-static/common/85c0f731b1fe913211f742b41f99dc8e/external_polymer_binary.js:22:48
b@https://ssl.gstatic.com/colaboratory-static/common/85c0f731b1fe913211f742b41f99dc8e/external_polymer_binary.js:22:309
  1. Where are all those

/files/var/colab/app.log, /api/contents/var/colab/ooms /api/contents/var/colab/events var/colab/events

and how do the relate to

/var/colab?

  1. What is the proper way to install run jupyter notebook ? Mine was installed with pip.

  2. What are other ways to troubleshoot the situation and how to find crash exceptions info, etc to fix?