search

jupyter / nb2kg

Other
73 stars 31 forks source link

Can not get kernel in notebook #17

Closed lw233 closed 5 years ago

lw233 commented 6 years ago

The notebook can not get kernel after updated the KG on z/os.
The python is v3.6. The jupyter-kernel-gateway is 2.1.0 and tried the nb2kg with 0.2.0.dev0/0.3.0/0.4.0.dev0. Do you have any advise on this issue? Thank you!

lw233 commented 6 years ago

The logs in notebook side:

[I 00:25:37.537 NotebookApp] 302 GET /?token=017e32763ddf5af89a8e976e20becd3e6e82a1e6bbc82ad6 (9.115.192.36) 1.49ms [I 00:25:38.752 NotebookApp] Request list kernel specs at: /api/kernelspecs [I 00:25:52.383 NotebookApp] Creating new notebook in [I 00:26:00.780 NotebookApp] Request list kernel specs at: /api/kernelspecs [I 00:26:08.879 NotebookApp] Request start kernel: kernel_id=None, path="" [I 00:26:08.879 NotebookApp] Request new kernel at: /api/kernels [E 00:26:08.957 NotebookApp] Uncaught exception POST /api/sessions (9.115.192.36) HTTPServerRequest(protocol='http', host='9.30.57.25:8888', method='POST', uri='/api/sessions', version='HTTP/1.1', remote_ip='9.115.192.36', headers={'Host': '9.30.57.25:8888', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0', 'Accept': 'application/json, text/javascript, /; q=0.01', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/json', 'X-Xsrftoken': '2|0bc7c884|1843e608872ef9aa3cd67a1ae146c3f6|1536212668', 'X-Requested-With': 'XMLHttpRequest', 'Referer': 'http://9.30.57.25:8888/notebooks/Untitled19.ipynb?kernel_name=imlpython3', 'Content-Length': '96', 'Cookie': 'username-9-30-57-25-8888="2|1:0|10:1536218737|24:username-9-30-57-25-8888|44:ZjJkNTEyMWUyZTI2NDVmMDllNWI2YmQ0NDVjMjY3MmE=|31dc4987688b47752bf9251725651033763c9360ab1e12ba022e92e00a004af8"; _xsrf=2|0bc7c884|1843e608872ef9aa3cd67a1ae146c3f6|1536212668', 'Connection': 'keep-alive'}) Traceback (most recent call last): File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/web.py", line 1511, in _execute result = yield result File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run value = future.result() File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run yielded = self.gen.throw(exc_info) File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/notebook/services/sessions/handlers.py", line 73, in post type=mtype)) File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run value = future.result() File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run yielded = self.gen.throw(exc_info) File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/nb2kg/managers.py", line 333, in create_session session_id, path, name, type, kernel_name, File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run value = future.result() File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run yielded = self.gen.throw(exc_info) File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 92, in start_kernel_for_session self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name) File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run value = future.result() File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run yielded = self.gen.throw(exc_info) File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/nb2kg/managers.py", line 145, in start_kernel response = yield fetch_kg(self.kernels_endpoint, method='POST', body=json_body) File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run value = future.result() File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run yielded = self.gen.throw(*exc_info) File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/nb2kg/managers.py", line 64, in fetch_kg response = yield client.fetch(url, *kwargs) File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run value = future.result() File "/usr/ibmwml/anaconda3/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info tornado.httpclient.HTTPError: HTTP 500: Internal Server Error [W 00:26:08.963 NotebookApp] Unhandled error [E 00:26:08.964 NotebookApp] { "Host": "9.30.57.25:8888", "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0", "Accept": "application/json, text/javascript, /*; q=0.01", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate", "Content-Type": "application/json", "X-Xsrftoken": "2|0bc7c884|1843e608872ef9aa3cd67a1ae146c3f6|1536212668", "X-Requested-With": "XMLHttpRequest", "Referer": "http://9.30.57.25:8888/notebooks/Untitled19.ipynb?kernel_name=imlpython3", "Content-Length": "96", "Cookie": "username-9-30-57-25-8888=\"2|1:0|10:1536218737|24:username-9-30-57-25-8888|44:ZjJkNTEyMWUyZTI2NDVmMDllNWI2YmQ0NDVjMjY3MmE=|31dc4987688b47752bf9251725651033763c9360ab1e12ba022e92e00a004af8\"; _xsrf=2|0bc7c884|1843e608872ef9aa3cd67a1ae146c3f6|1536212668", "Connection": "keep-alive" } [E 00:26:08.964 NotebookApp] 500 POST /api/sessions (9.115.192.36) 87.37ms referer=http://9.30.57.25:8888/notebooks/Untitled19.ipynb?kernel_name=imlpython3

lw233 commented 6 years ago

The log in KG side

[KernelGatewayApp] Starting kernel: ['/u/SPARK1/.local/share/jupyter/kernels/imlspark_scala/bin/run.sh', '--profile', '/u/jupyter/runtime/kernel-08530156-49e0-4c35-b4d4-1f34ab2a679a.json'] [KernelGatewayApp] ERROR | Failed to run command: ['/u/SPARK1/.local/share/jupyter/kernels/imlspark_scala/bin/run.sh', '--profile', '/u/SPARK1/.local/share/jupyter/runtime0-4c35-b4d4-1f34ab2a679a.json'] PATH='/home/mlzdev/userdir/spark1/conda/bin:/u/ZHHAO/scripts:/home/aln120/mlz/imlpython/bin:/home/aln120/mlz/bin:/homrk1/conda/bin:/bin:/bin/bash:/java8_64/J8.0_64/bin:/home/mlzdev/spark211/bin:/home/mlzdev/spark211/sbin:' with kwargs: {'stdin': -1, 'stdout': None, 'stderr': None, 'cwd': None, 'start_new_session': True}

[E 180905 22:31:08 web:1548] Uncaught exception POST /jupyter/v1/api/kernels (9.30.123.30) HTTPServerRequest(protocol='http', host='9.30.128.39:7889', method='POST', uri='/jupyter/v1/api/kernels', version='HT9.30.123.30', headers={'Authorization': 'token', 'Connection': 'close', 'Host': '9.30.128.39:7889', 'Content-Length': '37application/x-www-form-urlencoded', 'Accept-Encoding': 'gzip'}) Traceback (most recent call last): File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/web.py", line 1469, in _execute result = yield result File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/gen.py", line 1015, in run value = future.result() File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/concurrent.py", line 237, in result raise_exc_info(self._exc_info) File "", line 3, in raise_exc_info File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/gen.py", line 1021, in run yielded = self.gen.throw(exc_info) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/kernel_gateway/services/kernels/handlers.py", l yield super(MainKernelHandler, self).post() File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/gen.py", line 1015, in run value = future.result() File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/concurrent.py", line 237, in result raise_exc_info(self._exc_info) File "", line 3, in raise_exc_info File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/gen.py", line 1021, in run yielded = self.gen.throw(exc_info) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/notebook/services/kernels/handlers.py", line 47 kernel_id = yield gen.maybe_future(km.start_kernel(kernel_name=model['name'])) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/gen.py", line 1015, in run value = future.result() File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/concurrent.py", line 237, in result raise_exc_info(self._exc_info) File "", line 3, in raise_exc_info File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/gen.py", line 1021, in run yielded = self.gen.throw(exc_info) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/kernel_gateway/services/kernels/manager.py", liel kernel_id = yield gen.maybe_future(super(SeedingMappingKernelManager, self).start_kernel(args, kwargs)) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/gen.py", line 1015, in run value = future.result() File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/concurrent.py", line 237, in result raise_exc_info(self._exc_info) File "", line 3, in raise_exc_info File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/tornado/gen.py", line 285, in wrapper yielded = next(result) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/notebook/services/kernels/kernelmanager.py", linel super(MappingKernelManager, self).start_kernel(kwargs) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/jupyter_client/multikernelmanager.py", line 110 km.start_kernel(kwargs) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/jupyterclient/manager.py", line 259, in start kw) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/kernel_gateway/services/kernels/manager.py", liernel return super(KernelGatewayIOLoopKernelManager, self)._launch_kernel(kernel_cmd, kw) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/jupyter_client/manager.py", line 204, in _launc return launch_kernel(kernel_cmd, kw) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/site-packages/jupyter_client/launcher.py", line 128, in launc try: File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/subprocess.py", line 711, in init restore_signals, start_new_session) File "/home/mlzdev/userdir/spark1/conda/lib/python3.6/subprocess.py", line 1289, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 121] EDC5121I Invalid argument. [E 180905 22:31:08 web:1971] 500 POST /jupyter/v1/api/kernels (9.30.123.30) 30.27ms

kevin-bates commented 6 years ago

@lw233 - this strikes me as something unrelated to NB2KG and more to do with the kernel launch script on the Kernel Gateway server. This is based on these two entries in the KG log...

[KernelGatewayApp] ERROR | Failed to run command:
['/u/SPARK1/.local/share/jupyter/kernels/imlspark_scala/bin/run.sh', '--profile', '/u/SPARK1/.local/share/jupyter/runtime0-4c35-b4d4-1f34ab2a679a.json']
PATH='/home/mlzdev/userdir/spark1/conda/bin:/u/ZHHAO/scripts:/home/aln120/mlz/imlpython/bin:/home/aln120/mlz/bin:/homrk1/conda/bin:/bin:/bin/bash:/java8_64/J8.0_64/bin:/home/mlzdev/spark211/bin:/home/mlzdev/spark211/sbin:'
with kwargs:
{'stdin': -1, 'stdout': None, 'stderr': None, 'cwd': None, 'start_new_session': True}

OSError: [Errno 121] EDC5121I Invalid argument.

You opened the issue with this statement: "after updated the KG on z/os" leading to these questions...

  1. What exactly did you update on the KG server?
  2. Was this kernel working (via NB2KG) prior to that?

I suspect the update side-affected the /u/SPARK1/.local/share/jupyter/kernels/imlspark_scala/bin/run.sh script. As a result, you might try running that script manually if you're able to create a similar environment to how Kernel Gateway is running. (You'll probably want to capture the json file parameter from a previous run to use as that parameter and set similar env variables that the script requires.) I suspect you'll encounter the same issue, and should be able to more easily determine the problem.

kevin-bates commented 5 years ago

Closing as a misfiled issue. Please re-open and provide additional information if this isn't correct.