search

kubeflow-kale / kale

Kubeflow’s superfood for Data Scientists
http://kubeflow-kale.github.io
Apache License 2.0
632 stars 128 forks source link

Parser must be a string or character stream, not dict #315

Open kusirajiv opened 3 years ago

kusirajiv commented 3 years ago

While using Kale in AKS cluster, iam always getting following error while compiling and uploading the pipeline. image

If anyone has solution to this issue , kindly let me know.

Kale version: 0.5.0 AKS Version: v1.19.7 Kubeflow Version: 1.0.0

Here is the log details: 2021-03-01 11:45:37 run:83 [[DEBUG]] [TID=d458eymim5] [] Decoding ctx of RPC function 'rok.check_rok_availability' 2021-03-01 11:45:37 run:95 [[DEBUG]] [TID=d458eymim5] [] Decoding kwargs of RPC function 'rok.check_rok_availability' 2021-03-01 11:45:37 run:104 [[DEBUG]] [TID=d458eymim5] [] Importing RPC function 'rok.check_rok_availability' 2021-03-01 11:45:40 run:114 [[INFO]] [TID=d458eymim5] [] Executing RPC function 'check_rok_availability()' 2021-03-01 11:45:40 rok:123 [[ERROR]] [TID=d458eymim5] [] Failed to initialize RokClient Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability rok = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or theROK_GW_URL' environment variable. 2021-03-01 11:45:40 run:120 [[ERROR]] [TID=d458eymim5] [] RPC function 'check_rok_availability' raised an RPCError Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability rok = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or theROK_GW_URL' environment variable.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 126, in check_rok_availability trans_id=request.trans_id) kale.rpc.errors.RPCServiceUnavailableError 2021-03-01 11:46:03 run:83 [[DEBUG]] [TID=utv8q8mksy] [] Decoding ctx of RPC function 'nb.resume_notebook_path' 2021-03-01 11:46:03 run:95 [[DEBUG]] [TID=utv8q8mksy] [] Decoding kwargs of RPC function 'nb.resume_notebook_path' 2021-03-01 11:46:03 run:104 [[DEBUG]] [TID=utv8q8mksy] [] Importing RPC function 'nb.resume_notebook_path' 2021-03-01 11:46:04 run:114 [[INFO]] [TID=utv8q8mksy] [] Executing RPC function 'resume_notebook_path(server_root=/home/jovyan)' 2021-03-01 11:47:45 run:83 [[DEBUG]] [TID=h6zg5ogl3c] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 11:47:45 run:95 [[DEBUG]] [TID=h6zg5ogl3c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 11:47:45 run:104 [[DEBUG]] [TID=h6zg5ogl3c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 11:47:45 run:114 [[INFO]] [TID=h6zg5ogl3c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)' 2021-03-01 11:47:45 run:83 [[DEBUG]] [TID=759gi3qsbb] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 11:47:45 run:95 [[DEBUG]] [TID=759gi3qsbb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 11:47:45 run:104 [[DEBUG]] [TID=759gi3qsbb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 11:47:45 run:114 [[INFO]] [TID=759gi3qsbb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 11:47:45 run:83 [[DEBUG]] [TID=eaxdwp5at4] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 11:47:45 run:95 [[DEBUG]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 11:47:45 run:104 [[DEBUG]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 11:47:45 run:114 [[INFO]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 11:47:45 nb:205 [[INFO]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 11:47:45 nb:211 [[INFO]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 11:47:45 nb:215 [[INFO]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] PodDefault labels applied on server: 2021-03-01 11:47:46 run:83 [[DEBUG]] [TID=rekhin8drk] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 11:47:46 run:95 [[DEBUG]] [TID=rekhin8drk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 11:47:46 run:104 [[DEBUG]] [TID=rekhin8drk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 11:47:46 run:114 [[INFO]] [TID=rekhin8drk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'list_experiments()' 2021-03-01 11:47:46 run:125 [[ERROR]] [TID=rekhin8drk] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'list_experiments' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 34, in list_experiments c = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client _client = kfp.Client() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init self._configure_incluster_auth() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth self.set_user_namespace(old_ns_setting or _get_pod_namespace()) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace with open(Client.LOCAL_KFP_CONTEXT, 'w') as f: PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json' 2021-03-01 11:48:11 run:83 [[DEBUG]] [TID=d6shlr2egc] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 11:48:11 run:95 [[DEBUG]] [TID=d6shlr2egc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 11:48:11 run:104 [[DEBUG]] [TID=d6shlr2egc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 11:48:11 run:114 [[INFO]] [TID=d6shlr2egc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})' 2021-03-01 11:48:11 run:83 [[DEBUG]] [TID=ul8skouxpt] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 11:48:11 run:95 [[DEBUG]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 11:48:11 run:104 [[DEBUG]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 11:48:11 run:114 [[INFO]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)' 2021-03-01 11:48:11 core:146 [[DEBUG]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 11:48:11 core:205 [[INFO]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-w8cw0.kale.py 2021-03-01 11:48:12 run:83 [[DEBUG]] [TID=clr8x31dd9] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 11:48:12 run:95 [[DEBUG]] [TID=clr8x31dd9] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 11:48:12 run:104 [[DEBUG]] [TID=clr8x31dd9] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 11:48:12 run:114 [[INFO]] [TID=clr8x31dd9] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-w8cw0.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-w8cw0', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)' 2021-03-01 11:48:12 run:125 [[ERROR]] [TID=clr8x31dd9] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 93, in upload_pipeline client = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client _client = kfp.Client() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init self._configure_incluster_auth() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth self.set_user_namespace(old_ns_setting or _get_pod_namespace()) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace with open(Client.LOCAL_KFP_CONTEXT, 'w') as f: PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json' 2021-03-01 11:48:32 run:83 [[DEBUG]] [TID=1zmzvqqz8a] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 11:48:32 run:95 [[DEBUG]] [TID=1zmzvqqz8a] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 11:48:32 run:104 [[DEBUG]] [TID=1zmzvqqz8a] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 11:48:32 run:114 [[INFO]] [TID=1zmzvqqz8a] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})' 2021-03-01 11:48:32 run:83 [[DEBUG]] [TID=buox86rnmq] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 11:48:32 run:95 [[DEBUG]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 11:48:32 run:104 [[DEBUG]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 11:48:32 run:114 [[INFO]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)' 2021-03-01 11:48:32 core:146 [[DEBUG]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 11:48:32 core:205 [[INFO]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-9y29i.kale.py 2021-03-01 11:48:32 run:83 [[DEBUG]] [TID=9m6azmqhtp] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 11:48:32 run:95 [[DEBUG]] [TID=9m6azmqhtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 11:48:32 run:104 [[DEBUG]] [TID=9m6azmqhtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 11:48:32 run:114 [[INFO]] [TID=9m6azmqhtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-9y29i.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-9y29i', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)' 2021-03-01 11:48:32 run:125 [[ERROR]] [TID=9m6azmqhtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 93, in upload_pipeline client = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client _client = kfp.Client() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init self._configure_incluster_auth() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth self.set_user_namespace(old_ns_setting or _get_pod_namespace()) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace with open(Client.LOCAL_KFP_CONTEXT, 'w') as f: PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json' 2021-03-01 11:48:59 run:83 [[DEBUG]] [TID=6gh062igyc] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 11:48:59 run:95 [[DEBUG]] [TID=6gh062igyc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 11:48:59 run:104 [[DEBUG]] [TID=6gh062igyc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 11:48:59 run:114 [[INFO]] [TID=6gh062igyc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})' 2021-03-01 11:48:59 run:83 [[DEBUG]] [TID=jz1pmv0rtp] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 11:48:59 run:95 [[DEBUG]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 11:48:59 run:104 [[DEBUG]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 11:48:59 run:114 [[INFO]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)' 2021-03-01 11:48:59 core:146 [[DEBUG]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 11:49:00 core:205 [[INFO]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-4p5xw.kale.py 2021-03-01 11:49:08 run:83 [[DEBUG]] [TID=uglyl7e4k1] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 11:49:08 run:95 [[DEBUG]] [TID=uglyl7e4k1] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 11:49:08 run:104 [[DEBUG]] [TID=uglyl7e4k1] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 11:49:08 run:114 [[INFO]] [TID=uglyl7e4k1] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})' 2021-03-01 11:49:08 run:83 [[DEBUG]] [TID=qsz52aeqlg] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 11:49:08 run:95 [[DEBUG]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 11:49:08 run:104 [[DEBUG]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 11:49:08 run:114 [[INFO]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)' 2021-03-01 11:49:08 core:146 [[DEBUG]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 11:49:09 core:205 [[INFO]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-a7dfv.kale.py 2021-03-01 11:49:09 run:83 [[DEBUG]] [TID=lfe9ig9mgb] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 11:49:09 run:95 [[DEBUG]] [TID=lfe9ig9mgb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 11:49:09 run:104 [[DEBUG]] [TID=lfe9ig9mgb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 11:49:09 run:114 [[INFO]] [TID=lfe9ig9mgb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-a7dfv.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-a7dfv', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)' 2021-03-01 11:49:09 run:125 [[ERROR]] [TID=lfe9ig9mgb] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 93, in upload_pipeline client = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client _client = kfp.Client() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init self._configure_incluster_auth() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth self.set_user_namespace(old_ns_setting or _get_pod_namespace()) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace with open(Client.LOCAL_KFP_CONTEXT, 'w') as f: PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json' 2021-03-01 11:55:37 run:83 [[DEBUG]] [TID=q5dvo05iyu] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 11:55:37 run:95 [[DEBUG]] [TID=q5dvo05iyu] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 11:55:37 run:104 [[DEBUG]] [TID=q5dvo05iyu] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 11:55:37 run:114 [[INFO]] [TID=q5dvo05iyu] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})' 2021-03-01 11:55:37 run:83 [[DEBUG]] [TID=fwibookgb4] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 11:55:37 run:95 [[DEBUG]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 11:55:37 run:104 [[DEBUG]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 11:55:37 run:114 [[INFO]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)' 2021-03-01 11:55:37 core:146 [[DEBUG]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 11:55:37 core:205 [[INFO]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-tko28.kale.py 2021-03-01 11:55:38 run:83 [[DEBUG]] [TID=q0d19f9jev] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 11:55:38 run:95 [[DEBUG]] [TID=q0d19f9jev] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 11:55:38 run:104 [[DEBUG]] [TID=q0d19f9jev] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 11:55:38 run:114 [[INFO]] [TID=q0d19f9jev] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-tko28.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-tko28', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)' 2021-03-01 11:55:38 run:125 [[ERROR]] [TID=q0d19f9jev] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 93, in upload_pipeline client = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client _client = kfp.Client() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init self._configure_incluster_auth() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth self.set_user_namespace(old_ns_setting or _get_pod_namespace()) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace with open(Client.LOCAL_KFP_CONTEXT, 'w') as f: PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json' 2021-03-01 12:00:01 run:83 [[DEBUG]] [TID=rvov88tmid] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 12:00:01 run:95 [[DEBUG]] [TID=rvov88tmid] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 12:00:01 run:104 [[DEBUG]] [TID=rvov88tmid] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 12:00:01 run:114 [[INFO]] [TID=rvov88tmid] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb)' 2021-03-01 12:00:02 run:83 [[DEBUG]] [TID=vyqjly0bvm] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 12:00:02 run:95 [[DEBUG]] [TID=vyqjly0bvm] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 12:00:02 run:104 [[DEBUG]] [TID=vyqjly0bvm] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 12:00:02 run:114 [[INFO]] [TID=vyqjly0bvm] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 12:00:02 run:83 [[DEBUG]] [TID=1l989i2kh5] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:00:02 run:95 [[DEBUG]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:00:02 run:104 [[DEBUG]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:00:02 run:114 [[INFO]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 12:00:02 nb:205 [[INFO]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 12:00:02 nb:211 [[INFO]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 12:00:02 nb:215 [[INFO]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] PodDefault labels applied on server: 2021-03-01 12:00:02 run:83 [[DEBUG]] [TID=22s6a7ugxe] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 12:00:02 run:95 [[DEBUG]] [TID=22s6a7ugxe] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 12:00:02 run:104 [[DEBUG]] [TID=22s6a7ugxe] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 12:00:02 run:114 [[INFO]] [TID=22s6a7ugxe] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'list_experiments()' 2021-03-01 12:00:02 run:125 [[ERROR]] [TID=22s6a7ugxe] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'list_experiments' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 34, in list_experiments c = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client _client = kfp.Client() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init self._configure_incluster_auth() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth self.set_user_namespace(old_ns_setting or _get_pod_namespace()) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace with open(Client.LOCAL_KFP_CONTEXT, 'w') as f: PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json' 2021-03-01 12:00:24 run:83 [[DEBUG]] [TID=mj1gglvxxj] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 12:00:24 run:95 [[DEBUG]] [TID=mj1gglvxxj] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 12:00:24 run:104 [[DEBUG]] [TID=mj1gglvxxj] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 12:00:24 run:114 [[INFO]] [TID=mj1gglvxxj] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb)' 2021-03-01 12:00:24 run:83 [[DEBUG]] [TID=nd9wykhgs9] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 12:00:24 run:95 [[DEBUG]] [TID=nd9wykhgs9] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 12:00:24 run:104 [[DEBUG]] [TID=nd9wykhgs9] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 12:00:24 run:114 [[INFO]] [TID=nd9wykhgs9] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 12:00:24 run:83 [[DEBUG]] [TID=kwsd5bf0jm] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:00:24 run:95 [[DEBUG]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:00:24 run:104 [[DEBUG]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:00:24 run:114 [[INFO]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 12:00:24 nb:205 [[INFO]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 12:00:24 nb:211 [[INFO]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 12:00:24 nb:215 [[INFO]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] PodDefault labels applied on server: 2021-03-01 12:00:25 run:83 [[DEBUG]] [TID=xkqxkjz8u0] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 12:00:25 run:95 [[DEBUG]] [TID=xkqxkjz8u0] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 12:00:25 run:104 [[DEBUG]] [TID=xkqxkjz8u0] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 12:00:25 run:114 [[INFO]] [TID=xkqxkjz8u0] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'list_experiments()' 2021-03-01 12:00:25 run:125 [[ERROR]] [TID=xkqxkjz8u0] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'list_experiments' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 34, in list_experiments c = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client _client = kfp.Client() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init self._configure_incluster_auth() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth self.set_user_namespace(old_ns_setting or _get_pod_namespace()) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace with open(Client.LOCAL_KFP_CONTEXT, 'w') as f: PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json' 2021-03-01 12:00:47 run:83 [[DEBUG]] [TID=z1xgerubhz] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 12:00:47 run:95 [[DEBUG]] [TID=z1xgerubhz] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 12:00:47 run:104 [[DEBUG]] [TID=z1xgerubhz] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 12:00:47 run:114 [[INFO]] [TID=z1xgerubhz] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'new-ep', 'id': 'new'}, 'experiment_name': 'new-ep', 'katib_run': False, 'pipeline_description': '', 'pipeline_name': '', 'snapshot_volumes': False, 'steps_defaults': [], 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})' 2021-03-01 12:00:47 run:125 [[ERROR]] [TID=z1xgerubhz] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'validate_notebook' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/nb.py", line 107, in validate_notebook Kale(source_notebook_path, notebook_metadata_overrides) File "/usr/local/lib/python3.6/dist-packages/kale/core.py", line 61, in init self.pipeline_metadata = parse_metadata(notebook_metadata) File "/usr/local/lib/python3.6/dist-packages/kale/common/metadatautils.py", line 74, in parse_metadata raise ValueError("Pipeline name {}".format(KALE_NAME_MSG)) ValueError: Pipeline name must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character. 2021-03-01 12:01:05 run:83 [[DEBUG]] [TID=gh1j4snbz8] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 12:01:05 run:95 [[DEBUG]] [TID=gh1j4snbz8] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 12:01:05 run:104 [[DEBUG]] [TID=gh1j4snbz8] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 12:01:05 run:114 [[INFO]] [TID=gh1j4snbz8] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'asdfasdf', 'id': 'new'}, 'experiment_name': 'asdfasdf', 'katib_run': False, 'pipeline_description': '', 'pipeline_name': 'sdfasdf', 'snapshot_volumes': False, 'steps_defaults': [], 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})' 2021-03-01 12:01:05 run:83 [[DEBUG]] [TID=b5yfbhtrdt] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 12:01:05 run:95 [[DEBUG]] [TID=b5yfbhtrdt] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 12:01:05 run:104 [[DEBUG]] [TID=b5yfbhtrdt] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 12:01:05 run:114 [[INFO]] [TID=b5yfbhtrdt] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'asdfasdf', 'id': 'new'}, 'experiment_name': 'asdfasdf', 'katib_run': False, 'pipeline_description': '', 'pipeline_name': 'sdfasdf', 'snapshot_volumes': False, 'steps_defaults': [], 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)' 2021-03-01 12:01:05 run:125 [[ERROR]] [TID=b5yfbhtrdt] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'compile_notebook' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/nb.py", line 92, in compile_notebook pipeline_graph, pipeline_parameters = instance.notebook_to_graph() File "/usr/local/lib/python3.6/dist-packages/kale/core.py", line 102, in notebook_to_graph self.pipeline_metadata) File "/usr/local/lib/python3.6/dist-packages/kale/nbparser/parser.py", line 305, in parse_notebook tags = parse_metadata(c.metadata) File "/usr/local/lib/python3.6/dist-packages/kale/nbparser/parser.py", line 95, in parse_metadata raise ValueError("Unrecognized tag: {}".format(t)) ValueError: Unrecognized tag: parameters 2021-03-01 12:01:53 run:83 [[DEBUG]] [TID=o2nglb8cvr] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 12:01:53 run:95 [[DEBUG]] [TID=o2nglb8cvr] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 12:01:53 run:104 [[DEBUG]] [TID=o2nglb8cvr] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 12:01:53 run:114 [[INFO]] [TID=o2nglb8cvr] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb)' 2021-03-01 12:01:53 run:83 [[DEBUG]] [TID=kqml0x4kf1] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 12:01:53 run:95 [[DEBUG]] [TID=kqml0x4kf1] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 12:01:53 run:104 [[DEBUG]] [TID=kqml0x4kf1] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 12:01:53 run:114 [[INFO]] [TID=kqml0x4kf1] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 12:01:53 run:83 [[DEBUG]] [TID=von88m6u2t] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:01:53 run:95 [[DEBUG]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:01:53 run:104 [[DEBUG]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:01:53 run:114 [[INFO]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 12:01:53 nb:205 [[INFO]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 12:01:54 nb:211 [[INFO]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 12:01:54 nb:215 [[INFO]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] PodDefault labels applied on server: 2021-03-01 12:01:54 run:83 [[DEBUG]] [TID=bt1jayopsc] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 12:01:54 run:95 [[DEBUG]] [TID=bt1jayopsc] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 12:01:54 run:104 [[DEBUG]] [TID=bt1jayopsc] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 12:01:54 run:114 [[INFO]] [TID=bt1jayopsc] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'list_experiments()' 2021-03-01 12:01:54 run:125 [[ERROR]] [TID=bt1jayopsc] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'list_experiments' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 34, in list_experiments c = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client _client = kfp.Client() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init self._configure_incluster_auth() File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth self.set_user_namespace(old_ns_setting or _get_pod_namespace()) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace with open(Client.LOCAL_KFP_CONTEXT, 'w') as f: PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json' 2021-03-01 12:04:47 run:83 [[DEBUG]] [TID=a6sfukms9q] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 12:04:47 run:95 [[DEBUG]] [TID=a6sfukms9q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 12:04:47 run:104 [[DEBUG]] [TID=a6sfukms9q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 12:04:47 run:114 [[INFO]] [TID=a6sfukms9q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)' 2021-03-01 12:04:47 run:83 [[DEBUG]] [TID=03c3vvp33c] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 12:04:47 run:95 [[DEBUG]] [TID=03c3vvp33c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 12:04:47 run:104 [[DEBUG]] [TID=03c3vvp33c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 12:04:47 run:114 [[INFO]] [TID=03c3vvp33c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 12:04:48 run:83 [[DEBUG]] [TID=sna33bbvqp] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:04:48 run:95 [[DEBUG]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:04:48 run:104 [[DEBUG]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:04:48 run:114 [[INFO]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 12:04:48 nb:205 [[INFO]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 12:04:48 nb:211 [[INFO]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 12:04:48 nb:215 [[INFO]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] PodDefault labels applied on server: 2021-03-01 12:04:48 run:83 [[DEBUG]] [TID=abv7oeldfk] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 12:04:48 run:95 [[DEBUG]] [TID=abv7oeldfk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 12:04:48 run:104 [[DEBUG]] [TID=abv7oeldfk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 12:04:48 run:114 [[INFO]] [TID=abv7oeldfk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'list_experiments()' 2021-03-01 12:04:53 run:83 [[DEBUG]] [TID=7e8qrens2k] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 12:04:53 run:95 [[DEBUG]] [TID=7e8qrens2k] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 12:04:53 run:104 [[DEBUG]] [TID=7e8qrens2k] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 12:04:53 run:114 [[INFO]] [TID=7e8qrens2k] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})' 2021-03-01 12:04:54 run:83 [[DEBUG]] [TID=1lagbopnng] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 12:04:54 run:95 [[DEBUG]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 12:04:54 run:104 [[DEBUG]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 12:04:54 run:114 [[INFO]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)' 2021-03-01 12:04:54 core:146 [[DEBUG]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 12:04:54 core:205 [[INFO]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-yl6ca.kale.py 2021-03-01 12:04:54 run:83 [[DEBUG]] [TID=ieogqedcee] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 12:04:54 run:95 [[DEBUG]] [TID=ieogqedcee] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 12:04:54 run:104 [[DEBUG]] [TID=ieogqedcee] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 12:04:54 run:114 [[INFO]] [TID=ieogqedcee] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-yl6ca.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-yl6ca', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)' 2021-03-01 12:04:55 run:125 [[ERROR]] [TID=ieogqedcee] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 692, in upload_pipeline response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 56, in upload_pipeline (data) = self.upload_pipeline_with_http_info(uploadfile, kwargs) # noqa: E501 File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 139, in upload_pipeline_with_http_info collection_formats=collection_formats) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 330, in call_api _preload_content, _request_timeout) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 169, in call_api return_data = self.deserialize(response_data, response_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 241, in deserialize return self.deserialize(data, response_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 278, in deserialize return self.deserialize_datatime(data) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 590, in deserialize_datatime return parse(string) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse res, skipped_tokens = self._parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse l = _timelex.split(timestr) # Splits the timestr into tokens File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split return list(cls(s)) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init '{itype}'.format(itype=instream.class.name)) TypeError: Parser must be a string or character stream, not dict 2021-03-01 12:05:16 run:83 [[DEBUG]] [TID=61slb68pwk] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 12:05:16 run:95 [[DEBUG]] [TID=61slb68pwk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 12:05:16 run:104 [[DEBUG]] [TID=61slb68pwk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 12:05:16 run:114 [[INFO]] [TID=61slb68pwk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})' 2021-03-01 12:05:17 run:83 [[DEBUG]] [TID=7bcpkmjb2t] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 12:05:17 run:95 [[DEBUG]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 12:05:17 run:104 [[DEBUG]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 12:05:17 run:114 [[INFO]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)' 2021-03-01 12:05:17 core:146 [[DEBUG]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 12:05:17 core:205 [[INFO]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-o3pfg.kale.py 2021-03-01 12:05:17 run:83 [[DEBUG]] [TID=sjcjhls9p4] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 12:05:17 run:95 [[DEBUG]] [TID=sjcjhls9p4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 12:05:17 run:104 [[DEBUG]] [TID=sjcjhls9p4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 12:05:17 run:114 [[INFO]] [TID=sjcjhls9p4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline1-o3pfg.pipeline.yaml, pipeline_metadata={'experiment_name': 'aaa', 'pipeline_name': 'newpipeline1-o3pfg', 'pipeline_description': 'kjkjjkj', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'name': 'aaa', 'id': 'new'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)' 2021-03-01 12:05:18 run:125 [[ERROR]] [TID=sjcjhls9p4] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 692, in upload_pipeline response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 56, in upload_pipeline (data) = self.upload_pipeline_with_http_info(uploadfile, kwargs) # noqa: E501 File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 139, in upload_pipeline_with_http_info collection_formats=collection_formats) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 330, in call_api _preload_content, _request_timeout) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 169, in call_api return_data = self.deserialize(response_data, response_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 241, in deserialize return self.deserialize(data, response_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 278, in deserialize return self.deserialize_datatime(data) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 590, in deserialize_datatime return parse(string) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse res, skipped_tokens = self._parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse l = _timelex.split(timestr) # Splits the timestr into tokens File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split return list(cls(s)) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init '{itype}'.format(itype=instream.class.name)) TypeError: Parser must be a string or character stream, not dict 2021-03-01 12:06:26 run:83 [[DEBUG]] [TID=cpwxpbicsc] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 12:06:26 run:95 [[DEBUG]] [TID=cpwxpbicsc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 12:06:26 run:104 [[DEBUG]] [TID=cpwxpbicsc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 12:06:26 run:114 [[INFO]] [TID=cpwxpbicsc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})' 2021-03-01 12:06:26 run:83 [[DEBUG]] [TID=96geeq1vxj] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 12:06:26 run:95 [[DEBUG]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 12:06:26 run:104 [[DEBUG]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 12:06:26 run:114 [[INFO]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)' 2021-03-01 12:06:26 core:146 [[DEBUG]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 12:06:27 core:205 [[INFO]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-e0ggq.kale.py 2021-03-01 12:06:27 run:83 [[DEBUG]] [TID=aazcq2llyl] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 12:06:27 run:95 [[DEBUG]] [TID=aazcq2llyl] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 12:06:27 run:104 [[DEBUG]] [TID=aazcq2llyl] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 12:06:27 run:114 [[INFO]] [TID=aazcq2llyl] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline1-e0ggq.pipeline.yaml, pipeline_metadata={'experiment_name': 'aaa', 'pipeline_name': 'newpipeline1-e0ggq', 'pipeline_description': 'kjkjjkj', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'name': 'aaa', 'id': 'new'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)' 2021-03-01 12:06:27 run:125 [[ERROR]] [TID=aazcq2llyl] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 692, in upload_pipeline response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 56, in upload_pipeline (data) = self.upload_pipeline_with_http_info(uploadfile, kwargs) # noqa: E501 File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 139, in upload_pipeline_with_http_info collection_formats=collection_formats) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 330, in call_api _preload_content, _request_timeout) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 169, in call_api return_data = self.deserialize(response_data, response_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 241, in deserialize return self.deserialize(data, response_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 278, in deserialize return self.deserialize_datatime(data) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 590, in deserialize_datatime return parse(string) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse res, skipped_tokens = self._parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse l = _timelex.split(timestr) # Splits the timestr into tokens File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split return list(cls(s)) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init '{itype}'.format(itype=instream.class.name)) TypeError: Parser must be a string or character stream, not dict 2021-03-01 12:07:22 run:83 [[DEBUG]] [TID=dujb1945gv] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 12:07:22 run:95 [[DEBUG]] [TID=dujb1945gv] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 12:07:22 run:104 [[DEBUG]] [TID=dujb1945gv] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 12:07:22 run:114 [[INFO]] [TID=dujb1945gv] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})' 2021-03-01 12:07:22 run:83 [[DEBUG]] [TID=g4c8akha7j] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 12:07:22 run:95 [[DEBUG]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 12:07:22 run:104 [[DEBUG]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 12:07:22 run:114 [[INFO]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)' 2021-03-01 12:07:22 core:146 [[DEBUG]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 12:07:23 core:205 [[INFO]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-3qkiy.kale.py 2021-03-01 12:07:23 run:83 [[DEBUG]] [TID=is7qozmeux] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 12:07:23 run:95 [[DEBUG]] [TID=is7qozmeux] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 12:07:23 run:104 [[DEBUG]] [TID=is7qozmeux] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 12:07:23 run:114 [[INFO]] [TID=is7qozmeux] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline1-3qkiy.pipeline.yaml, pipeline_metadata={'experiment_name': 'aaa', 'pipeline_name': 'newpipeline1-3qkiy', 'pipeline_description': 'kjkjjkj', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'name': 'aaa', 'id': 'new'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)' 2021-03-01 12:07:23 run:125 [[ERROR]] [TID=is7qozmeux] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name) File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 692, in upload_pipeline response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 56, in upload_pipeline (data) = self.upload_pipeline_with_http_info(uploadfile, kwargs) # noqa: E501 File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 139, in upload_pipeline_with_http_info collection_formats=collection_formats) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 330, in call_api _preload_content, _request_timeout) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 169, in call_api return_data = self.deserialize(response_data, response_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 241, in deserialize return self.deserialize(data, response_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 278, in deserialize return self.deserialize_datatime(data) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 590, in deserialize_datatime return parse(string) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse res, skipped_tokens = self._parse(timestr, **kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse l = _timelex.split(timestr) # Splits the timestr into tokens File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split return list(cls(s)) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init '{itype}'.format(itype=instream.class.name)) TypeError: Parser must be a string or character stream, not dict 2021-03-01 12:08:13 run:83 [[DEBUG]] [TID=bigecwbchg] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 12:08:13 run:95 [[DEBUG]] [TID=bigecwbchg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 12:08:13 run:104 [[DEBUG]] [TID=bigecwbchg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 12:08:13 run:114 [[INFO]] [TID=bigecwbchg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'asdfasdfasdfasdf', 'id': 'new'}, 'experiment_name': 'asdfasdfasdfasdf', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})' 2021-03-01 12:08:13 run:83 [[DEBUG]] [TID=kv6vuclz2u] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 12:08:13 run:95 [[DEBUG]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 12:08:13 run:104 [[DEBUG]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 12:08:13 run:114 [[INFO]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'asdfasdfasdfasdf', 'id': 'new'}, 'experiment_name': 'asdfasdfasdfasdf', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)' 2021-03-01 12:08:13 core:146 [[DEBUG]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 12:08:13 core:205 [[INFO]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-afrqj.kale.py 2021-03-01 12:12:40 run:83 [[DEBUG]] [TID=a8lyhkspxp] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 12:12:40 run:95 [[DEBUG]] [TID=a8lyhkspxp] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 12:12:40 run:104 [[DEBUG]] [TID=a8lyhkspxp] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 12:12:40 run:114 [[INFO]] [TID=a8lyhkspxp] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb)' 2021-03-01 12:12:40 run:83 [[DEBUG]] [TID=ea3v1ay03k] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 12:12:40 run:95 [[DEBUG]] [TID=ea3v1ay03k] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 12:12:40 run:104 [[DEBUG]] [TID=ea3v1ay03k] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 12:12:40 run:114 [[INFO]] [TID=ea3v1ay03k] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 12:12:41 run:83 [[DEBUG]] [TID=ftiq7qpllf] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:12:41 run:95 [[DEBUG]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:12:41 run:104 [[DEBUG]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:12:41 run:114 [[INFO]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 12:12:41 nb:205 [[INFO]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 12:12:41 nb:211 [[INFO]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 12:12:41 nb:215 [[INFO]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] PodDefault labels applied on server: 2021-03-01 12:12:41 run:83 [[DEBUG]] [TID=bf8b9ie7x8] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 12:12:41 run:95 [[DEBUG]] [TID=bf8b9ie7x8] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 12:12:41 run:104 [[DEBUG]] [TID=bf8b9ie7x8] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 12:12:41 run:114 [[INFO]] [TID=bf8b9ie7x8] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'list_experiments()' 2021-03-01 12:12:51 run:83 [[DEBUG]] [TID=8ma26jsqn4] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 12:12:51 run:95 [[DEBUG]] [TID=8ma26jsqn4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 12:12:51 run:104 [[DEBUG]] [TID=8ma26jsqn4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 12:12:51 run:114 [[INFO]] [TID=8ma26jsqn4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)' 2021-03-01 12:13:36 run:83 [[DEBUG]] [TID=zhriwapn38] [] Decoding ctx of RPC function 'rok.check_rok_availability' 2021-03-01 12:13:36 run:95 [[DEBUG]] [TID=zhriwapn38] [] Decoding kwargs of RPC function 'rok.check_rok_availability' 2021-03-01 12:13:36 run:104 [[DEBUG]] [TID=zhriwapn38] [] Importing RPC function 'rok.check_rok_availability' 2021-03-01 12:13:38 run:114 [[INFO]] [TID=zhriwapn38] [] Executing RPC function 'check_rok_availability()' 2021-03-01 12:13:38 rok:123 [[ERROR]] [TID=zhriwapn38] [] Failed to initialize RokClient Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability rok = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or theROK_GW_URL' environment variable. 2021-03-01 12:13:38 run:120 [[ERROR]] [TID=zhriwapn38] [] RPC function 'check_rok_availability' raised an RPCError Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability rok = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or theROK_GW_URL' environment variable.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, **kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 126, in check_rok_availability trans_id=request.trans_id) kale.rpc.errors.RPCServiceUnavailableError 2021-03-01 12:13:39 run:83 [[DEBUG]] [TID=2zzlbq8ork] [] Decoding ctx of RPC function 'nb.resume_notebook_path' 2021-03-01 12:13:39 run:95 [[DEBUG]] [TID=2zzlbq8ork] [] Decoding kwargs of RPC function 'nb.resume_notebook_path' 2021-03-01 12:13:39 run:104 [[DEBUG]] [TID=2zzlbq8ork] [] Importing RPC function 'nb.resume_notebook_path' 2021-03-01 12:13:40 run:114 [[INFO]] [TID=2zzlbq8ork] [] Executing RPC function 'resume_notebook_path(server_root=/home/jovyan)' 2021-03-01 12:13:41 run:83 [[DEBUG]] [TID=468beo3ggg] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 12:13:41 run:95 [[DEBUG]] [TID=468beo3ggg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 12:13:41 run:104 [[DEBUG]] [TID=468beo3ggg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 12:13:41 run:114 [[INFO]] [TID=468beo3ggg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)' 2021-03-01 12:13:41 run:83 [[DEBUG]] [TID=18t185jz2q] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 12:13:41 run:95 [[DEBUG]] [TID=18t185jz2q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 12:13:41 run:104 [[DEBUG]] [TID=18t185jz2q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 12:13:41 run:114 [[INFO]] [TID=18t185jz2q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 12:13:41 run:83 [[DEBUG]] [TID=zbymr0zgsp] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:13:41 run:95 [[DEBUG]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:13:41 run:104 [[DEBUG]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:13:41 run:114 [[INFO]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 12:13:41 nb:205 [[INFO]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 12:13:41 nb:211 [[INFO]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 12:13:41 nb:215 [[INFO]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] PodDefault labels applied on server: 2021-03-01 12:13:41 run:83 [[DEBUG]] [TID=mlxm0ps9wi] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 12:13:41 run:95 [[DEBUG]] [TID=mlxm0ps9wi] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 12:13:41 run:104 [[DEBUG]] [TID=mlxm0ps9wi] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 12:13:41 run:114 [[INFO]] [TID=mlxm0ps9wi] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'list_experiments()' 2021-03-01 12:31:12 run:83 [[DEBUG]] [TID=byphdyfl4h] [] Decoding ctx of RPC function 'rok.check_rok_availability' 2021-03-01 12:31:12 run:95 [[DEBUG]] [TID=byphdyfl4h] [] Decoding kwargs of RPC function 'rok.check_rok_availability' 2021-03-01 12:31:12 run:104 [[DEBUG]] [TID=byphdyfl4h] [] Importing RPC function 'rok.check_rok_availability' 2021-03-01 12:31:14 run:114 [[INFO]] [TID=byphdyfl4h] [] Executing RPC function 'check_rok_availability()' 2021-03-01 12:31:14 rok:123 [[ERROR]] [TID=byphdyfl4h] [] Failed to initialize RokClient Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability rok = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or theROK_GW_URL' environment variable. 2021-03-01 12:31:14 run:120 [[ERROR]] [TID=byphdyfl4h] [] RPC function 'check_rok_availability' raised an RPCError Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability rok = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or theROK_GW_URL' environment variable.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 126, in check_rok_availability trans_id=request.trans_id) kale.rpc.errors.RPCServiceUnavailableError 2021-03-01 12:31:15 run:83 [[DEBUG]] [TID=y9i32q6ip1] [] Decoding ctx of RPC function 'nb.resume_notebook_path' 2021-03-01 12:31:15 run:95 [[DEBUG]] [TID=y9i32q6ip1] [] Decoding kwargs of RPC function 'nb.resume_notebook_path' 2021-03-01 12:31:15 run:104 [[DEBUG]] [TID=y9i32q6ip1] [] Importing RPC function 'nb.resume_notebook_path' 2021-03-01 12:31:16 run:114 [[INFO]] [TID=y9i32q6ip1] [] Executing RPC function 'resume_notebook_path(server_root=/home/jovyan)' 2021-03-01 12:31:16 run:83 [[DEBUG]] [TID=iwbk0qequh] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 12:31:16 run:95 [[DEBUG]] [TID=iwbk0qequh] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 12:31:16 run:104 [[DEBUG]] [TID=iwbk0qequh] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 12:31:16 run:114 [[INFO]] [TID=iwbk0qequh] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)' 2021-03-01 12:31:17 run:83 [[DEBUG]] [TID=1rfupjkvvf] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 12:31:17 run:95 [[DEBUG]] [TID=1rfupjkvvf] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 12:31:17 run:104 [[DEBUG]] [TID=1rfupjkvvf] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 12:31:17 run:114 [[INFO]] [TID=1rfupjkvvf] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 12:31:17 run:83 [[DEBUG]] [TID=mxdqele2cx] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:31:17 run:95 [[DEBUG]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:31:17 run:104 [[DEBUG]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:31:17 run:114 [[INFO]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 12:31:17 nb:205 [[INFO]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 12:31:17 nb:211 [[INFO]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 12:31:17 nb:215 [[INFO]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] PodDefault labels applied on server: 2021-03-01 12:31:17 run:83 [[DEBUG]] [TID=br5u2s032y] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 12:31:17 run:95 [[DEBUG]] [TID=br5u2s032y] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 12:31:17 run:104 [[DEBUG]] [TID=br5u2s032y] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 12:31:17 run:114 [[INFO]] [TID=br5u2s032y] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'list_experiments()' 2021-03-01 12:31:35 run:83 [[DEBUG]] [TID=rnxs15j17o] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 12:31:35 run:95 [[DEBUG]] [TID=rnxs15j17o] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 12:31:35 run:104 [[DEBUG]] [TID=rnxs15j17o] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 12:31:35 run:114 [[INFO]] [TID=rnxs15j17o] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': 'new', 'name': 'asdfasdfasdfasdf'}, 'experiment_name': 'asdfasdfasdfasdf', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})' 2021-03-01 12:31:35 run:83 [[DEBUG]] [TID=l5gn4ok577] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 12:31:35 run:95 [[DEBUG]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 12:31:35 run:104 [[DEBUG]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 12:31:35 run:114 [[INFO]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': 'new', 'name': 'asdfasdfasdfasdf'}, 'experiment_name': 'asdfasdfasdfasdf', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)' 2021-03-01 12:31:35 core:146 [[DEBUG]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run ------------- 2021-03-01 12:31:36 core:205 [[INFO]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-goxmo.kale.py 2021-03-01 12:31:36 run:83 [[DEBUG]] [TID=fd0gzcnyl5] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 12:31:36 run:95 [[DEBUG]] [TID=fd0gzcnyl5] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 12:31:36 run:104 [[DEBUG]] [TID=fd0gzcnyl5] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 12:31:36 run:114 [[INFO]] [TID=fd0gzcnyl5] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline1-goxmo.pipeline.yaml, pipeline_metadata={'experiment_name': 'asdfasdfasdfasdf', 'pipeline_name': 'newpipeline1-goxmo', 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': 'new', 'name': 'asdfasdfasdfasdf'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)' 2021-03-01 12:31:36 run:125 [[ERROR]] [TID=fd0gzcnyl5] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp/_client.py", line 881, in upload_pipeline response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 69, in upload_pipeline return self.upload_pipeline_with_http_info(uploadfile, kwargs) # noqa: E501 File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 177, in upload_pipeline_with_http_info collection_formats=collection_formats) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 369, in call_api _preload_content, _request_timeout, _host) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 208, in call_api return_data = self.deserialize(response_data, response_type) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize(data, response_type) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 319, in deserialize return self.deserialize_model(data, klass) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 658, in deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 319, in deserialize return self.deserialize_model(data, klass) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 658, in __deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 317, in deserialize return self.deserialize_datetime(data) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 624, in deserialize_datetime return parse(string) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse res, skipped_tokens = self._parse(timestr, **kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse l = _timelex.split(timestr) # Splits the timestr into tokens File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split return list(cls(s)) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init '{itype}'.format(itype=instream.class.name)) TypeError: Parser must be a string or character stream, not dict 2021-03-01 12:39:10 run:83 [[DEBUG]] [TID=eq33czreem] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 12:39:10 run:95 [[DEBUG]] [TID=eq33czreem] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 12:39:10 run:104 [[DEBUG]] [TID=eq33czreem] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 12:39:10 run:114 [[INFO]] [TID=eq33czreem] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb)' 2021-03-01 12:39:10 run:83 [[DEBUG]] [TID=1b0dgoptqq] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 12:39:10 run:95 [[DEBUG]] [TID=1b0dgoptqq] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 12:39:10 run:104 [[DEBUG]] [TID=1b0dgoptqq] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 12:39:10 run:114 [[INFO]] [TID=1b0dgoptqq] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 12:39:10 run:83 [[DEBUG]] [TID=2iygvtbobb] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:39:10 run:95 [[DEBUG]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:39:10 run:104 [[DEBUG]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 12:39:10 run:114 [[INFO]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 12:39:10 nb:205 [[INFO]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 12:39:11 nb:211 [[INFO]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 12:39:11 nb:215 [[INFO]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] PodDefault labels applied on server: 2021-03-01 12:39:11 run:83 [[DEBUG]] [TID=pfq35lwh69] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 12:39:11 run:95 [[DEBUG]] [TID=pfq35lwh69] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 12:39:11 run:104 [[DEBUG]] [TID=pfq35lwh69] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 12:39:11 run:114 [[INFO]] [TID=pfq35lwh69] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'list_experiments()' 2021-03-01 12:39:42 run:83 [[DEBUG]] [TID=kujjlftjrv] [] Decoding ctx of RPC function 'nb.validate_notebook' 2021-03-01 12:39:42 run:95 [[DEBUG]] [TID=kujjlftjrv] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook' 2021-03-01 12:39:42 run:104 [[DEBUG]] [TID=kujjlftjrv] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.validate_notebook' 2021-03-01 12:39:42 run:114 [[INFO]] [TID=kujjlftjrv] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb, notebook_metadata_overrides={'experiment': {'id': 'new', 'name': 'dog-breed-katib'}, 'experiment_name': 'dog-breed-katib', 'katib_metadata': {'parameters': [{'feasibleSpace': {'max': '2100', 'min': '128', 'step': '128'}, 'name': 'nodes_number', 'parameterType': 'double'}, {'feasibleSpace': {'list': ['0.0001', '0.0005', '0.001', '0.01', '0.1', '0.005', '0.05', '0.5']}, 'name': 'lr', 'parameterType': 'categorical'}], 'objective': {'additionalMetricNames': [], 'goal': 90, 'objectiveMetricName': 'test-accuracy-resnet', 'type': 'maximize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 200, 'maxFailedTrialCount': 3, 'parallelTrialCount': 1}, 'katib_run': False, 'pipeline_description': '', 'pipeline_name': 'dog-breed', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'autosnapshot': False, 'snapshot_volumes': False, 'steps_defaults': []})' 2021-03-01 12:39:42 run:83 [[DEBUG]] [TID=6ozisotmwb] [] Decoding ctx of RPC function 'nb.compile_notebook' 2021-03-01 12:39:42 run:95 [[DEBUG]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook' 2021-03-01 12:39:42 run:104 [[DEBUG]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.compile_notebook' 2021-03-01 12:39:42 run:114 [[INFO]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb, notebook_metadata_overrides={'experiment': {'id': 'new', 'name': 'dog-breed-katib'}, 'experiment_name': 'dog-breed-katib', 'katib_metadata': {'parameters': [{'feasibleSpace': {'max': '2100', 'min': '128', 'step': '128'}, 'name': 'nodes_number', 'parameterType': 'double'}, {'feasibleSpace': {'list': ['0.0001', '0.0005', '0.001', '0.01', '0.1', '0.005', '0.05', '0.5']}, 'name': 'lr', 'parameterType': 'categorical'}], 'objective': {'additionalMetricNames': [], 'goal': 90, 'objectiveMetricName': 'test-accuracy-resnet', 'type': 'maximize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 200, 'maxFailedTrialCount': 3, 'parallelTrialCount': 1}, 'katib_run': False, 'pipeline_description': '', 'pipeline_name': 'dog-breed', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'autosnapshot': False, 'snapshot_volumes': False, 'steps_defaults': []}, debug=False)' 2021-03-01 12:39:42 core:146 [[DEBUG]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] ------------- Kale Start Run ------------- 2021-03-01 12:39:44 core:205 [[INFO]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Pipeline code saved at /home/jovyan/kale/examples/dog-breed-classification/dog-breed-a9bpu.kale.py 2021-03-01 12:39:44 driver:124 [[INFO]] Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt 2021-03-01 12:39:44 driver:124 [[INFO]] Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt 2021-03-01 12:39:45 run:83 [[DEBUG]] [TID=n8twyqkq8p] [] Decoding ctx of RPC function 'kfp.upload_pipeline' 2021-03-01 12:39:45 run:95 [[DEBUG]] [TID=n8twyqkq8p] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline' 2021-03-01 12:39:45 run:104 [[DEBUG]] [TID=n8twyqkq8p] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'kfp.upload_pipeline' 2021-03-01 12:39:45 run:114 [[INFO]] [TID=n8twyqkq8p] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=kale/examples/dog-breed-classification/dog-breed-a9bpu.pipeline.yaml, pipeline_metadata={'experiment_name': 'dog-breed-katib', 'pipeline_name': 'dog-breed-a9bpu', 'pipeline_description': '', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan/kale/examples/dog-breed-classification', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': 'new', 'name': 'dog-breed-katib'}, 'katib_metadata': {'parameters': [{'feasibleSpace': {'max': '2100', 'min': '128', 'step': '128'}, 'name': 'nodes_number', 'parameterType': 'double'}, {'feasibleSpace': {'list': ['0.0001', '0.0005', '0.001', '0.01', '0.1', '0.005', '0.05', '0.5']}, 'name': 'lr', 'parameterType': 'categorical'}], 'objective': {'additionalMetricNames': [], 'goal': 90, 'objectiveMetricName': 'test-accuracy-resnet', 'type': 'maximize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 200, 'maxFailedTrialCount': 3, 'parallelTrialCount': 1}, 'katib_run': False, 'parameters_names': ['learning_rate', 'nodes_number'], 'parameters_types': ['float', 'int'], 'parameters_values': [0.0001, 256]}, overwrite=False)' 2021-03-01 12:39:45 run:125 [[ERROR]] [TID=n8twyqkq8p] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] RPC function 'upload_pipeline' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp/_client.py", line 881, in upload_pipeline response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 69, in upload_pipeline return self.upload_pipeline_with_http_info(uploadfile, kwargs) # noqa: E501 File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 177, in upload_pipeline_with_http_info collection_formats=collection_formats) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 369, in call_api _preload_content, _request_timeout, _host) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 208, in call_api return_data = self.deserialize(response_data, response_type) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 280, in deserialize return self.deserialize(data, response_type) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 319, in deserialize return self.deserialize_model(data, klass) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 658, in deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 319, in deserialize return self.deserialize_model(data, klass) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 658, in __deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 317, in deserialize return self.deserialize_datetime(data) File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 624, in deserialize_datetime return parse(string) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse res, skipped_tokens = self._parse(timestr, kwargs) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse l = _timelex.split(timestr) # Splits the timestr into tokens File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split return list(cls(s)) File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init '{itype}'.format(itype=instream.class.name)) TypeError: Parser must be a string or character stream, not dict 2021-03-01 13:17:00 run:83 [[DEBUG]] [TID=qhlnogi3bb] [] Decoding ctx of RPC function 'rok.check_rok_availability' 2021-03-01 13:17:00 run:95 [[DEBUG]] [TID=qhlnogi3bb] [] Decoding kwargs of RPC function 'rok.check_rok_availability' 2021-03-01 13:17:00 run:104 [[DEBUG]] [TID=qhlnogi3bb] [] Importing RPC function 'rok.check_rok_availability' 2021-03-01 13:17:02 run:114 [[INFO]] [TID=qhlnogi3bb] [] Executing RPC function 'check_rok_availability()' 2021-03-01 13:17:02 rok:123 [[ERROR]] [TID=qhlnogi3bb] [] Failed to initialize RokClient Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability rok = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or theROK_GW_URL' environment variable. 2021-03-01 13:17:02 run:120 [[ERROR]] [TID=qhlnogi3bb] [] RPC function 'check_rok_availability' raised an RPCError Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability rok = _get_client() File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or theROK_GW_URL' environment variable.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, **kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 126, in check_rok_availability trans_id=request.trans_id) kale.rpc.errors.RPCServiceUnavailableError 2021-03-01 13:17:04 run:83 [[DEBUG]] [TID=573x0u1g4z] [] Decoding ctx of RPC function 'nb.resume_notebook_path' 2021-03-01 13:17:04 run:95 [[DEBUG]] [TID=573x0u1g4z] [] Decoding kwargs of RPC function 'nb.resume_notebook_path' 2021-03-01 13:17:04 run:104 [[DEBUG]] [TID=573x0u1g4z] [] Importing RPC function 'nb.resume_notebook_path' 2021-03-01 13:17:05 run:114 [[INFO]] [TID=573x0u1g4z] [] Executing RPC function 'resume_notebook_path(server_root=/home/jovyan)' 2021-03-01 13:17:06 run:83 [[DEBUG]] [TID=paouw6e22o] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 13:17:06 run:95 [[DEBUG]] [TID=paouw6e22o] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 13:17:06 run:104 [[DEBUG]] [TID=paouw6e22o] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 13:17:06 run:114 [[INFO]] [TID=paouw6e22o] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb)' 2021-03-01 13:17:07 run:83 [[DEBUG]] [TID=4cheyuki97] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 13:17:07 run:95 [[DEBUG]] [TID=4cheyuki97] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 13:17:07 run:104 [[DEBUG]] [TID=4cheyuki97] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 13:17:07 run:114 [[INFO]] [TID=4cheyuki97] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 13:17:07 run:83 [[DEBUG]] [TID=vg4gmhe3yb] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 13:17:07 run:95 [[DEBUG]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 13:17:07 run:104 [[DEBUG]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 13:17:07 run:114 [[INFO]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 13:17:07 nb:205 [[INFO]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 13:17:07 nb:211 [[INFO]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 13:17:07 nb:215 [[INFO]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] PodDefault labels applied on server: 2021-03-01 13:17:08 run:83 [[DEBUG]] [TID=hkaedck5w5] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 13:17:08 run:95 [[DEBUG]] [TID=hkaedck5w5] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 13:17:08 run:104 [[DEBUG]] [TID=hkaedck5w5] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 13:17:08 run:114 [[INFO]] [TID=hkaedck5w5] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'list_experiments()'

elikatsis commented 3 years ago

Hi, could you also provide

This sounds like an old KFP API server bug: https://github.com/kubeflow/pipelines/issues/2764, https://github.com/kubeflow/pipelines/issues/3441, https://github.com/kubeflow/kubeflow/issues/4830