search

getindata / kedro-kubeflow

Kedro Plugin to support running workflows on Kubeflow Pipelines
https://kedro-kubeflow.readthedocs.io
Apache License 2.0
50 stars 21 forks source link

kedro kubeflow run-once fails with python3.6 #122

Closed balakavinpon closed 2 years ago

balakavinpon commented 2 years ago

We just starting to use Kedro on GCP. Your project is an ideal one for us to go with in put GPU journey (Vertex AI). For some internal reasons, we are using Python3.6 as the standard. my job was working fine when I go kedro run in the local, but if I run kedro kubeflow run-once I'm getting below error.

Note: kedro kubeflow compile -o file_name.json - works well and If I manually uploaded this to Vertex AI, it works perfectly

Error: $ kedro kubeflow run-once 2022-03-30 11:32:00,475 - root - INFO - Registered hooks from 1 installed plugin(s): kedro-kubeflow-0.6.0 2022-03-30 11:32:00,538 - root - INFO - Registered CLI hooks from 1 installed plugin(s): kedro-telemetry-0.1.4 2022-03-30 11:32:01,289 - root - INFO - Registered CLI hooks from 1 installed plugin(s): kedro-telemetry-0.1.4 Kedro-Telemetry is installed, but you have opted out of sharing usage analytics so none will be collected. 2022-03-30 11:32:01,371 - kedro.framework.session.store - INFO - read() not implemented for BaseSessionStore. Assuming empty store. /tmp/kedro-env/lib/python3.6/site-packages/kfp/v2/google/client/client.py:173: FutureWarning: AIPlatformClient will be deprecated in v2.0.0. Please use PipelineJob https://googleapis.dev/python/aiplatform/latest/_modules/google/cloud/aiplatform/pipeline_jobs.html in Vertex SDK. Install the SDK using "pip install google-cloud-aiplatform" category=FutureWarning, Traceback (most recent call last): File "/tmp/kedro-env/bin/kedro", line 8, in sys.exit(main()) File "/tmp/kedro-env/lib/python3.6/site-packages/kedro/framework/cli/cli.py", line 266, in main cli_collection() File "/tmp/kedro-env/lib/python3.6/site-packages/click/core.py", line 829, in call return self.main(*args, kwargs) File "/tmp/kedro-env/lib/python3.6/site-packages/kedro/framework/cli/cli.py", line 216, in main extra, File "/tmp/kedro-env/lib/python3.6/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/tmp/kedro-env/lib/python3.6/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/kedro-env/lib/python3.6/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/kedro-env/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/tmp/kedro-env/lib/python3.6/site-packages/click/core.py", line 610, in invoke return callback(*args, *kwargs) File "/tmp/kedro-env/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), args, kwargs) File "/tmp/kedro-env/lib/python3.6/site-packages/kedro_kubeflow/cli.py", line 103, in run_once parameters=format_params(params), TypeError: run_once() got an unexpected keyword argument 'parameters'

szczeles commented 2 years ago

Hey @balakavinpon , thanks for the issue description. We're going to drop support for VertexAI in kedro-kubeflow plugin soon, as we published a new plugin, focuses on VertexAI support: https://github.com/getindata/kedro-vertexai/

Please check if the latest relese of kedro-vertexai solves your issue. It not, please open a new one on kedro-vertexai issues.

em-pe commented 2 years ago

Following up on what @szczeles wrote - we have a separate plugin for kedro-vertexai that we will actively maintain, however it still requires python 3.8 od higher.

Have you considered using conda or dockerized environment to run it?

em-pe commented 2 years ago

Closing this issue as we don't plan to support old python versions.