search

canonical / mlflow-operator

MLFlow Operators
Apache License 2.0
12 stars 11 forks source link

relational-db-relation-changed fails when minio is not separated #265

Closed natalytvinova closed 1 week ago

natalytvinova commented 1 month ago

Bug Description

Hi team,

I deployed mlflow-server and related it to a clustered mysql. This leads to relational-db-relation-changed hooks failure.

I also tried the same bundle, but without clustering mysql and the charm works fine.

Bundles: auth-overlay.yaml.txt storage-overlay.yaml.txt mlflow-overlay.yaml.txt kubeflow.yaml.txt cos-integration-overlay.yaml.txt

To Reproduce

  1. juju deploy --overlay auth-overlay.yaml --overlay cos-integration-overlay.yaml --overlay mlflow-overlay.yaml --overlay storage-overlay.yaml ./kubeflow.yaml --trust -m kubeflow

Environment

Juju 3.5.3 Kubernetes: AKS version 1.29

Relevant Log Output

2024-09-09 07:23:02,058 DEBUG    ops 2.14.0 up and running.
2024-09-09 07:23:02,071 WARNING  2 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion.
2024-09-09 07:23:02,083 DEBUG    Emitting Juju event relational_db_relation_changed.
2024-09-09 07:23:02,210 DEBUG    Got following database data: {75: {'database': 'mlflow', 'endpoints': 'mlflow-mysql-primary.kubeflow.svc.cluster.local:3306', 'password': '', 'read-only-endpoints': 'mlflow-mysql-replicas:3306', 'username': 'relation-75_8cde42d8e98b49', 'version': '8.0.37-0ubuntu0.22.04.3'}}
2024-09-09 07:23:02,215 DEBUG    Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2024-09-09 07:23:02,220 DEBUG    Changing event name from before-call.apigateway to before-call.api-gateway
2024-09-09 07:23:02,224 DEBUG    Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2024-09-09 07:23:02,229 DEBUG    Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2024-09-09 07:23:02,233 DEBUG    Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2024-09-09 07:23:02,238 DEBUG    Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2024-09-09 07:23:02,242 DEBUG    Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2024-09-09 07:23:02,247 DEBUG    Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2024-09-09 07:23:02,251 DEBUG    Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2024-09-09 07:23:02,255 DEBUG    Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2024-09-09 07:23:02,259 DEBUG    Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2024-09-09 07:23:02,265 DEBUG    Loading JSON file: /var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/data/endpoints.json
2024-09-09 07:23:02,278 DEBUG    Loading JSON file: /var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/data/sdk-default-configuration.json
2024-09-09 07:23:02,283 DEBUG    Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fa90ef81670>
2024-09-09 07:23:02,295 DEBUG    Loading JSON file: /var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/data/s3/2006-03-01/service-2.json
2024-09-09 07:23:02,313 DEBUG    Loading JSON file: /var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json
2024-09-09 07:23:02,320 DEBUG    Loading JSON file: /var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/data/partitions.json
2024-09-09 07:23:02,325 DEBUG    Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fa90f04b940>
2024-09-09 07:23:02,330 DEBUG    Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7fa90ee8daf0>
2024-09-09 07:23:02,345 DEBUG    Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fa90f04b700>
2024-09-09 07:23:02,353 DEBUG    Setting s3 timeout as (60, 60)
2024-09-09 07:23:02,362 DEBUG    Loading JSON file: /var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/data/_retry.json
2024-09-09 07:23:02,366 DEBUG    Registering retry handlers for service: s3
2024-09-09 07:23:02,371 DEBUG    Registering S3 region redirector handler
2024-09-09 07:23:02,376 DEBUG    Event before-parameter-build.s3.HeadBucket: calling handler <function validate_bucket_name at 0x7fa90ef0fd30>
2024-09-09 07:23:02,380 DEBUG    Event before-parameter-build.s3.HeadBucket: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fa90ef17b80>
2024-09-09 07:23:02,384 DEBUG    Event before-parameter-build.s3.HeadBucket: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fa90e43c880>>
2024-09-09 07:23:02,388 DEBUG    Event before-parameter-build.s3.HeadBucket: calling handler <function generate_idempotent_uuid at 0x7fa90ef0fb80>
2024-09-09 07:23:02,393 DEBUG    Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fa90ef17d30>
2024-09-09 07:23:02,397 DEBUG    Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fa90e43c880>>
2024-09-09 07:23:02,402 DEBUG    Calling endpoint provider with parameters: {'Bucket': 'mlflow', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'http://minio.kubeflow:9000', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-09-09 07:23:02,407 DEBUG    Endpoint provider result: http://minio.kubeflow:9000/mlflow
2024-09-09 07:23:02,412 DEBUG    Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-09-09 07:23:02,416 DEBUG    Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2024-09-09 07:23:02,421 DEBUG    Event before-call.s3.HeadBucket: calling handler <function add_expect_header at 0x7fa90ef160d0>
2024-09-09 07:23:02,424 DEBUG    Event before-call.s3.HeadBucket: calling handler <function add_recursion_detection_header at 0x7fa90ef0f820>
2024-09-09 07:23:02,428 DEBUG    Event before-call.s3.HeadBucket: calling handler <function inject_api_version_header_if_needed at 0x7fa90ef17430>
2024-09-09 07:23:02,432 DEBUG    Making request for OperationModel(name=HeadBucket) with params: {'url_path': '', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.28.25 md/Botocore#1.31.25 ua/2.0 os/linux#5.15.0-1071-azure md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.25'}, 'body': b'', 'auth_path': '/mlflow/', 'url': 'http://minio.kubeflow:9000/mlflow', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fa90e635b50>, 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'mlflow', 'params': {'Bucket': 'mlflow'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}}
2024-09-09 07:23:02,436 DEBUG    Event request-created.s3.HeadBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa90e635670>>
2024-09-09 07:23:02,440 DEBUG    Event choose-signer.s3.HeadBucket: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fa90ee5b1c0>>
2024-09-09 07:23:02,444 DEBUG    Event choose-signer.s3.HeadBucket: calling handler <function set_operation_specific_signer at 0x7fa90ef0fa60>
2024-09-09 07:23:02,448 DEBUG    Event before-sign.s3.HeadBucket: calling handler <function remove_arn_from_signing_path at 0x7fa90ef17ca0>
2024-09-09 07:23:02,453 DEBUG    Calculating signature using v4 auth.
2024-09-09 07:23:02,456 DEBUG    CanonicalRequest:
HEAD
/mlflow

host:minio.kubeflow:9000
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240909T072302Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-09-09 07:23:02,460 DEBUG    StringToSign:
AWS4-HMAC-SHA256
20240909T072302Z
20240909/us-east-1/s3/aws4_request
3641f1173040f2047f9e4ee7e61db14cfac36b24851294d4532379f78ebb94c7
2024-09-09 07:23:02,464 DEBUG    Signature:
a2878f680550b667650d285d5a6a51c7699c2a79113be9e4c5d556204b94d252
2024-09-09 07:23:02,468 DEBUG    Event request-created.s3.HeadBucket: calling handler <function add_retry_headers at 0x7fa90ef17af0>
2024-09-09 07:23:02,472 DEBUG    Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=http://minio.kubeflow:9000/mlflow, headers={'User-Agent': b'Boto3/1.28.25 md/Botocore#1.31.25 ua/2.0 os/linux#5.15.0-1071-azure md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.25', 'X-Amz-Date': b'20240909T072302Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=vgnrkqel543g\n/20240909/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a2878f680550b667650d285d5a6a51c7699c2a79113be9e4c5d556204b94d252', 'amz-sdk-invocation-id': b'911b6cc0-fb3c-4bbb-9dc4-0fff4642e26d', 'amz-sdk-request': b'attempt=1'}>
2024-09-09 07:23:02,477 DEBUG    Starting new HTTP connection (1): minio.kubeflow:9000
2024-09-09 07:23:02,481 DEBUG    Exception received when sending urllib3 HTTP request
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/httpsession.py", line 465, in send
    urllib_response = conn.urlopen(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/urllib3/connectionpool.py", line 415, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/awsrequest.py", line 96, in request
    rval = super().request(method, url, body, headers, *args, **kwargs)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1297, in _send_request
    self.putheader(hdr, value)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/urllib3/connection.py", line 224, in putheader
    _HTTPConnection.putheader(self, header, *values)
  File "/usr/lib/python3.8/http/client.py", line 1234, in putheader
    raise ValueError('Invalid header value %r' % (values[i],))
ValueError: Invalid header value b'AWS4-HMAC-SHA256 Credential=vgnrkqel543g\n/20240909/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a2878f680550b667650d285d5a6a51c7699c2a79113be9e4c5d556204b94d252'
2024-09-09 07:23:02,487 DEBUG    Event needs-retry.s3.HeadBucket: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa90e43c820>
2024-09-09 07:23:02,497 ERROR    Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/httpsession.py", line 465, in send
    urllib_response = conn.urlopen(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/urllib3/connectionpool.py", line 415, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/awsrequest.py", line 96, in request
    rval = super().request(method, url, body, headers, *args, **kwargs)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1297, in _send_request
    self.putheader(hdr, value)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/urllib3/connection.py", line 224, in putheader
    _HTTPConnection.putheader(self, header, *values)
  File "/usr/lib/python3.8/http/client.py", line 1234, in putheader
    raise ValueError('Invalid header value %r' % (values[i],))
ValueError: Invalid header value b'AWS4-HMAC-SHA256 Credential=vgnrkqel543g\n/20240909/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a2878f680550b667650d285d5a6a51c7699c2a79113be9e4c5d556204b94d252'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 499, in <module>
    main(MlflowCharm)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/ops/main.py", line 548, in main
    manager.run()
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/ops/main.py", line 527, in run
    self._emit()
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/ops/main.py", line 516, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 450, in _on_event
    if not self._validate_default_s3_bucket_name_and_access(
  File "./src/charm.py", line 358, in _validate_default_s3_bucket_name_and_access
    is_bucket_accessible = s3_wrapper.check_if_bucket_accessible(bucket_name)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/src/services/s3.py", line 40, in check_if_bucket_accessible
    self.client.head_bucket(Bucket=bucket_name)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/client.py", line 963, in _make_api_call
    http, parsed_response = self._make_request(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/client.py", line 986, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/retryhandler.py", line 307, in _should_retry
    return self._checker(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/var/lib/juju/agents/unit-mlflow-server-0/charm/venv/botocore/httpsession.py", line 510, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: Invalid header value b'AWS4-HMAC-SHA256 Credential=vgnrkqel543g\n/20240909/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a2878f680550b667650d285d5a6a51c7699c2a79113be9e4c5d556204b94d252'

Additional Context

No response

syncronize-issues-to-jira[bot] commented 1 month ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6213.

This message was autogenerated

kimwnasptd commented 1 month ago

@natalytvinova could you give some more context on what is included in the mlflow-overlay.yaml?

We can't reproduce the issue without the context of what's in those bundles.

kimwnasptd commented 1 month ago

Also, what do you mean with clustered mysql?

natalytvinova commented 1 month ago

@kimwnasptd I'm sorry, I didn't get your question about the context. Can you see the bundle attached correctly or are there any issues? Clustered mysql I mean is 3 mysql units

natalytvinova commented 1 month ago

Hm, I'm actually able to reproduce this even with 1 mlflow-mysql unit

But maybe some resources didn't clean up like this time: https://github.com/canonical/istio-operators/issues/549

misohu commented 1 month ago

I have synced with @natalytvinova the problem was that in the original deployment we were using the same minio as for kfp ... After we have deployed separate minio and related it to mlflow the issue got resolved.

@natalytvinova will let me know if we need to use the same mino for kfp and mlflow.

kimwnasptd commented 1 month ago

@misohu what is the issue with relating KFP MinIO with MLflow and how is it related to mysql relations?

We will need to ensure that users could also use KFP MinIO if needed. Should we have a dedicated issue with logs around this to track the KFP MinIO integration in this case?

misohu commented 1 month ago

I dont think this issue has anything to do with mysql ... also the error above is from botocore which is lib used in mlflow to talk to minio (thats how I knew its minio).

The original design for mlflow was that it has its own minio so we never tested that situation. I did not dig deeper for now. so @kimwnasptd if we think we should support this one mino solution I think we can track it in this issue. WDYT ?

Barteus commented 1 month ago

If we have two minio, which one will be added as a PodDefault to the notebooks?

misohu commented 1 month ago

so mlflow has its own poddefaults called 

"mlflow-server-access-minio",
"mlflow-server-minio",

you can enable those when creating notebook in config

here is the content for each of those.

SO if you pick that mlflow config you will use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY pointing to mlflow's minio in the notbook

misohu commented 1 month ago

I can see a problem where user may want to use mlflow from inside the notebook and talk to kfp minio at the same time ... the env variables will be set to MLflow minio. But there is no other poddefault for the kfp minio. Only if you create one manually. So if you have notebook where you want to talk to kfp minio not to mlflow then dont select the mlflow poddefault.

misohu commented 1 month ago

@Barteus So we distinguish between 3 usecases for minio:

  1. Using minio between pipeline steps to move data around: This works in case of 2 minios as the kfp components are using the kfp minio secret behind the scenes.
  2. Accessing kfp minio from user notebook: This is not possible out of the box (not even for the environment without mlflow) as we are not creating a poddefault with minio secrets. User here needs to create their own poddefault with the details about kfp minio.
  3. Accesing mlflow's minio: When deploying kubeflow + resource dispatcher + mlflow we are creating two poddefaults. 
    "mlflow-server-access-minio",
"mlflow-server-minio",

    Which mounts credentials for mlflow's minio to user notebook under AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

So if you want to have a notebook which can talk to mlflow and kfp minio you need to:

  1. Create custom poddefault with different env variable names (e.g KFP_MINIO_ACCESS_KEY_ID and KFP_MINIO_SECRET_ACCESS_KEY and the variables for URL).
  2. Choose both poddefaults for your notebook.
  3. Use Minio client which allows you to speciffy values for access key and secret access key. E.g. this one. 
    # Initialize a MinIO client
mc = Minio(
    endpoint=KFP_MINIO_HOST,
    access_key=os.environ["KFP_MINIO_ACCESS_KEY_ID"],
    secret_key=os.environ["KFP_MINIO_SECRET_ACCESS_KEY"],
    secure=False,
)
mvlassis commented 1 week ago

Closing this issue, if you have a similar issue please refer to this comment for context.