Failed to get object from minio on microk8s

[kbegiedza]

I have kubeflow running via MicroK8s on VM. When I want to access my pipeline component's output artifact:

error occurs:

Failed to get object in bucket mlpipeline at path artifacts/hello-world-5f52l/hello-world-5f52l-940018/mlpipeline-metrics.tgz:
S3Error: The request signature we calculated does not match the signature you provided.
Check your key and signing method.

When I head to Artifacts console shows me:

Environment:

• Kubeflow version: dev_local - commit: 743746b96e6efc502c33e1e529f2fe89ce09481c
• Kubernetes platform: microk8s
• Kubernetes version: v1.19.8-34+811e9feeade1d3
• OS: Ubuntu 18.04.5 LTS

[knkski]

@Ursanon: Can you post the pipeline that you ran and encountered this issue with?

[kbegiedza]

@knkski I ran this pipeline:

Code:

import kfp
from kfp import dsl
from kfp.components import InputPath, OutputArtifact, OutputPath, create_component_from_func

def run_nvidia_smi():
    return dsl.ContainerOp(
        name='run_nvidia_smi',
        image='tensorflow/tensorflow:latest-gpu',
        command=['sh', '-c'],
        arguments=['nvidia-smi']
    )

def add_numbers(numbers: list) -> int:
    return sum(numbers)

def avg_numbers(sum: int, numbers: list) -> float:
    return sum / len(numbers)

def save_metrics(sum: int, avg: float, mlpipeline_metrics_path: OutputPath('Metrics')):
    import json
    metrics = {
        'metrics': [
            {
                'name': 'add-result',
                'numberValue': sum,
                'format': 'RAW'
            },
            {
                'name': 'avg-result',
                'numberValue': avg,
                'format': 'RAW'
            },
        ]
    }

    with open(mlpipeline_metrics_path, 'w') as f:
        json.dump(metrics, f)

def save_artifact(sum: int, avg: float, output_models_path: OutputPath('Models')):
    import json
    metrics = {
        'metrics': [
            {
                'name': 'add-result',
                'numberValue': sum,
                'format': 'RAW'
            },
            {
                'name': 'avg-result',
                'numberValue': avg,
                'format': 'RAW'
            },
        ]
    }

    with open(output_models_path, 'w') as f:
        json.dump(metrics, f)

add_op = create_component_from_func(add_numbers)
avg_op = create_component_from_func(avg_numbers)
save_metrics_op = create_component_from_func(save_metrics)
save_artifact_op = create_component_from_func(save_artifact)

@ dsl.pipeline(
    name='Hello world',
    description='Hello world description',

)
def hello_world(numbers: list):
    run_nvidia_smi()

    add_result = add_op(numbers)
    add_output = add_result.outputs['Output']

    avg_result = avg_op(add_output, numbers)
    avg_output = avg_result.outputs['Output']

    save_metrics_op(add_output, avg_output)
    save_artifact_op(add_output, avg_output)

if __name__ == '__main__':
    kfp.compiler.Compiler().compile(hello_world, 'add_and_avg.zip')

Zipped version: add_and_avg.zip

[kbegiedza]

Can be reproduced with mnist sample: https://github.com/canonical/bundle-kubeflow/blob/master/tests/pipelines/mnist.py

Metrics are saved correctly

[kbegiedza]

@knkski Are you able to reproduce this issue?

[DomFleischmann]

This should be fixed with the latest release of Charmed Kubeflow. Please repopen if this issue still occurs.