search

sky-uk / kfp-operator

https://sky-uk.github.io/kfp-operator
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

KFP Provider does not populate artifacts in run completion events #318

Open jmendesky opened 9 months ago

jmendesky commented 9 months ago

Describe the bug

Run completion events emitted by the KFP provider only contain ServingModelArtifacts, not Artifacts. This means that dependent run configurations do not work using this provider.

grahamia commented 9 months ago

On starting to develop this it became apparent that a functional regression has been made on upgrading to TFX > 1.2.0 which stops us from implementing a fix.

The "name" custom property which is relied upon to do the filtering in the metadata store in getArtifacts is now not set.

When executing the following gRPC calls:

grpcurl -d '{"type_name": "pipeline_run", "context_name": "kfp-operator-quickstart-kfp-k49nf"}' -proto ml_metadata/proto/metadata_store_service.proto -plaintext 127.0.0.1:8085 ml_metadata.MetadataStoreService.GetContextByTypeAndName
> returns the content_id to use in following call

grpcurl -d '{"context_id":"143456"}' -proto ml_metadata/proto/metadata_store_service.proto -plaintext 127.0.0.1:8085 ml_metadata.MetadataStoreService.GetArtifactsByContext

In older 1.2.0 TFX working version:

    {
      "id": "83917",
      "typeId": "33",
      "uri": ".../kfp-quickstart/Pusher/pushed_model/60170",
      "customProperties": {
        "name": {
          "stringValue": "kfp-quickstart:kfp-quickstart-kfp-877lkshjj4-6168-502898645:Pusher:pushed_model:0"
        },
        "pushed": {
          "intValue": "1"
        },
        "pushed_destination": {
          "stringValue": ".../kfp-quickstart/serving/1709567369"
        },
        "pushed_version": {
          "stringValue": "1709567369"
        },
        "tfx_version": {
          "stringValue": "1.2.0"
        }
      },
      "state": "LIVE",
      "createTimeSinceEpoch": "1709567384577",
      "lastUpdateTimeSinceEpoch": "1709567384577"
    }

In newer 1.13.0 no name property is included:

    {
      "id": "83922",
      "typeId": "33",
      "uri": ".../kfp-operator-quickstart-kfp/Pusher/pushed_model/60174",
      "customProperties": {
        "is_external": {
          "intValue": "0"
        },
        "pushed": {
          "intValue": "1"
        },
        "pushed_destination": {
          "stringValue": ".../kfp-operator-quickstart-kfp/serving/1709570854"
        },
        "pushed_version": {
          "stringValue": "1709570854"
        },
        "tfx_version": {
          "stringValue": "1.13.0"
        }
      },
      "state": "LIVE",
      "createTimeSinceEpoch": "1709570871359",
      "lastUpdateTimeSinceEpoch": "1709570871359"
    }