search

sky-uk / kfp-operator

https://sky-uk.github.io/kfp-operator
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

KFP Operator Support for TFX 1.15 #382

Closed grahamia closed 3 weeks ago

grahamia commented 3 weeks ago

Describe the bug On trying to use tfx >= 1.15 compile pipeline workflow is fine and passes but on creating a run or runconfiguration against the pipeline that workflow fails with 

message: invalid character '#' looking for beginning of value

This seems to be when the pipeline is loaded from pipeline storage bucket.

To Reproduce Us TFX 1.15.1 and deploy pipeline and runconfiguration resource

Expected behavior Should both deploy successfully

Version and configuration If applicable, add configuration details to help explain your problem.

Logs Workflow has (not much else):

message: invalid character '#' looking for beginning of value
grahamia commented 3 weeks ago

I can see on looking at the pipeline file produced the contents produced in TFX 1.14 is json but in 1.15 (change here it changes to yaml. The code in the VAI provider in enrichJobWithSpecFromTemplateUri does err = json.Unmarshal(buf.Bytes(), &raw) which fails as the file contents is now yaml.

I think it is yaml because we say to produce output file resource.yaml here

It may just be a case we need to change the resource.yaml to be resource.json and it will work