Deploy a TFX pipeline from the command line

[ianhellstrom]

Is it possible to deploy a TFX pipeline from the command line? The examples in notebooks all show the notebook has to run inside the Kubernetes cluster that the pipelines are supposed to run.

It would be helpful to not have to go through the UI for deploying pipelines (and experiments/runs).

[swiftdiaries]

You can do this now from the Pipelines CLI Look here for instructions to install the CLI

[ianhellstrom]

That does not seem to work on my system:

go get -u github.com/kubeflow/pipelines/backend/src/cmd/ml
go: finding github.com/kubeflow/pipelines/backend/src/cmd/ml latest
go: finding github.com/kubeflow/pipelines/backend/src/cmd latest
go: finding github.com/kubeflow/pipelines/backend/src latest
go: finding github.com/kubeflow/pipelines/backend latest
go: finding github.com/kubeflow/pipelines latest
go: finding github.com/modern-go/concurrent latest
go: finding github.com/gopherjs/gopherjs latest
go: finding golang.org/x/time latest
go: finding github.com/VividCortex/mysqlerr latest
go: finding github.com/globalsign/mgo latest
go: finding github.com/smartystreets/assertions latest
go: finding k8s.io/apimachinery latest
go: finding github.com/docker/spdystream latest
go: finding golang.org/x/net latest
go: finding github.com/lann/builder latest
go: finding k8s.io/apiextensions-apiserver latest
go: finding github.com/ugorji/go/codec latest
go: finding golang.org/x/sys latest
go: finding golang.org/x/sync latest
go: finding golang.org/x/crypto latest
go: finding github.com/golang/groupcache latest
go: finding github.com/erikstmartin/go-testdb latest
go: finding gopkg.in/tomb.v1 latest
go: finding github.com/smartystreets/goconvey latest
go: finding github.com/elazarl/goproxy latest
go: finding github.com/robfig/cron latest
go: finding github.com/lann/ps latest
go: finding github.com/asaskevich/govalidator latest
go: finding github.com/prometheus/procfs latest
go: finding github.com/mxk/go-flowrate latest
go: finding google.golang.org/genproto latest
go: finding github.com/jinzhu/inflection latest
go: finding github.com/mailru/easyjson latest
go: finding github.com/armon/consul-api latest
go: finding github.com/denisenkom/go-mssqldb latest
go: finding github.com/mattbaird/jsonpatch latest
go: finding k8s.io/apiserver latest
go: finding github.com/gregjones/httpcache latest
go: finding github.com/golang/glog latest
go: finding golang.org/x/exp latest
go: finding golang.org/x/lint latest
go: finding golang.org/x/tools latest
go: finding github.com/alecthomas/template latest
go: finding github.com/golang/lint latest
go: finding k8s.io/kube-openapi latest
go: finding golang.org/x/arch latest
go: finding github.com/beorn7/perks latest
go: finding golang.org/x/image latest
go: finding github.com/google/pprof latest
go: finding honnef.co/go/tools latest
go: finding github.com/kr/logfmt latest
go: finding golang.org/x/mobile latest
go: finding k8s.io/api latest
go: finding github.com/mwitkow/go-conntrack latest
go: finding github.com/pquerna/ffjson latest
go: finding golang.org/x/oauth2 latest
go: finding github.com/pquerna/cachecontrol latest
go: finding gopkg.in/check.v1 latest
go: finding golang.org/x/build latest
go: finding github.com/coreos/go-systemd latest
go: finding github.com/tmc/grpc-websocket-proxy latest
go: finding k8s.io/utils latest
go: finding github.com/Azure/go-ansiterm latest
go: finding github.com/BurntSushi/xgb latest
go: finding github.com/gliderlabs/ssh v0.1.3
go: finding github.com/rcrowley/go-metrics latest
go: github.com/golang/lint@v0.0.0-20190409202823-959b441ac422: parsing go.mod: unexpected module path "golang.org/x/lint"
go: finding k8s.io/component-base latest
go: finding github.com/coreos/pkg latest
go: finding github.com/prometheus/client_model latest
go get: error loading module requirements

After that I try the ml command but it's not in my $GOPATH because the installation failed. Any ideas on how to fix that. I'm running go1.12.3 darwin/amd64 on MacOS.

[ianhellstrom]

I tried it without setting GO111MODULE=on and it worked.

[ianhellstrom]

However, when trying to create a pipeline from a local tar.gz file it fails with an internal server error.

I tried: ml pipeline create -n kubeflow --name 'Whatever' --url $(pwd)/pipeline.tar.gz --debug

The response:

{"error":"InternalServerError: Failed to download the pipeline from /Users/ian/tfx/pipeline.tar.gz Please double check the URL is valid and can be accessed by the pipeline system.: Get /Users/ian/tfx/pipeline.tar.gz: unsupported protocol scheme \"\"","message":"InternalServerError: Failed to download the pipeline from /Users/ian/tfx/pipeline.tar.gz Please double check the URL is valid and can be accessed by the pipeline system.: Get /Users/ian/tfx/pipeline.tar.gz: unsupported protocol scheme \"\"","code":13,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Internal Server Error","error_details":"InternalServerError: Failed to download the pipeline from /Users/ian/tfx/pipeline.tar.gz Please double check the URL is valid and can be accessed by the pipeline system.: Get /Users/ian/tfx/pipeline.tar.gz: unsupported protocol scheme \"\""}]}
InternalServerError: Failed to download the pipeline from /Users/ian/tfx/pipeline.tar.gz Please double check the URL is valid and can be accessed by the pipeline system.: Get /Users/ian/tfx/pipeline.tar.gz: unsupported protocol scheme "" (code: 13)

[ianhellstrom]

Sorry, for the spam: I should have checked the upload command. That works.

[swiftdiaries]

Awesome ! Glad it works :)

[hamedhsn]

@swiftdiaries the link is broken can you point me to the right url. thanks!