User Images: How/should we support users submitting custom images?

[ca-scribner]

@Colette-G issue #39 will build examples of workarounds we can show to users to decide whether we need to prioritize this issue.

[justbert]

Just putting some information that may be interesting for a more container-centric solution: Harbor is a k8s native repository that also incorporates pieces like Notary (image signing - used in Docker Content Trust) and Clair (Vulnerability scanning).

[ca-scribner]

Rescoped issue to be specifically about submitting user images.

Argument for user images:

• Kubeflow Pipelines are structured around running self-contained docker containers as pipeline steps. To deliver full KFP flexibility to users, they need to be able to build their own custom containers
• For users to experiment/iterate on their pipelines, they need a way to update their custom containers easily (ideally without human blessing)

Argument against:

• security risk

Possible solutions:

• Harbor as mentioned by @justbert ?
• daaas-containers model, which lets users PR specs for images which are then human checked and accepted

[ca-scribner]

My preference is against the human-checked model. I think it:

• bakes in maintenance costs for us (we provide the humans)
• is error prone (our humans need to catch the vulnerabilities)
• is slow for the user when iterating

[JosephKuchar]

Is there any progress on this issue? I was working on developing a new pipeline for our processing, and after creating the Python scripts and Dockerfile I was a little surprised to discover I can't actually build the image within the AAW. If Pipelines are a core functionality of the environment, then I think it's important that users can create new pipelines (which necessitates creating containers) without too much headache. Thanks, Joseph

[blairdrummond]

Hey @JosephKuchar ,

It's admittedly not a great solution, but the current approach is to push images through here:

https://github.com/StatCan/daaas-containers/

We do want to add support for building images, for instance using kaniko or podman, but managing the security around that is a surprisingly tough problem...

[JosephKuchar]

Thanks @blairdrummond !

For what it's worth, I'll add that for building I can move my work to my personal Digital Ocean server, so it's not too much of an inconvenience for me right now, but more generally any functionality that's removed from things like the AAW forces users to use their own computers or external solutions, which is really not ideal. I'm also thinking forward to when I present my work on the AAW to the rest of my team - if on the one hand I can demonstrate cool pipeline functionality, but on the other hand have to say "to build this you need access to a separate linux environment," then I think that will turn off a lot of people. Not to diminish the security issue, which I'm sure is a tough one, but just considerations from a user's perspective.

[blairdrummond]

Well, as I said, there is a way to build and push images into the AAW for users; it's via

https://github.com/StatCan/daaas-containers/

I.e. it goes through a scanning process to make sure it doesn't introduce security-vulnerabilities. Anyone can use github in order to do these image builds for free, so it doesn't require anyone to acquire a separate environment. We're definitely looking to make this smoother, but at the moment ProB and such is a pretty high priority for us, and so it might be a bit of time before we can invest in the UX of that.

If you want to push your images into AAW, for now daaas-containers is the way to go.

[ca-scribner]

I think Joseph is already taking advantage of this stuff. What he’s looking for is something more integrated than what we have, that lets him iteratively design.

On Mon, Dec 14, 2020 at 12:43 Blair Drummond notifications@github.com wrote:

Well, as I said, there is a way to build and push images into the AAW for users; it's via

https://github.com/StatCan/daaas-containers/

I.e. it goes through a scanning process to make sure it doesn't introduce security-vulnerabilities. Anyone can use github in order to do these image builds for free, so it doesn't require anyone to acquire a separate environment. We're definitely looking to make this smoother, but at the moment ProB and such is a pretty high priority for us, and so it might be a bit of time before we can invest in the UX of that.

If you want to push your images into AAW, for now daaas-containers is the way to go.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/StatCan/daaas/issues/6#issuecomment-744600569, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALPFPI6ESWLC4CEJKTRP4GTSUZFFDANCNFSM4MM25R7A .