Image volumes and container volumes

[thockin]

This would map closely to Docker's native volumes support, and allow people to build and version pre-baked data as containers. Maybe read-only? Haven't thought that far...

[brendandburns]

I guess so? But why not just use a git repo?

On Thu, Aug 7, 2014 at 10:11 PM, Tim Hockin notifications@github.com wrote:

This would map closely to Docker's native volumes support, and allow people to build and version pre-baked data as containers. Maybe read-only? Haven't thought that far...

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831.

[thockin]

More plugins more better? I wanted to put it out there, since we do sort of diverge from Docker's native volumes support. Clearly not urgent :)

On Thu, Aug 7, 2014 at 10:18 PM, brendandburns notifications@github.com wrote:

I guess so? But why not just use a git repo?

On Thu, Aug 7, 2014 at 10:11 PM, Tim Hockin notifications@github.com wrote:

This would map closely to Docker's native volumes support, and allow people to build and version pre-baked data as containers. Maybe read-only? Haven't thought that far...

Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831.

Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-51564381 .

[bgrant0607]

More potential uses of this:

• Deployment of scripts/programs for lifecycle hooks: One of the main points of the lifecycle hooks (#140) is to decouple applications from the execution environment (Kubernetes in this case). If the hook scripts/programs must be deployed as part of the application container, that compromises this objective.
• Dynamic package composition more generally: This would be more similar to our internal package model, where we can independently manage the base filesystem, language runtime, application, utility programs for debugging, etc.
• Configuration deployment
• Input data deployment

A git repo could be used for some of these cases, but for others it would be less than ideal.

[erictune]

If the base image of a docker file (e.g. FROM fedora) is a linux distro, then isn't it going to be annoying to have a bunch of Linux Standard Base type of files in what is really supposed to be a data-only packge?

On the other hand, if it is creating using tar -c . | docker import - myimage, then what is the advantage is of a docker image over a tar file?

On Tue, Sep 30, 2014 at 9:00 PM, bgrant0607 notifications@github.com wrote:

More potential uses of this:

• Deployment of scripts/programs for lifecycle hooks: One of the main points of the lifecycle hooks (#140 https://github.com/GoogleCloudPlatform/kubernetes/issues/140) is to decouple applications from the execution environment (Kubernetes in this case). If the hook scripts/programs must be deployed as part of the application container, that compromises this objective.
• Dynamic package composition more generally: This would be more similar to our internal package model, where we can independently manage the base filesystem, language runtime, application, utility programs for debugging, etc.
• Configuration deployment
• Input data deployment

A git repo could be used for some of these cases, but for others it would be less than ideal.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-57416301 .

[thockin]

How about the VOLUMES directives of a container? Any container can declare itself to be exposing any number of volumes. Maybe the functionality to expose is not the whole container, but just the volumes from that container?

On the other hand, people can create data containers "FROM scratch", and who are we to say it's annoying?

On Wed, Oct 1, 2014 at 7:20 AM, erictune notifications@github.com wrote:

If the base image of a docker file (e.g. FROM fedora) is a linux distro, then isn't it going to be annoying to have a bunch of Linux Standard Base type of files in what is really supposed to be a data-only packge?

On the other hand, if it is creating using tar -c . | docker import - myimage, then what is the advantage is of a docker image over a tar file?

On Tue, Sep 30, 2014 at 9:00 PM, bgrant0607 notifications@github.com wrote:

More potential uses of this:

• Deployment of scripts/programs for lifecycle hooks: One of the main points of the lifecycle hooks (#140 https://github.com/GoogleCloudPlatform/kubernetes/issues/140) is to decouple applications from the execution environment (Kubernetes in this case). If the hook scripts/programs must be deployed as part of the application container, that compromises this objective.
• Dynamic package composition more generally: This would be more similar to our internal package model, where we can independently manage the base filesystem, language runtime, application, utility programs for debugging, etc.
• Configuration deployment
• Input data deployment

A git repo could be used for some of these cases, but for others it would be less than ideal.

Reply to this email directly or view it on GitHub < https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-57416301>

.

Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-57471069 .

[dchen1107]

@thockin I like the initial idea of having new volume type to support docker's data volume container, which matches common package, shared package etc. concept internally. I also can see the potential use cases listed by @bgrant0607. But please don't go down to the road like Docker had today: declare a container as data volume, in which case, we are going to introduce another level complexity of dependencies to containers within a pod, or even dependencies between pods if a pod only has data volume container, etc. I think your initial idea of having a volume type which actually refers as a docker volume container or other read-only volume is a better approach for a long run.

[thockin]

The interesting thing about docker volumes is that a container does not have to be RUNNING for the volumes to exist. It's a weird model, but I think it could work.

I don't think we know what people really want in this space yet, though.

On Wed, Oct 1, 2014 at 10:57 AM, Dawn Chen notifications@github.com wrote:

@thockin https://github.com/thockin I like the initial idea of having new volume type to support docker's data volume container, which matches common package, shared package etc. concept internally. I also can see the potential use cases listed by @bgrant0607 https://github.com/bgrant0607. But please don't go down to the road like Docker had today: declare a container as data volume, in which case, we are going to introduce another level complexity of dependencies to containers within a pod, or even dependencies between pods if a pod only has data volume container, etc. I think your initial idea of having a volume type which actually refers as a docker volume container or other read-only volume is a better approach for a long run.

Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-57507672 .

[erictune]

It appears the same net effect as this issue can be achieved without introducing a new volume type using two containers in a pod and some shell wrapped around the underling container command lines (see https://github.com/GoogleCloudPlatform/kubernetes/issues/1589#issuecomment-58043463).

How to decide on container-as-a-volume vs. command-line-based sequencing?

• more portability between Kubernetes and non-Kubernetes docker use cases with container-as-volume.
• easier for user to discover container-as-volume concept and identify that it is the right solution?
• having fewer and less general mechanisms for setting up "packages" may lend itself to more tightly integrated build/deploy systems. But, maybe that is not a goal for Kubernetes.
• either solution can integrate with data durability, I think.
• liveness checking is more complex with command-line-based sequencing, since the pod to be liveness checked goes through a waiting phase and then a running phase.

[stp-ip]

There are a lot of ways to go about it and a new volume type is in my opinion not needed. I tried to standardize the way we structure data and make different volume providers possible. These range from Host Volumes, Data Volume Containers, Side Containers with additional logic to Volume as a Service, which is where k8s could integrate greatly. The start is already available via git as volume. I think the native Volumes in Docker are enough, but just lack a standard. The more detailed ideas are available at docker/docker#9277.

[bgrant0607]

I think the question is whether the data volume container should be represented as a container or as a volume. I prefer to think of them as volumes and find passive containers to be non-intuitive for users, problematic for management systems to deal with, and the source of weird corner-case behaviors.

[stp-ip]

@bgrant0607 still they are supported in docker and therefore we should acknowledge they exist. I would love to see more integrated methods in k8s itself, which just expose a specific type of volume. I was hinting at that in my proposal via the VaaS approach. But I would dislike this approach reducing compatibility.

[mindscratch]

+1 for supporting a container as a volume. I have a scenario where I have a container that has a bunch of data baked into it for use by other containers, helps keep the data "local" to the work being done.

[rehevkor5]

Whatever you decide, I hope you will make it clear in the documentation to save people time of searching around to find this information. Currently, the documentation for both compute and container engine:

1. makes no mention that VOLUME/--volumes-from/container-as-volume is not supported
2. makes no mention of possible work-arounds
3. makes no mention of the possibility of retrieving things from a git repo

It's important to note that using a git repo isn't the same. It requires the git repo to be securely accessible from Google Cloud (or wherever Kubernetes is being used). Further, it's unclear how non-public repositories would be accessible, unless the user & password is hard-coded into the Kubernetes GitRepo#repository JSON/YAML string. Also, it requires that the desired artifact(s) be checked in to source control. And it decouples the Docker image from the artifact (which may or may not be desirable).

I will be working around this issue by moving the data that's in my container volume into a Dockerfile that layers on top of the container that wanted to use the volume, with ADD. The problem you're running into is that the community at large is encouraging the "container as volume" approach in websites and blog posts, and as a result people will continue to have difficulty. For example, the docker website itself says, "If you have some persistent data that you want to share between containers, or want to use from non-persistent containers, it's best to create a named Data Volume Container, and then to mount the data from it." (emphasis mine).

Also, @erictune a container-only volume can be (and probably should be) written as "FROM scratch". I'd argue that if the user doesn't do it that way, that's their choice.

[AshleyAitken]

+1 @rehevkor5

I am disappointed to hear that k8s doesn't support data volume containers.

I am not sure how I am supposed to abstract r/w data away from the host now. I was under the impression that k8s was about abstracting away the host, but a host volume introduces all sorts of host-specfics, like having to share the same username/group for access to the data.

@rehevkor5 I thought the same thing about data volume containers at first (should be written as FROM scratch) until I read this (which may or may not be correct): http://container42.com/2014/11/18/data-only-container-madness/ Your workaround seems to do just this?

[thockin]

There are a few things going on here, most importantly (I think) some confusion.

Kubernetes supports the notion of a writable "empty volume" that is shared across containers within a pod, without resorting to host directories - this is what an emptyDir volume is.

Now the question comes down to "but I don't want an empty directory, I want ". The question I think we need to sort out is what the is. We currently support pull-from-git, which is just a single instance of a larger pattern - "go fetch some pre-cooked data once before starting my pod".

We could support myriad such "fetch once" volume kinds, git, cvs, svn, docker containers(more below), URLs, in-line base64-encoded tarfiles, stdout of another program, etc. I do NOT think we want to support those all as independent volume plugins - they can almost all be done by an unprivileged container without any help from kubelet. More, you quickly arrive at the followup features like "...and re-pull from git every 10 minutes" - things that stop being "fetch once" and start being active management, but do not require privileges. We make great use of such things internally.

IMO: all of these things that can be run as a side-car container in your pod (writing to a shared emptyDir) SHOULD BE. Git should stop being a first-class volume, and should instead be a container of some sort. This brings a slew of new design questions: Is it just a container like all the other app containers? How do I ensure it runs BEFORE my app containers? What if it experiences a failure? I don't have answers to all of these yet.

Now, let's think about the case of docker data volume containers. What is the semantic that people are really asking for? Is it:

• "run" a data container and then expose the entire chroot of that run?
• "run" a data container and then expose all of the VOLUME (from Dockerfile) dirs as kubernetes volumes?
• "run" a data container and then do the equivalent of --volumes-from into kubernetes containers?

These are all subtly different semantically, especially in the face of a data container that has multiple VOLUME statements. Some operating modes also make it hard to verify input until after a pod has been accepted, scheduled, and attempted on a kubelet (we try to validate as much as we can up front).

ACTION ITEM: I'd very much like for people who use docker data containers to describe what behavior they would expect here.

Back to side-car containers as volumes. I could imagine something like:

Pod { spec: { volumes: [ { Name: "git-data", Source: FromContainer { Name: "awkward", // what goes here? Image: "kubernetes/git-volume" EnvVars: [ { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] } } ], containers: [ ... use git-data ...] }

Sadly, there's not much of the container-ness that you can really hide, so you end up re-using the Container schema, which is at least somewhat redundant and awkward.

Alternately, I could imagine just making them extra containers:

Pod { spec: { volumes: [ { Name: "git-data", Source: EmptyDir {} } ], containers: [ { Name: "git-puller", Image: "kubernetes/git-volume" EnvVars: [ { DATADIR: "/vol" }, { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] VolumeMounts: [ { Name: "git-data", MountPath: "/vol" } ] }, { ... container that reads git-data ... } ] } }

It's still pretty verbose. Can we do better?

In this sort of model, almost any sort of logic can be bundled up as a container and published, and anyone can use it immediately.

Something like this is, I think, the way to go. Details to emerge, input wanted.

On Sat, Jan 10, 2015 at 5:42 PM, Ashley Aitken notifications@github.com wrote:

+1 @rehevkor5 https://github.com/rehevkor5

I am disappointed to hear that k8s doesn't support data volume containers.

I am not sure how I am supposed to abstract r/w data away from the host now. I was under the impression that k8s was about abstracting away the host, but a host volume introduces all sorts of host-specfics, like having to share the same username/group for access to the data.

@rehevkor5 https://github.com/rehevkor5 I thought the same thing about data volume containers at first (should be written as FROM scratch) until I read this (which may or may not be correct): http://container42.com/2014/11/18/data-only-container-madness/

Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69480057 .

[thockin]

To follow up to myself - all of this assumes that any data mutations you make have a lifetime equivalent to the pod. If the pod dies for any reason (the machine goes down, it gets deleted in the API, some non-recoverable failure in kubelet, etc) the data dies with it.

Durable data is a MUCH larger topic :)

On Sat, Jan 10, 2015 at 9:12 PM, Tim Hockin thockin@google.com wrote:

There are a few things going on here, most importantly (I think) some confusion.

Kubernetes supports the notion of a writable "empty volume" that is shared across containers within a pod, without resorting to host directories - this is what an emptyDir volume is.

Now the question comes down to "but I don't want an empty directory, I want ". The question I think we need to sort out is what the is. We currently support pull-from-git, which is just a single instance of a larger pattern - "go fetch some pre-cooked data once before starting my pod".

We could support myriad such "fetch once" volume kinds, git, cvs, svn, docker containers(more below), URLs, in-line base64-encoded tarfiles, stdout of another program, etc. I do NOT think we want to support those all as independent volume plugins - they can almost all be done by an unprivileged container without any help from kubelet. More, you quickly arrive at the followup features like "...and re-pull from git every 10 minutes" - things that stop being "fetch once" and start being active management, but do not require privileges. We make great use of such things internally.

IMO: all of these things that can be run as a side-car container in your pod (writing to a shared emptyDir) SHOULD BE. Git should stop being a first-class volume, and should instead be a container of some sort. This brings a slew of new design questions: Is it just a container like all the other app containers? How do I ensure it runs BEFORE my app containers? What if it experiences a failure? I don't have answers to all of these yet.

Now, let's think about the case of docker data volume containers. What is the semantic that people are really asking for? Is it:

• "run" a data container and then expose the entire chroot of that run?
• "run" a data container and then expose all of the VOLUME (from Dockerfile) dirs as kubernetes volumes?
• "run" a data container and then do the equivalent of --volumes-from into kubernetes containers?

These are all subtly different semantically, especially in the face of a data container that has multiple VOLUME statements. Some operating modes also make it hard to verify input until after a pod has been accepted, scheduled, and attempted on a kubelet (we try to validate as much as we can up front).

ACTION ITEM: I'd very much like for people who use docker data containers to describe what behavior they would expect here.

Back to side-car containers as volumes. I could imagine something like:

Pod { spec: { volumes: [ { Name: "git-data", Source: FromContainer { Name: "awkward", // what goes here? Image: "kubernetes/git-volume" EnvVars: [ { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] } } ], containers: [ ... use git-data ...] }

Sadly, there's not much of the container-ness that you can really hide, so you end up re-using the Container schema, which is at least somewhat redundant and awkward.

Alternately, I could imagine just making them extra containers:

Pod { spec: { volumes: [ { Name: "git-data", Source: EmptyDir {} } ], containers: [ { Name: "git-puller", Image: "kubernetes/git-volume" EnvVars: [ { DATADIR: "/vol" }, { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] VolumeMounts: [ { Name: "git-data", MountPath: "/vol" } ] }, { ... container that reads git-data ... } ] } }

It's still pretty verbose. Can we do better?

In this sort of model, almost any sort of logic can be bundled up as a container and published, and anyone can use it immediately.

Something like this is, I think, the way to go. Details to emerge, input wanted.

On Sat, Jan 10, 2015 at 5:42 PM, Ashley Aitken notifications@github.com wrote:

+1 @rehevkor5 https://github.com/rehevkor5

I am disappointed to hear that k8s doesn't support data volume containers.

I am not sure how I am supposed to abstract r/w data away from the host now. I was under the impression that k8s was about abstracting away the host, but a host volume introduces all sorts of host-specfics, like having to share the same username/group for access to the data.

@rehevkor5 https://github.com/rehevkor5 I thought the same thing about data volume containers at first (should be written as FROM scratch) until I read this (which may or may not be correct): http://container42.com/2014/11/18/data-only-container-madness/

Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69480057 .

[mindscratch]

I'll try to describe a use case for a data container. I'll have a pod with 3 containers I'll name them "ingest", "process" and "data".

The ingest container is responsible for getting messages in some fashion and telling the "process" container to do work.

The process container does work, but it requires access to data provided by the "data" container, outside of kubernetes this is done using dockers "volumes-from". This "data" can be 100's of megabytes, but most often is 10-15 gigabytes.

The data container has a process responsible for pulling the data that will be needed by the process container. While the process container is doing work, it's possible that a new set of data becomes available. The data container can fetch the data and use something like symlinks to swap it so the next time the process container begins a new process it's using the newly available data.

Hopefully that makes some sense.

Thanks

On Sun, Jan 11, 2015 at 12:18 AM, Tim Hockin notifications@github.com wrote:

To follow up to myself - all of this assumes that any data mutations you make have a lifetime equivalent to the pod. If the pod dies for any reason (the machine goes down, it gets deleted in the API, some non-recoverable failure in kubelet, etc) the data dies with it.

Durable data is a MUCH larger topic :)

On Sat, Jan 10, 2015 at 9:12 PM, Tim Hockin thockin@google.com wrote:

There are a few things going on here, most importantly (I think) some confusion.

Kubernetes supports the notion of a writable "empty volume" that is shared across containers within a pod, without resorting to host directories - this is what an emptyDir volume is.

Now the question comes down to "but I don't want an empty directory, I want ". The question I think we need to sort out is what the is. We currently support pull-from-git, which is just a single instance of a larger pattern - "go fetch some pre-cooked data once before starting my pod".

We could support myriad such "fetch once" volume kinds, git, cvs, svn, docker containers(more below), URLs, in-line base64-encoded tarfiles, stdout of another program, etc. I do NOT think we want to support those all as independent volume plugins - they can almost all be done by an unprivileged container without any help from kubelet. More, you quickly arrive at the followup features like "...and re-pull from git every 10 minutes" - things that stop being "fetch once" and start being active management, but do not require privileges. We make great use of such things internally.

IMO: all of these things that can be run as a side-car container in your pod (writing to a shared emptyDir) SHOULD BE. Git should stop being a first-class volume, and should instead be a container of some sort. This brings a slew of new design questions: Is it just a container like all the other app containers? How do I ensure it runs BEFORE my app containers? What if it experiences a failure? I don't have answers to all of these yet.

Now, let's think about the case of docker data volume containers. What is the semantic that people are really asking for? Is it:

• "run" a data container and then expose the entire chroot of that run?
• "run" a data container and then expose all of the VOLUME (from Dockerfile) dirs as kubernetes volumes?
• "run" a data container and then do the equivalent of --volumes-from into kubernetes containers?

These are all subtly different semantically, especially in the face of a data container that has multiple VOLUME statements. Some operating modes also make it hard to verify input until after a pod has been accepted, scheduled, and attempted on a kubelet (we try to validate as much as we can up front).

ACTION ITEM: I'd very much like for people who use docker data containers to describe what behavior they would expect here.

Back to side-car containers as volumes. I could imagine something like:

Pod { spec: { volumes: [ { Name: "git-data", Source: FromContainer { Name: "awkward", // what goes here? Image: "kubernetes/git-volume" EnvVars: [ { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] } } ], containers: [ ... use git-data ...] }

Sadly, there's not much of the container-ness that you can really hide, so you end up re-using the Container schema, which is at least somewhat redundant and awkward.

Alternately, I could imagine just making them extra containers:

Pod { spec: { volumes: [ { Name: "git-data", Source: EmptyDir {} } ], containers: [ { Name: "git-puller", Image: "kubernetes/git-volume" EnvVars: [ { DATADIR: "/vol" }, { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] VolumeMounts: [ { Name: "git-data", MountPath: "/vol" } ] }, { ... container that reads git-data ... } ] } }

It's still pretty verbose. Can we do better?

In this sort of model, almost any sort of logic can be bundled up as a container and published, and anyone can use it immediately.

Something like this is, I think, the way to go. Details to emerge, input wanted.

On Sat, Jan 10, 2015 at 5:42 PM, Ashley Aitken <notifications@github.com

wrote:

+1 @rehevkor5 https://github.com/rehevkor5

I am disappointed to hear that k8s doesn't support data volume containers.

I am not sure how I am supposed to abstract r/w data away from the host now. I was under the impression that k8s was about abstracting away the host, but a host volume introduces all sorts of host-specfics, like having to share the same username/group for access to the data.

@rehevkor5 https://github.com/rehevkor5 I thought the same thing about data volume containers at first (should be written as FROM scratch) until I read this (which may or may not be correct): http://container42.com/2014/11/18/data-only-container-madness/

Reply to this email directly or view it on GitHub < https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69480057

.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69484090 .

https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links

[thockin]

To me, this does not make a strong argument. Everything you describe is possible if your data container just writes to a shared emptyDir volume.

Now, there IS a gotcha with the initial load of data, but that has to be handled in any similar model. Either the data is immutable, in which case the data container can load it once and go to sleep, or else it is changing over time, time, in which case you have to wait for it to get current. In the former case, the initial data is ALL that matters. Is that an interesting use? In the latter, does the initial data matter, or only "current" data?

The only other argument is to be exactly docker compatible semantically, but frankly the volumes-from behavior is so semantically rigid, it may not be worth being compatible with. On Jan 11, 2015 9:26 AM, "Craig Wickesser" notifications@github.com wrote:

I'll try to describe a use case for a data container. I'll have a pod with 3 containers I'll name them "ingest", "process" and "data".

The ingest container is responsible for getting messages in some fashion and telling the "process" container to do work.

The process container does work, but it requires access to data provided by the "data" container, outside of kubernetes this is done using dockers "volumes-from". This "data" can be 100's of megabytes, but most often is 10-15 gigabytes.

The data container has a process responsible for pulling the data that will be needed by the process container. While the process container is doing work, it's possible that a new set of data becomes available. The data container can fetch the data and use something like symlinks to swap it so the next time the process container begins a new process it's using the newly available data.

Hopefully that makes some sense.

Thanks

On Sun, Jan 11, 2015 at 12:18 AM, Tim Hockin notifications@github.com wrote:

To follow up to myself - all of this assumes that any data mutations you make have a lifetime equivalent to the pod. If the pod dies for any reason (the machine goes down, it gets deleted in the API, some non-recoverable failure in kubelet, etc) the data dies with it.

Durable data is a MUCH larger topic :)

On Sat, Jan 10, 2015 at 9:12 PM, Tim Hockin thockin@google.com wrote:

There are a few things going on here, most importantly (I think) some confusion.

Kubernetes supports the notion of a writable "empty volume" that is shared

across containers within a pod, without resorting to host directories

this is what an emptyDir volume is.

Now the question comes down to "but I don't want an empty directory, I want ". The question I think we need to sort out is what the is. We currently support pull-from-git, which is just a single instance of a larger pattern - "go fetch some pre-cooked data once before starting my pod".

We could support myriad such "fetch once" volume kinds, git, cvs, svn, docker containers(more below), URLs, in-line base64-encoded tarfiles, stdout of another program, etc. I do NOT think we want to support those all as independent volume plugins - they can almost all be done by an unprivileged container without any help from kubelet. More, you quickly arrive at the followup features like "...and re-pull from git every 10 minutes" - things that stop being "fetch once" and start being active management, but do not require privileges. We make great use of such things internally.

IMO: all of these things that can be run as a side-car container in your pod (writing to a shared emptyDir) SHOULD BE. Git should stop being a first-class volume, and should instead be a container of some sort. This brings a slew of new design questions: Is it just a container like all the other app containers? How do I ensure it runs BEFORE my app containers? What if it experiences a failure? I don't have answers to all of these yet.

Now, let's think about the case of docker data volume containers. What is the semantic that people are really asking for? Is it:

• "run" a data container and then expose the entire chroot of that run?
• "run" a data container and then expose all of the VOLUME (from Dockerfile) dirs as kubernetes volumes?
• "run" a data container and then do the equivalent of --volumes-from into kubernetes containers?

These are all subtly different semantically, especially in the face of a data container that has multiple VOLUME statements. Some operating modes also make it hard to verify input until after a pod has been accepted, scheduled, and attempted on a kubelet (we try to validate as much as we can up front).

ACTION ITEM: I'd very much like for people who use docker data containers to describe what behavior they would expect here.

Back to side-car containers as volumes. I could imagine something like:

Pod { spec: { volumes: [ { Name: "git-data", Source: FromContainer { Name: "awkward", // what goes here? Image: "kubernetes/git-volume" EnvVars: [ { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] } } ], containers: [ ... use git-data ...] }

Sadly, there's not much of the container-ness that you can really hide, so you end up re-using the Container schema, which is at least somewhat redundant and awkward.

Alternately, I could imagine just making them extra containers:

Pod { spec: { volumes: [ { Name: "git-data", Source: EmptyDir {} } ], containers: [ { Name: "git-puller", Image: "kubernetes/git-volume" EnvVars: [ { DATADIR: "/vol" }, { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] VolumeMounts: [ { Name: "git-data", MountPath: "/vol" } ] }, { ... container that reads git-data ... } ] } }

It's still pretty verbose. Can we do better?

In this sort of model, almost any sort of logic can be bundled up as a container and published, and anyone can use it immediately.

Something like this is, I think, the way to go. Details to emerge, input wanted.

On Sat, Jan 10, 2015 at 5:42 PM, Ashley Aitken < notifications@github.com

wrote:

+1 @rehevkor5 https://github.com/rehevkor5

I am disappointed to hear that k8s doesn't support data volume containers.

I am not sure how I am supposed to abstract r/w data away from the host now. I was under the impression that k8s was about abstracting away the host, but a host volume introduces all sorts of host-specfics, like having to share the same username/group for access to the data.

@rehevkor5 https://github.com/rehevkor5 I thought the same thing about data volume containers at first (should be written as FROM scratch) until I read this (which may or may not be correct): http://container42.com/2014/11/18/data-only-container-madness/

Reply to this email directly or view it on GitHub <

https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69480057

.

Reply to this email directly or view it on GitHub < https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69484090>

.

https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links

Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69502422 .

[mindscratch]

Using an emptydir that is available between containers sounds sufficient. The initial data could actually be baked into the docker image, then the process in the "data" container could make sure it updates it when necessary.

After understanding "emptydir" better, I agree, the use case I provided would work with what Kubernetes supports today.

Thanks.

On Sun, Jan 11, 2015 at 1:36 PM, Tim Hockin notifications@github.com wrote:

To me, this does not make a strong argument. Everything you describe is possible if your data container just writes to a shared emptyDir volume.

Now, there IS a gotcha with the initial load of data, but that has to be handled in any similar model. Either the data is immutable, in which case the data container can load it once and go to sleep, or else it is changing over time, time, in which case you have to wait for it to get current. In the former case, the initial data is ALL that matters. Is that an interesting use? In the latter, does the initial data matter, or only "current" data?

The only other argument is to be exactly docker compatible semantically, but frankly the volumes-from behavior is so semantically rigid, it may not be worth being compatible with. On Jan 11, 2015 9:26 AM, "Craig Wickesser" notifications@github.com wrote:

I'll try to describe a use case for a data container. I'll have a pod with 3 containers I'll name them "ingest", "process" and "data".

The ingest container is responsible for getting messages in some fashion and telling the "process" container to do work.

The process container does work, but it requires access to data provided by the "data" container, outside of kubernetes this is done using dockers "volumes-from". This "data" can be 100's of megabytes, but most often is 10-15 gigabytes.

The data container has a process responsible for pulling the data that will be needed by the process container. While the process container is doing work, it's possible that a new set of data becomes available. The data container can fetch the data and use something like symlinks to swap it so the next time the process container begins a new process it's using the newly available data.

Hopefully that makes some sense.

Thanks

On Sun, Jan 11, 2015 at 12:18 AM, Tim Hockin notifications@github.com wrote:

To follow up to myself - all of this assumes that any data mutations you make have a lifetime equivalent to the pod. If the pod dies for any reason (the machine goes down, it gets deleted in the API, some non-recoverable failure in kubelet, etc) the data dies with it.

Durable data is a MUCH larger topic :)

On Sat, Jan 10, 2015 at 9:12 PM, Tim Hockin thockin@google.com wrote:

There are a few things going on here, most importantly (I think) some confusion.

Kubernetes supports the notion of a writable "empty volume" that is shared

across containers within a pod, without resorting to host directories

this is what an emptyDir volume is.

Now the question comes down to "but I don't want an empty directory, I want ". The question I think we need to sort out is what the is. We currently support pull-from-git, which is just a single instance of a larger pattern - "go fetch some pre-cooked data once before starting my pod".

We could support myriad such "fetch once" volume kinds, git, cvs, svn, docker containers(more below), URLs, in-line base64-encoded tarfiles, stdout of another program, etc. I do NOT think we want to support those all as independent volume plugins - they can almost all be done by an unprivileged container without any help from kubelet. More, you quickly arrive at the followup features like "...and re-pull from git every 10 minutes" - things that stop being "fetch once" and start being active management, but do not require privileges. We make great use of such things internally.

IMO: all of these things that can be run as a side-car container in your pod (writing to a shared emptyDir) SHOULD BE. Git should stop being a first-class volume, and should instead be a container of some sort. This brings a slew of new design questions: Is it just a container like all the other app containers? How do I ensure it runs BEFORE my app containers? What if it experiences a failure? I don't have answers to all of these yet.

Now, let's think about the case of docker data volume containers. What is the semantic that people are really asking for? Is it:

• "run" a data container and then expose the entire chroot of that run?
• "run" a data container and then expose all of the VOLUME (from Dockerfile) dirs as kubernetes volumes?
• "run" a data container and then do the equivalent of --volumes-from into kubernetes containers?

These are all subtly different semantically, especially in the face of a data container that has multiple VOLUME statements. Some operating modes also make it hard to verify input until after a pod has been accepted, scheduled, and attempted on a kubelet (we try to validate as much as we can up front).

ACTION ITEM: I'd very much like for people who use docker data containers to describe what behavior they would expect here.

Back to side-car containers as volumes. I could imagine something like:

Pod { spec: { volumes: [ { Name: "git-data", Source: FromContainer { Name: "awkward", // what goes here? Image: "kubernetes/git-volume" EnvVars: [ { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] } } ], containers: [ ... use git-data ...] }

Sadly, there's not much of the container-ness that you can really hide, so you end up re-using the Container schema, which is at least somewhat redundant and awkward.

Alternately, I could imagine just making them extra containers:

Pod { spec: { volumes: [ { Name: "git-data", Source: EmptyDir {} } ], containers: [ { Name: "git-puller", Image: "kubernetes/git-volume" EnvVars: [ { DATADIR: "/vol" }, { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] VolumeMounts: [ { Name: "git-data", MountPath: "/vol" } ] }, { ... container that reads git-data ... } ] } }

It's still pretty verbose. Can we do better?

In this sort of model, almost any sort of logic can be bundled up as a container and published, and anyone can use it immediately.

Something like this is, I think, the way to go. Details to emerge, input wanted.

On Sat, Jan 10, 2015 at 5:42 PM, Ashley Aitken < notifications@github.com

wrote:

+1 @rehevkor5 https://github.com/rehevkor5

I am disappointed to hear that k8s doesn't support data volume containers.

I am not sure how I am supposed to abstract r/w data away from the host now. I was under the impression that k8s was about abstracting away the host, but a host volume introduces all sorts of host-specfics, like having to share the same username/group for access to the data.

@rehevkor5 https://github.com/rehevkor5 I thought the same thing about data volume containers at first (should be written as FROM scratch) until I read this (which may or may not be correct): http://container42.com/2014/11/18/data-only-container-madness/

Reply to this email directly or view it on GitHub <

https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69480057

.

Reply to this email directly or view it on GitHub <

https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69484090

.

https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links

Reply to this email directly or view it on GitHub < https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69502422

.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69505207 .

https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links

[thockin]

I am not trying to dodge supporting a cool feature. In the case of immutable data, emptyDir would incur a copy and 2x disk usage. That is not acceptable. The question I have is whether such a use case is interesting, and if so, what semantic people would think is "obvious".

Then there is the issue of live-updated data. Does the initial payload matter? What sort of API would capture this idea? Is static data just a special case of this? On Jan 11, 2015 10:43 AM, "Craig Wickesser" notifications@github.com wrote:

Using an emptydir that is available between containers sounds sufficient. The initial data could actually be baked into the docker image, then the process in the "data" container could make sure it updates it when necessary.

After understanding "emptydir" better, I agree, the use case I provided would work with what Kubernetes supports today.

Thanks.

On Sun, Jan 11, 2015 at 1:36 PM, Tim Hockin notifications@github.com wrote:

To me, this does not make a strong argument. Everything you describe is possible if your data container just writes to a shared emptyDir volume.

Now, there IS a gotcha with the initial load of data, but that has to be handled in any similar model. Either the data is immutable, in which case the data container can load it once and go to sleep, or else it is changing over time, time, in which case you have to wait for it to get current. In the former case, the initial data is ALL that matters. Is that an interesting use? In the latter, does the initial data matter, or only "current" data?

The only other argument is to be exactly docker compatible semantically, but frankly the volumes-from behavior is so semantically rigid, it may not be worth being compatible with. On Jan 11, 2015 9:26 AM, "Craig Wickesser" notifications@github.com wrote:

I'll try to describe a use case for a data container. I'll have a pod with 3 containers I'll name them "ingest", "process" and "data".

The ingest container is responsible for getting messages in some fashion and telling the "process" container to do work.

The process container does work, but it requires access to data provided by the "data" container, outside of kubernetes this is done using dockers "volumes-from". This "data" can be 100's of megabytes, but most often is 10-15 gigabytes.

The data container has a process responsible for pulling the data that will be needed by the process container. While the process container is doing work, it's possible that a new set of data becomes available. The data container can fetch the data and use something like symlinks to swap it so the next time the process container begins a new process it's using the newly available data.

Hopefully that makes some sense.

Thanks

On Sun, Jan 11, 2015 at 12:18 AM, Tim Hockin notifications@github.com

wrote:

To follow up to myself - all of this assumes that any data mutations you make have a lifetime equivalent to the pod. If the pod dies for any reason (the machine goes down, it gets deleted in the API, some non-recoverable failure in kubelet, etc) the data dies with it.

Durable data is a MUCH larger topic :)

On Sat, Jan 10, 2015 at 9:12 PM, Tim Hockin thockin@google.com wrote:

There are a few things going on here, most importantly (I think) some confusion.

Kubernetes supports the notion of a writable "empty volume" that is shared across containers within a pod, without resorting to host

directories

this is what an emptyDir volume is.

Now the question comes down to "but I don't want an empty directory, I want ". The question I think we need to sort out is what the is. We currently support pull-from-git, which is just a single instance of a larger pattern - "go fetch some pre-cooked data once before starting my pod".

We could support myriad such "fetch once" volume kinds, git, cvs, svn, docker containers(more below), URLs, in-line base64-encoded tarfiles, stdout of another program, etc. I do NOT think we want to support those all as independent volume plugins - they can almost all be done by an unprivileged container without any help from kubelet. More, you quickly arrive at the followup features like "...and re-pull from git every 10 minutes" - things that stop being "fetch once" and start being active management, but do not require privileges. We make great use of such things internally.

IMO: all of these things that can be run as a side-car container in your pod (writing to a shared emptyDir) SHOULD BE. Git should stop being a first-class volume, and should instead be a container of some sort. This brings a slew of new design questions: Is it just a container like all the other app containers? How do I ensure it runs BEFORE my app containers? What if it experiences a failure? I don't have answers to all of these yet.

Now, let's think about the case of docker data volume containers. What is the semantic that people are really asking for? Is it:

• "run" a data container and then expose the entire chroot of that run?
• "run" a data container and then expose all of the VOLUME (from Dockerfile) dirs as kubernetes volumes?
• "run" a data container and then do the equivalent of --volumes-from into kubernetes containers?

These are all subtly different semantically, especially in the face of a data container that has multiple VOLUME statements. Some operating modes also make it hard to verify input until after a pod has been accepted, scheduled, and attempted on a kubelet (we try to validate as much as we can up front).

ACTION ITEM: I'd very much like for people who use docker data containers to describe what behavior they would expect here.

Back to side-car containers as volumes. I could imagine something like:

Pod { spec: { volumes: [ { Name: "git-data", Source: FromContainer { Name: "awkward", // what goes here? Image: "kubernetes/git-volume" EnvVars: [ { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] } } ], containers: [ ... use git-data ...] }

Sadly, there's not much of the container-ness that you can really hide, so you end up re-using the Container schema, which is at least somewhat redundant and awkward.

Alternately, I could imagine just making them extra containers:

Pod { spec: { volumes: [ { Name: "git-data", Source: EmptyDir {} } ], containers: [ { Name: "git-puller", Image: "kubernetes/git-volume" EnvVars: [ { DATADIR: "/vol" }, { REPO: "http://github.com/yourname/something"}, { RESYNC: "true" }, { RESYNC_INTERVAL: 60 } ] VolumeMounts: [ { Name: "git-data", MountPath: "/vol" } ] }, { ... container that reads git-data ... } ] } }

It's still pretty verbose. Can we do better?

In this sort of model, almost any sort of logic can be bundled up as a container and published, and anyone can use it immediately.

Something like this is, I think, the way to go. Details to emerge, input wanted.

On Sat, Jan 10, 2015 at 5:42 PM, Ashley Aitken < notifications@github.com

wrote:

+1 @rehevkor5 https://github.com/rehevkor5

I am disappointed to hear that k8s doesn't support data volume containers.

I am not sure how I am supposed to abstract r/w data away from the host now. I was under the impression that k8s was about abstracting away the host, but a host volume introduces all sorts of host-specfics, like having to share the same username/group for access to the data.

@rehevkor5 https://github.com/rehevkor5 I thought the same thing about data volume containers at first (should be written as FROM scratch) until I read this (which may or may not be correct): http://container42.com/2014/11/18/data-only-container-madness/

Reply to this email directly or view it on GitHub <

https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69480057

.

Reply to this email directly or view it on GitHub <

https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69484090

.

https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links

Reply to this email directly or view it on GitHub <

https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69502422

.

Reply to this email directly or view it on GitHub < https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69505207>

.

https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links

Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-69505477 .

[bgrant0607]

P2 to at least document this better.

[pwFoo]

Wouldn't docker data only container be persistent? Would be great to use it with kubernetes, because I need persistent data without iscsi, NFS vor host_dir (permission problems). I would use Fedora Atomic as a single minion instance.

[MalteJ]

@pwFoo currently there is no mechanism to share data only containers between multiple minions.

[pwFoo]

I have a single host / minion setup and need local persistent storage. Hostdir without permission fix isn't a solution, because the user inside the container haven't permissions to write to the volume.

[erictune]

Kubernetes is optimized for the many-node use case. Sometimes, as a consequence, things that seem like they should be easy in a single-node case may not be supported.

[thockin]

Most docker containers today run as root, so you should be able to write to a hostPath mount. If you are running your containers as non-root (good for you!) I would be amenable to some different tweak on the model that gave you what you wanted. For example, something like

"hostPath": { "path": "/tmp/mypod", "uid": 93, "gid": 76, "mode": 0700 }

...with well-defined semantics (e.g. only applies if the path does not exist or already exists in the exact same config). or even something higher level to abstract the real host path away.

But first - can you explain what "persistent" means to you? Under what conditions is the data retained? What is the bounding lifetime?

On Wed, Apr 22, 2015 at 7:36 AM, pwFoo notifications@github.com wrote:

I have a single host / minion setup and need local persistent storage. Hostdir without permission fix isn't a solution, because the user inside the container haven't permissions to write to the volume.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-95210227 .

[pwFoo]

Hello thockin,

I need reboot save volumes used to store mysql databases or webspace content. So lifetime needed is forever (not dependent on service / pod / host uptime). Changes should be saved directly.

Because I have a single minion setup, hostdir with correct / adjustable permissions (uid / gid) should be fine at the moment.

Regards

[thockin]

OK, I'll give this some thought. No promises on timeframe, though.

On Fri, Apr 24, 2015 at 12:47 AM, pwFoo notifications@github.com wrote:

Hello thockin,

I need reboot save volumes used to store mysql databases or webspace content. So lifetime needed is forever (not dependent on service / pod / host uptime). Changes should be saved directly.

Because I have a single minion setup, hostdir with correct / adjustable permissions (uid / gid) should be fine at the moment.

Regards

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-95837246 .

[pwFoo]

Thanks @thockin :)

[pmorie]

I want to be able to mount the busybox binary from a busybox image into a container with an image that doesn't have a shell.

[thockin]

I think that's a great goal. I'm not sure we have bandwidth to hit this before 1.0 - especially considering therer are a bunch of issues to iron out around data containers and containers as volumes.

On Tue, May 12, 2015 at 4:53 PM, Paul Morie notifications@github.com wrote:

I want to be able to mount the busybox binary from a busybox image into a container with an image that doesn't have a shell.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-101462375 .

[pmorie]

No doubt @thockin, just wanted to record it On Tue, May 12, 2015 at 8:18 PM Tim Hockin notifications@github.com wrote:

I think that's a great goal. I'm not sure we have bandwidth to hit this before 1.0 - especially considering therer are a bunch of issues to iron out around data containers and containers as volumes.

On Tue, May 12, 2015 at 4:53 PM, Paul Morie notifications@github.com wrote:

I want to be able to mount the busybox binary from a busybox image into a container with an image that doesn't have a shell.

— Reply to this email directly or view it on GitHub < https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-101462375

.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/issues/831#issuecomment-101465996 .

[bgrant0607]

I think that people will want to separately manage several distinct file sets:

• base image
• debugging tools (like sh)
• general language runtime / environment / prerequisites
• other app dependencies
• app
• app static data
• app configuration
• various secrets (SSL, ssh, ...)

[elibixby]

+1 to wanting this!

My use case is as follows:

Have a cluster with a "cluster master". Have a set of scripts + libraries that "slave" images need to run to register with a cluster master. Would like users to be able to use images out-of-the-box as slaves, (without making a child image just for this use-case). Solution is to have a data volume image with scripts+libraries that users can mount read-only with "--volumes-from" and specify a command in their pod config running the script.

Current workaround is to make GCEPersistentDisk which originates from the data volume image, and mount that, but there is no reason this couldn't be made cloud-provider agnostic.

[aronchick]

@elibixby Interesting - could you just mount those scripts and libraries in an NFS volume? Or is that overly complex for the solution?

[elibixby]

Yeah it's a possible workaround. That becomes a lot more heavyweight if I want a large "library" of scripts or programs that I want to attach to containers. Much easier to maintain images in a image repository, than maintain a fleet of disks.

[resouer]

ping @thockin , what do you think if we support volumes-from within the Pod as I mentioned in #12526 ?

Good: It will not change the behavior of Pod It will release me of doing cp things to empty_dir (I can't do cp in Dockerfile as dest is a volume dir) It solves 90% of what people want in this issue and bring a new way to deploy apps

Bad: Kind of docker way, some owners may not like it ...

Other work around: we can make empty_dir do not clean the contents in Volume dir. Just by backing up and restore the contents to host dir (this is just what docker -v did).

cc @dchen1107 @erictune

[dalanlan]

/cc @dalanlan

[pmorie]

@resouer A couple thoughts on volumes-from:

1. This approach will be racy, since the source container will have to copy data into the volume. The container consuming data from the volume will have to:
   1. Know what to expect in the volume
   2. Monitor the volume and wait for the data to be copied in
2. As you say, this is a docker-only feature; I'm hesitant to depend on this since ideally all runtimes will be able to have parity as backends to run pods. As an alternative (though I think this would be really tricky), we could try exploding an image's rootfs into a volume directory, but this isn't a great solution.

[janosi]

+1 My humble use case is:

• configuration of applications would be on a tmpfs based voume, represented by a data container.(Docker support for tmpfs comes soon). There would be a single pod of this data container per host.
• application pods use this data pod to read their config. There can be multiple application pods using this single data pod on the same host. Using persistent storage is not OK, the configuration data is used very frequently, actually. I must have the speed of in-memory storage.
• the benefit of having a single instance is obvious: no need to reserve memory for the same data for each and every application pod.

OK, it is a "bit" different from your use case of having a "container" type volume inside a pod. My use case is about having a "pod" container type, which is accessible by other pods on the same host.

[bkeroackdsc]

Another +1 for this idea. A "container" volume type would simplify our current workflow of using a shared emptyDir volume that has to have shared data copied to it, and the consuming containers synchronized somehow.

[thockin]

@saad-ali another for the queue

[amygdala]

+1, I have just needed something like this as well.

[pmorie]

A couple thoughts / ideas:

1. We should decide whether this feature would depend on the configuration of the host; #8702 deals with generalizing image representation in the API; I'd prefer not to make a docker-only feature if we can avoid it. I'm not sure that we should go as far as promising to support whatever image format a user wants to use regardless of host configuration, but there is definitely some implicit linkage potential between this feature and what container runtime you're using, and that should at least be made explicit.
2. I don't like the idea of containers having to synchronize on the data to avoid races. I think if we move forward with this feature the data should be available in the pod when the pod is started. To accomplish this with docker, I think we would need to run a container, docker copy a path out of the container onto the host, and delete the container. Maybe there's more complicated stuff we can do to cache things to avoid doing this each time we start a pod.

[resouer]

@pmorie

1. yes, I agree with you. Docker specific features is not good. At least, we need to decide that based on containerRuntime and warn user
2. actually,copy data out is what I originally trying to do in #12526. but for now Pod doesn't support one-time-off task well: a) RC will continually restart the container, b) restartPolicy is not container specific.

So, as restartPolicy is Pod level, that remind me if we can allow a specific "task" container running in a Pod, things will be much better.

[janosi]

What do you think about having a "container manager" (e.g. Docker) specific part in the pod manifest? So, the owner of the pod descriptor could define here container manager specific options, and the task of kubelet would be to deliver those options to the container manager when the containers are created. A kind of transparent tool for delivering container manager specific data through the Kubernetes framework. It may help in other cases, too. E.g. "incubation" of new container manager features. Once such feature becomes a kind of standard for all container managers, it can be moved from this transparent part to the structured part of the pod descriptor.

[bgrant0607]

This issue is how to create a convenient, extensible way to get data into volumes. Persistence of local data is another issue -- #7562. The case of using --volumes-from just to keep track of a shared volume while it's in use isn't relevant to Kubernetes.

I think there are at least 2 cases we could support here:

1. Passive containers, where the image is pulled and the file system (the whole thing or parts of it) of the unpacked image could be mounted directly (i.e., not copied), as readonly. Just (ab)using images as a generic data delivery mechanism. I have no idea whether Docker users are doing this or not.
2. "Setup" containers that are executed prior to executing the other containers of the pod, for the purpose of populating volumes.

Some issues to work out with the latter:

• Retry behavior -- for instance, distinguishing permanent from retry-able failure
• Update behavior
  • Do we allow the setup container to be updated?
  • If so, does that kill all running pod containers? Or just ones depending on the volume(s) it writes?
• What the execution context is, such as allowed security context, secrets available, other volumes visible, what happens to logs, etc.
  • Answers would be simpler if it were just a container with a simple dependency model (i.e., all setup containers concurrently or in an arbitrary order, then all ordinary pod containers if/once all of the setup containers have succeeded).
  • However, not knowing where the data is written (e.g., which volume), whether the container is safe to execute on non-empty volumes, whether it has other side effects, etc. could make retry and update semantics harder, and would decrease isolation between the setup container and the rest of the pod (e.g., potentially exposing secrets).
• Ideally, any existing storage backend -- EmptyDir (tmpfs or disk), PD, NFS, etc. -- would be usable. For anything other than a fresh EmptyDir, the container would need to be safe to invoke on a non-empty volume.
  • This suggests that the mechanism shouldn't be represented as a new volume type, but possibly as a volume initialization hook.
  • If the whole pod were visible to the container, it would make more sense to represent the container as a pod hook or special container property than as a volume hook.
• Surfacing errors (and maybe progress?)
• Provenance, auditing, etc.

[kjvalencik]

@bgrant0607, #1 is a pretty common use case (at least for us). We build a web application container that provides both a server component as well as static assets. Ideally, we would be able to initialize a read only volume from that same image so that an nginx container can serve the assets.

This is probably similar to the one of the use cases for the git container. However, it's handy to use docker images as the only way to distribute artifacts as well as not check-in minified css / js.

Currently, we accomplish this in one of two ways:

1. Copying files in on start from another container in the pod.
2. Installing nginx in our image so that the same container can be used either for static assets or for application server.

Cheers!

[scrisenbery]

@kjvalencik How are you achieving your first solution right now? I am trying a similar solution but it doesn't appear as though the container holding the files is starting first, so the container that needs the files is failing to start because it cannot find them.

[kjvalencik]

@scrisenbery A bash start script that waits for the files to exist. Example:

Container providing files

# Copy the files to a temporary shared folder
cp -r /src/files /mnt/shared/files_tmp

# Rename the folder so that all files are available atomically
mv /mnt/shared/files_tmp /mnt/shared/files

Container that requires files

# Sleep until the shared files exist
until [ -d /mnt/shared/files ]; do sleep 1; done

# Start the application
./start.sh