search

weaveworks / footloose

Container Machines - Containers that look like Virtual Machines
Apache License 2.0
1.59k stars 122 forks source link

Changing backends orphans running machines #218

Open stealthybox opened 5 years ago

stealthybox commented 5 years ago

For this to behave any differently, we would either:

check for equivalently named/labelled machines in every backend (could be slow with a cluster or other networked backend)

or

store some local state about what backends are being used for which groups or machines. (adds imperative state / could be really buggy)

It might just be better to admit that footloose configs are declarative, but footloose commands for machine-management are imperative. I believe docker-compose has some similar behavior limitations.

Here's a detailed example:

If I create two groups of machines using the docker backend like so: 

cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 1
  spec:
    image: quay.io/footloose/ubuntu18.04
    name: a-%d
- count: 1
  spec:
    image: quay.io/footloose/ubuntu18.04
    name: b-%d
footloose create
INFO[0000] Docker Image: quay.io/footloose/ubuntu18.04 present locally 
INFO[0000] Docker Image: quay.io/footloose/ubuntu18.04 present locally 
INFO[0000] Creating machine: cluster-a-0 ...            
INFO[0001] Creating machine: cluster-b-0 ... 

footloose show
NAME          HOSTNAME   PORTS   IP           IMAGE                           CMD          STATE     BACKEND
cluster-a-0   a-0                172.17.0.2   quay.io/footloose/ubuntu18.04   /sbin/init   Running   
cluster-b-0   b-0                172.17.0.3   quay.io/footloose/ubuntu18.04   /sbin/init   Running

Then I edit one of the groups to use the ignite backend:

cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 1
  spec:
    image: quay.io/footloose/ubuntu18.04
    name: a-%d
- count: 1
  spec:
    image: quay.io/footloose/ubuntu18.04
    name: b-%d
    backend: ignite

As soon as I've edited the file, footloose will orphan the machines running for that spec:

sudo footloose show
NAME          HOSTNAME   PORTS   IP           IMAGE                           CMD          STATE         BACKEND
cluster-a-0   a-0                172.17.0.2   quay.io/footloose/ubuntu18.04   /sbin/init   Running       
cluster-b-0   b-0                             quay.io/footloose/ubuntu18.04                Not created   ignite
# notice the state for b-0 is 'Not created'

Let's try creating again in this state:

sudo footloose create
INFO[0000] Docker Image: quay.io/footloose/ubuntu18.04 present locally 
INFO[0000] Docker Image: quay.io/footloose/ubuntu18.04 present locally 
INFO[0000] Creating machine: cluster-a-0 ...            
INFO[0000] Machine cluster-a-0 is already created...    
INFO[0000] Creating machine: cluster-b-0 ...            
# b-0 is re-created as an ignite vm -- the running docker container is ignored

# now we have 2 containers (1 orphaned) + 1 ignite vm
docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS               NAMES
919da6ca901c        weaveworks/ignite:v0.5.2        "/usr/local/bin/igni…"   28 seconds ago      Up 27 seconds                           ignite-8abc2902767aff2b
ea479c46c743        quay.io/footloose/ubuntu18.04   "/sbin/init"             4 minutes ago       Up 4 minutes        22/tcp              cluster-b-0
6c83e1eb0aa9        quay.io/footloose/ubuntu18.04   "/sbin/init"             4 minutes ago       Up 4 minutes        22/tcp              cluster-a-0

ignite vm
VM ID           IMAGE                   KERNEL                  SIZE    CPUS    MEMORY      CREATED STATUS  IPS     PORTS   NAME
8abc2902767aff2b    quay.io/footloose/ubuntu18.04:latest    weaveworks/ignite-kernel:4.19.47    4.0 GB  2   1024.0 MB   38s ago Up 38s  172.17.0.4      cluster-b-0

and deleting everything:

sudo footloose delete
INFO[0000] Machine cluster-a-0 is started, stopping and deleting machine... 
INFO[0000] Deleting machine: 919da6ca901c ...           

ignite vm
VM ID   IMAGE   KERNEL  SIZE    CPUS    MEMORY  CREATED STATUS  IPS PORTS   NAME

docker ps
CONTAINER ID        IMAGE                           COMMAND             CREATED             STATUS              PORTS               NAMES
ea479c46c743        quay.io/footloose/ubuntu18.04   "/sbin/init"        5 minutes ago       Up 5 minutes        22/tcp              cluster-b-0
# deleting everything leaves behind the orphaned container

If we switch back to our first file that was only using docker, we can delete the orphaned container:

footloose delete
INFO[0000] Machine cluster-a-0 hasn't been created...   
INFO[0000] Machine cluster-b-0 is started, stopping and deleting machine...

Should we do anything to change this?

Again, this may be fine for the constraints of our design, but we should document this behavior. TODO: Find best spot in the docs for this info.

dlespiau commented 5 years ago

Thanks for the well written issue!

Currently footloose only acts on a single cluster, the one described in the config file and that's indeed the main limitation here.

I can see a world where footloose queries back some state from either docker or a CRI runtime (the CRI interface has some provision to store metadata with each container). This way we would be able to list the clusters that have been created in the past and act on them.

This seems like a desirable behaviour in the future!