search

NetApp / netappdvp

A Docker volume plugin for NetApp storage
96 stars 33 forks source link

Stack deployments fail when using UCP 2.2.5 and Collections #117

Closed lordgnu closed 6 years ago

lordgnu commented 6 years ago

We upgraded to UCP 2.2.5 this morning and suddenly all of our deployments are failing when using the netappdvp plugin and collections. We always seem to get an error similar to this: failed to update service 7c1e4382_vtest_db-redis: Error response from daemon: located 1 sets of existing volumes named "7c1e4382_vtest_db_redis" with different collection labels. Please make sure all existing volumes with the same name share the same collection or don't have one at all

Our deployment system inspects services about to be deployed and creates the volumes ahead of time (with the correct collection labels) in the event they don't already exists. This was because (prior to UCP 2.2.5) volume labels were ignored when doing a docker stack deploy to a swarm. It seems the labels are now honored, but past the initial deployment, all subsequent deployments fail with the aforementioned error.

We are using version 17.10.1 of the netappdvp plugin with UCP 2.2.5 and Docker 17.06.2-ee-6

Below is some output from our deployment system showing the errors returned (as well as what labels were applied to the service). Following this output is the result of a docker volume inspect of the same volume raising the error from a management node, and that shows that the collection labels all match.

[2018-01-18T12:38:51-06:00][start] Beginning deployment 
[2018-01-18T12:38:51-06:00][info] Cloning repository from REDACTED 
[2018-01-18T12:38:52-06:00][info] Generating swarm.yml 
[2018-01-18T12:38:52-06:00][info] Resolved swarm.yml path as /tmp/swarm-test-7c50c1c67a2f4eba6f4afcfee2b8d163e42c942adbedb0adafb098960cb1e96f/docker/swarm/swarm.yml 
[2018-01-18T12:38:52-06:00][info] Processing services for image / network replacement 
[2018-01-18T12:38:52-06:00][info] Checking service: db-redis 
[2018-01-18T12:38:52-06:00][info] Overriding image: REDACTED/platform/db-redis:0.0.26 
[2018-01-18T12:38:52-06:00][info] Checking networks 
[2018-01-18T12:38:52-06:00][info] Assigning deployment to collection: /vtest 
[2018-01-18T12:38:52-06:00][info] Tagging deployment with user: dbauer 
[2018-01-18T12:38:52-06:00][info] Appending YML labels 
[2018-01-18T12:38:52-06:00][info] - Adding label: delivery.user=dbauer 
[2018-01-18T12:38:52-06:00][info] - Adding label: delivery.deployment=swarm-test-7c50c1c67a2f4eba6f4afcfee2b8d163e42c942adbedb0adafb098960cb1e96f 
[2018-01-18T12:38:52-06:00][info] - Adding label: delivery.master=true 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.collection.root=true 
[2018-01-18T12:38:52-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true 
[2018-01-18T12:38:52-06:00][info] Checking root volume definitions 
[2018-01-18T12:38:52-06:00][info] - Checking volume: db_redis (7c1e4382_vtest_db_redis) 
[2018-01-18T12:38:52-06:00][info] -- Found volume: db_redis (7c1e4382_vtest_db_redis) 
[2018-01-18T12:38:52-06:00][info] Checking root network definitions 
[2018-01-18T12:38:52-06:00][info] Processing network: 7c1e4382-redis-network 
[2018-01-18T12:38:52-06:00][info] Deploying service to swarm 
[2018-01-18T12:38:53-06:00][stderr] Updating service 7c1e4382_vtest_db-redis (id: nk918xdmi97hm9st35gygbd98) 
[2018-01-18T12:38:53-06:00][stderr] failed to update service 7c1e4382_vtest_db-redis: Error response from daemon: located 1 sets of existing volumes named "7c1e4382_vtest_db_redis" with different collection labels. Please make sure all existing volumes with the same name share the same collection or don't have one at all 
[2018-01-18T12:38:53-06:00][stderr] exit status 1 
[2018-01-18T12:38:53-06:00][error] Deployment has failed: an error was encountered during swarm deployment: exit status 1 
[2018-01-18T12:38:53-06:00][complete] Deployment failed

Volume inspection output:

docker volume inspect 7c1e4382_vtest_db_redis
[
    {
        "Driver": "netapp_sync:latest",
        "Labels": {
            "com.docker.swarm.whitelists": "[\"node==REDACTED\"]",
            "com.docker.ucp.access.label": "/vtest",
            "com.docker.ucp.collection": "b8dee80c-6b58-4d79-a33f-236c6fc8b4bb",
            "com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb": "true",
            "com.docker.ucp.collection.root": "true",
            "com.docker.ucp.collection.swarm": "true",
            "delivery.user": "dbauer"
        },
        "Mountpoint": "/var/lib/docker/plugins/65801f6b71e89c320b3fde10638395ed676beeb1cbaaa7fc6f15ac06695c3af3/rootfs",
        "Name": "7c1e4382_vtest_db_redis",
        "Options": {
            "size": "2G"
        },
        "Scope": "global",
        "Status": {
            "Snapshots": []
        }
    }
]

We have attempted this every way we can think of (starting from scratch) and all followup deployments seem to fail whether or not we manually create the volume or let docker handle that for us in the initial deployment; whether or not we omit the volume labels entirely, or only add "com.docker.ucp.access.label", or verbosely add every ucp label...

lordgnu commented 6 years ago

Update on this... It appears as long as you explicitly add ALL UCP collection labels to both the service and the volumes in the swarm template, the initial deployment works as well as the follow-up deployments. Caveat is you MUST create the volume ahead of time with all those labels defined as well, and all pre-existing volumes, even if they belong to the correct collection according to UCP must be destroyed and recreated (meaning whole stacks need to be torn down) in order for the update deployments to operate as expected.

Here's my logs from the attempt that proved to be successful:

######################################################################################
# Attempt : Deployment with ALL collections labels on volumes/services, pre-creation #
######################################################################################

Initial Deployment (Successful):

[2018-01-18T14:35:45-06:00][start] Beginning deployment
[2018-01-18T14:35:45-06:00][info] Cloning repository from ssh://REDACTED/db-redis.git
[2018-01-18T14:35:46-06:00][info] Generating swarm.yml
[2018-01-18T14:35:46-06:00][info] Resolved swarm.yml path as /tmp/swarm-test-e54b41ad3d63aae0ba06f9a9c5cf1c26a372d7d74331716b2696b1a9514a0035/docker/swarm/swarm.yml
[2018-01-18T14:35:46-06:00][info] Processing services for image / network replacement
[2018-01-18T14:35:46-06:00][info] Checking service: db-redis
[2018-01-18T14:35:46-06:00][info] Overriding image: REDACTED/platform/db-redis:master
[2018-01-18T14:35:46-06:00][info] Checking networks
[2018-01-18T14:35:48-06:00][info] Assigning deployment to collection: /vtest
[2018-01-18T14:35:48-06:00][info] Tagging deployment with user: dbauer
[2018-01-18T14:35:49-06:00][info] Appending YML labels at path services.db-redis.deploy.labels
[2018-01-18T14:35:49-06:00][info] - Adding label: delivery.user=dbauer
[2018-01-18T14:35:49-06:00][info] - Adding label: delivery.deployment=swarm-test-e54b41ad3d63aae0ba06f9a9c5cf1c26a372d7d74331716b2696b1a9514a0035
[2018-01-18T14:35:49-06:00][info] - Adding label: delivery.master=true
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.collection.root=true
[2018-01-18T14:35:49-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true
[2018-01-18T14:35:49-06:00][info] Checking root volume definitions
[2018-01-18T14:35:49-06:00][info] - Checking volume: db_redis (7c1e4382_vtest_db_redis)
[2018-01-18T14:35:49-06:00][info] -- Creating volume: db_redis (7c1e4382_vtest_db_redis)
[2018-01-18T14:35:53-06:00][info] Appending YML labels at path volumes.db_redis.labels
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.collection.root=true
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true
[2018-01-18T14:35:53-06:00][info] - Adding label: delivery.user=dbauer
[2018-01-18T14:35:53-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest
[2018-01-18T14:35:53-06:00][info] Checking root network definitions
[2018-01-18T14:35:53-06:00][info] Processing network: 7c1e4382-redis-network
[2018-01-18T14:35:53-06:00][info] Deploying service to swarm
[2018-01-18T14:35:54-06:00][stdout] Creating service 7c1e4382_vtest_db-redis
[2018-01-18T14:35:54-06:00][info] Waiting for service to become healthy
[2018-01-18T14:35:54-06:00][info] Will poll once every 30 seconds for up to 20 minutes...

Volume Inspection:

# docker volume inspect 7c1e4382_vtest_db_redis
[
    {
        "Driver": "netapp_sync:latest",
        "Labels": {
            "com.docker.swarm.whitelists": "[\"node==REDACTED\"]",
            "com.docker.ucp.access.label": "/vtest",
            "com.docker.ucp.collection": "b8dee80c-6b58-4d79-a33f-236c6fc8b4bb",
            "com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb": "true",
            "com.docker.ucp.collection.root": "true",
            "com.docker.ucp.collection.swarm": "true",
            "delivery.user": "dbauer"
        },
        "Mountpoint": "/var/lib/docker/plugins/9d30000b1042920d3636aade2291c2a8d54cd1521320ef58ca54b593daef8414/rootfs",
        "Name": "7c1e4382_vtest_db_redis",
        "Options": {
            "size": "2G"
        },
        "Scope": "global",
        "Status": {
            "Snapshots": []
        }
    }
]

Follow-up deployment with same settings:

[2018-01-18T14:37:23-06:00][start] Beginning deployment
[2018-01-18T14:37:23-06:00][info] Cloning repository from ssh://REDACTED/db-redis.git
[2018-01-18T14:37:24-06:00][info] Generating swarm.yml
[2018-01-18T14:37:24-06:00][info] Resolved swarm.yml path as /tmp/swarm-test-8f0cf7534731795a1f9cc7cd7f206f1af65ec49b37e839a0fdb579e57735d89b/docker/swarm/swarm.yml
[2018-01-18T14:37:24-06:00][info] Processing services for image / network replacement
[2018-01-18T14:37:24-06:00][info] Checking service: db-redis
[2018-01-18T14:37:24-06:00][info] Overriding image: REDACTED/platform/db-redis:0.0.26
[2018-01-18T14:37:24-06:00][info] Checking networks
[2018-01-18T14:37:24-06:00][info] Assigning deployment to collection: /vtest
[2018-01-18T14:37:24-06:00][info] Tagging deployment with user: dbauer
[2018-01-18T14:37:24-06:00][info] Appending YML labels at path services.db-redis.deploy.labels
[2018-01-18T14:37:24-06:00][info] - Adding label: delivery.user=dbauer
[2018-01-18T14:37:24-06:00][info] - Adding label: delivery.deployment=swarm-test-8f0cf7534731795a1f9cc7cd7f206f1af65ec49b37e839a0fdb579e57735d89b
[2018-01-18T14:37:24-06:00][info] - Adding label: delivery.master=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.root=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true
[2018-01-18T14:37:24-06:00][info] Checking root volume definitions
[2018-01-18T14:37:24-06:00][info] - Checking volume: db_redis (7c1e4382_vtest_db_redis)
[2018-01-18T14:37:24-06:00][info] -- Found volume: db_redis (7c1e4382_vtest_db_redis)
[2018-01-18T14:37:24-06:00][info] Appending YML labels at path volumes.db_redis.labels
[2018-01-18T14:37:24-06:00][info] - Adding label: delivery.user=dbauer
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.access.label=/vtest
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection=b8dee80c-6b58-4d79-a33f-236c6fc8b4bb
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.b8dee80c-6b58-4d79-a33f-236c6fc8b4bb=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.root=true
[2018-01-18T14:37:24-06:00][info] - Adding label: com.docker.ucp.collection.swarm=true
[2018-01-18T14:37:24-06:00][info] Checking root network definitions
[2018-01-18T14:37:24-06:00][info] Processing network: 7c1e4382-redis-network
[2018-01-18T14:37:24-06:00][info] Deploying service to swarm
[2018-01-18T14:37:25-06:00][stdout] Updating service 7c1e4382_vtest_db-redis (id: q3t32wmyoyku2bkgmrs4a8z43)
[2018-01-18T14:37:25-06:00][info] Waiting for service to become healthy
[2018-01-18T14:37:25-06:00][info] Will poll once every 30 seconds for up to 20 minutes...
adkerr commented 6 years ago

Moved to netapp/trident#87