Docker volume reverted to very old version after reboot

[aleskinen]

I am running 3 photon 2 cluster and using vsphere docker volumes. It seems, that docker volume used by our mariadb data volume has reverted to several months old version. I tried to copy 2 days old vmdk file from backupd from equallogic and still seems to contain old files. How is this possible?

Is there any tool to show files in vsphere docker volume vmdk?

I was using few months old version (latest) tried to update to new latest.

[govint]

@aleskinen,

1. Could you share the /var/log/docker-volume-vsphere.log or the /var/log/vsphere-storage-for-docker.log. Please also upload the /var/log/vmware/vmdk_ops.log from the ESX host.
2. Can you confirm if the logs would cover the time when the most current data was available to the present where it looks like the data has reverted.
3. Please also indicate what versions of docker and the plugin and the VIB on ESX are in use - both earlier and the upgraded version.
4. Can you also share where the volumes are located on ESX - it should be in a folder called "dockvols" on the same datastore as where the VM(s) is located. Please upload a listing of the contents of the folder where the volumes are residing and indicate which volume is the one in question.
5. You can list all the volumes via docker volume ls and it should show the ones created and managed via the vsphere plugin.

I'm curious if for some reason a different volume is getting attached now that was perhaps earlier in use for this DB. Once we know the volume used by the DB we should be able to track it usage from the time that the data was available till the present.

[aleskinen]

Attached logs. I made test: Created new volume Util with same properties. ssh to vmware host and changed name of mariadb-fata-flat.vmdk to Util-flat.vmdk while container using Util was not running. Restarted container. File list on volume did not changed. How does sycronization between progated- docker-volume-vsphere.log vmdk_ops.log

mount and vmdk contents get syncronized?

[aleskinen]

I have copies of vmdk files before accident. It would be really nice to be able to check what was the situation. Is there any easy way for to do that?

[govint]

@aleskinen any reason why the flat vmdk names are being changed. User isn't expected to do this. Is it done to use the maria-db data via a different vmdk (Util.vmdk).

VMDK content is the same as whats available via the propagated mount.

The logs are the only way to figure which disk was in use earlier and what happened recently.

Are you saying the problem happened after the update of the plugin?

[aleskinen]

No the problem occurred before update and I updated only to see if it would help something. And copying vmdk-files top of files same or similar in vmware host is only way I know to mount content so that I can manage files inside. I hve once managed to save one backup like that. We have snapshotting enabled in our iscsi disk server so it is possibe to find old vmdk files. But I do not know any way to access content in those.

[aleskinen]

docker info Containers: 8 Running: 8 Paused: 0 Stopped: 0 Images: 238 Server Version: 17.06.0-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: active Is Manager: true Managers: 3 Nodes: 3 Orchestration: Task History Retention Limit: 5 Raft: Snapshot Interval: 10000 Number of Old Snapshots to Retain: 0 Heartbeat Tick: 1 Election Tick: 3 Dispatcher: Heartbeat Period: 5 seconds CA Configuration: Expiry Duration: 3 months Force Rotate: 0 Root Rotation In Progress: false Manager Addresses: Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4 init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 4.9.80-1.ph2-esx Operating System: VMware Photon OS/Linux OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.86GiB Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

And plugin has been latest both before and after update, so I do not know what was before.

[aleskinen]

DRIVER VOLUME NAME local 5292c0afe62a9663dd29d66313b9d252a4c08d4f36b32e5e70c5ea564b6e2111 local customtheme vsphere:latest mariadb-data@d1 vsphere:latest odoo-db@d1 vsphere:latest odoo-lib@d1 local registry vsphere:latest tek-webroot@d1 vsphere:latest ttkirja-webroot@d1 vsphere:latest tukoke-webroot@d1 vsphere:latest yhteiso-files@d1

[aleskinen]

[root@vhost1r630:/vmfs/volumes/58da1e8b-9ac19bfb-5c9c-f04da23e46d4/dockvols/11111111-1111-1111-1111-111111111111] ls -l total 109743176 -rw------- 1 root root 4096 Apr 5 07:51 mariadb-data-3797b56d3a7e9693.vmfd -rw------- 1 root root 4096 Apr 4 09:37 mariadb-data-3797b56d3a7e9693.vmfd04-02 -rw------- 1 root root 42949672960 Apr 5 07:51 mariadb-data-flat.vmdk -rw------- 1 root root 42949672960 Apr 5 10:06 mariadb-data-flat.vmdk04-02 -rw------- 1 root root 628 Apr 5 07:51 mariadb-data.vmdk -rw------- 1 root root 628 Apr 4 09:37 mariadb-data.vmdk04-02 -rw------- 1 root root 42949672960 Apr 5 09:51 mariadb-flat.vmdk -rw------- 1 root root 4096 Jan 9 10:13 odoo-db-4d54519c5d4faaea.vmfd -rw------- 1 root root 3277312 Apr 4 18:30 odoo-db-ctk.vmdk -rw------- 1 root root 53687091200 Jan 9 10:14 odoo-db-flat.vmdk -rw------- 1 root root 678 Feb 23 15:01 odoo-db.vmdk -rw------- 1 root root 4096 Mar 9 12:11 odoo-lib-61e5a6b3044be2bb.vmfd -rw------- 1 root root 6554112 Apr 4 18:30 odoo-lib-ctk.vmdk -rw------- 1 root root 107374182400 Mar 9 12:11 odoo-lib-flat.vmdk -rw------- 1 root root 682 Mar 9 12:11 odoo-lib.vmdk -rw------- 1 root root 6554112 Apr 4 18:25 tek-webroot-ctk.vmdk -rw------- 1 root root 4096 Apr 3 19:57 tek-webroot-f0298c874b2b09ab.vmfd -rw------- 1 root root 107374182400 Apr 5 11:40 tek-webroot-flat.vmdk -rw------- 1 root root 691 Apr 4 18:24 tek-webroot.vmdk -rw------- 1 root root 4096 Apr 4 10:59 ttkirja-webroot-3960a17f36abba62.vmfd -rw------- 1 root root 2621952 Apr 4 18:11 ttkirja-webroot-ctk.vmdk -rw------- 1 root root 42949672960 Apr 4 10:59 ttkirja-webroot-flat.vmdk -rw------- 1 root root 701 Apr 4 10:59 ttkirja-webroot.vmdk -rw------- 1 root root 4096 Apr 4 12:54 tukoke-webroot-0233489a55f88b3c.vmfd -rw------- 1 root root 2621952 Apr 4 18:11 tukoke-webroot-ctk.vmdk -rw------- 1 root root 42949672960 Apr 4 12:54 tukoke-webroot-flat.vmdk -rw------- 1 root root 698 Apr 4 12:54 tukoke-webroot.vmdk -rw------- 1 root root 2621952 Apr 4 19:15 yhteiso-files-ctk.vmdk -rw------- 1 root root 4096 Apr 4 19:14 yhteiso-files-debd7c3a1e235e48.vmfd -rw------- 1 root root 42949672960 Apr 4 19:15 yhteiso-files-flat.vmdk -rw------- 1 root root 641 Apr 4 19:14 yhteiso-files.vmdk -rw------- 1 root root 4096 Jan 30 10:13 yhteiso-tmp-7944d313e7fc767c.vmfd.orig -rw------- 1 root root 42949672960 Jan 30 10:13 yhteiso-tmp-flat.vmdk.orig -rw------- 1 root root 572 Jan 30 09:28 yhteiso-tmp.vmdk.orig

[aleskinen]

Based on host backups it seems that content of propagated mount has not been same that was in running container. I have use independent-persistent volumes so they are not in host backup.

Compose file for mariadb was: version: '3.3'

services: mariadb: image: mariadb:10.3 volumes:

• mariadb-data:/var/lib/mysql
• /srv/docker/mariadb-backups:/backups/mysql environment:
• MYSQL_ROOT_PASSWORD_FILE=/run/secrets/mariadb_root_pwd networks:
• mariadb secrets:
• mariadb_root_pwd deploy: replicas: 1 placement: constraints: [node.hostname == dca]

networks: mariadb: external: true

volumes: mariadb-data: external: true

secrets: mariadb_root_pwd: external: true

[govint]

@aleskinen thanks for the logs and the details I'll review and update.