Closed ignazio-bovo closed 3 months ago
A quick inspection to the rsync logs for copying the data :
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315019
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315121
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/131515
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315601
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315604
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315799
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315800
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315840
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315842
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315844
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315845
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1315846
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316003
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316005
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316007
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316008
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316009
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316011
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316013
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316015
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316025
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316026
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316030
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316032
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/1316037
overlay2/012dde5cafd3740ef6f34c6a784899617a83f4b77945e9abe55a6dcb43b3555a/diff/distributor_data/131617
And so on, probably revealing that the approach in the dwg-production repo needs to make the operator conscious about the distributor-data
folder
I have observed that on my server the
/var/lib/docker
used as mount point for the overlay volume is occupying 850GB that is all the available space in the vg0 partition for my partition schema.Long term solution
Investigate on the docker overlay volume size culprits
Quick solution
Reconfigure docker so that the
data-root
folder points to a directory in a bigger volume (like the one used to store asset, usually several TBs large)Moving the Docker
overlay2
volume to another directory within the same filesystem involves a process that ensures all your Docker data is safely transferred to the new location without data loss. Here's how you can do it:Step 1: Stop Docker Service
First, you need to stop Docker to ensure that no containers are running and no changes are being made to the data during the move. You can stop Docker using the following command:
Step 2: Copy the Docker Data Directory
Next, copy the entire Docker data directory (
/var/lib/docker
by default) to the new location. You can use thersync
command for this purpose as it's efficient and can resume interrupted transfers. Replace/new/docker/data/location
with the path to where you want to move the Docker data.Step 3: Update Docker's Storage Location
After copying the data, you need to tell Docker to use the new location for its data. This is done by editing or creating the Docker daemon configuration file, usually found at
/etc/docker/daemon.json
. If this file does not exist, you will create it; if it does, add or modify thedata-root
attribute.Make sure to replace
/new/docker/data/location
with the actual new path of your Docker data.Step 4: Restart Docker Service
Now, restart Docker to apply the changes. Docker will now use the new directory for all its data, including
overlay2
.Step 5: Verify and Clean Up
After Docker starts, verify that your containers and images are correctly recognized and that everything operates as expected. If all is well, you can remove the old Docker data directory to free up space on your system. Ensure that everything is working correctly before doing this, as this step is irreversible.
Important Considerations
By following these steps, you should be able to move the Docker
overlay2
volume to a new directory on the same filesystem without losing your Docker containers and images.