search

simonsobs-uk / data-centre

This tracks the issues in the baseline design of the SO:UK Data Centre at Blackett
https://souk-data-centre.readthedocs.io
BSD 3-Clause "New" or "Revised" License
2 stars 1 forks source link

More questions related to CVMFS synchronization time #38

Closed ickc closed 4 months ago

ickc commented 7 months ago

@rwf14f, @afortiorama,

I've run some test about CVMFS synchronization time and here's what I got:

For a file wrote at 20231215-121641 on the publishing server, it is sync'd at 20231215-151407 on vm77. At 20231215-151407, files up to 20231215-145741 is sync'd, and any after that including 20231215-145841 is not sync'd yet. (These tests are done in 1 minute resolution. Files are created per minute on the publishing node, and monitored per minute on vm77.)

I don't quite understand these results, it probably means that there are multi-stage synchronization lag. And empirically it takes around 3 hours to sync. Is it possible to reduce this?

Another question I have is about synchronization order. Will files synchronize chronologically? I.e. is there a guarantee that files create later will always shows up later?

I'm trying to deploy softwares with a symlink that always point to the latest. E.g. immediately after so-pmpm-py310-mkl-x86-64-v3-mpich-20231215 is unpacked, so-pmpm-py310-mkl-x86-64-v3-mpich-latest -> so-pmpm-py310-mkl-x86-64-v3-mpich-20231215 is created. I wonder if there's an moment that on the client nodes (vm77 or any worker nodes) that this symlink will points to a directory that has not been sync'd yet, or worse yet where its contents are partially sync'd

[northgridsgm@cvmfs-uploader02 pmpm]$ pwd
/cvmfs/northgrid.gridpp.ac.uk/simonsobservatory/pmpm
[northgridsgm@cvmfs-uploader02 pmpm]$ ls -alh
total 599K
drwxr-xr-x 34 northgridsgm cvmfsvos 42 Dec 15 11:50 .
drwxr-xr-x  8 northgridsgm cvmfsvos  9 Dec 15 12:16 ..
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 12 13:59 so-pmpm-py310-mkl-x86-64-v3-mpich-20231212
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 13 01:11 so-pmpm-py310-mkl-x86-64-v3-mpich-20231213
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 14 01:10 so-pmpm-py310-mkl-x86-64-v3-mpich-20231214
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 15 01:12 so-pmpm-py310-mkl-x86-64-v3-mpich-20231215
lrwxrwxrwx  1 northgridsgm cvmfsvos 42 Dec 15 11:48 so-pmpm-py310-mkl-x86-64-v3-mpich-latest -> so-pmpm-py310-mkl-x86-64-v3-mpich-20231215
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 12 13:59 so-pmpm-py310-mkl-x86-64-v3-openmpi-20231212
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 13 01:11 so-pmpm-py310-mkl-x86-64-v3-openmpi-20231213
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 14 01:10 so-pmpm-py310-mkl-x86-64-v3-openmpi-20231214
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 15 01:12 so-pmpm-py310-mkl-x86-64-v3-openmpi-20231215
lrwxrwxrwx  1 northgridsgm cvmfsvos 44 Dec 15 11:47 so-pmpm-py310-mkl-x86-64-v3-openmpi-latest -> so-pmpm-py310-mkl-x86-64-v3-openmpi-20231215
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 12 13:59 so-pmpm-py310-mkl-x86-64-v4-mpich-20231212
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 13 01:12 so-pmpm-py310-mkl-x86-64-v4-mpich-20231213
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 14 01:10 so-pmpm-py310-mkl-x86-64-v4-mpich-20231214
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 15 01:12 so-pmpm-py310-mkl-x86-64-v4-mpich-20231215
lrwxrwxrwx  1 northgridsgm cvmfsvos 42 Dec 15 11:49 so-pmpm-py310-mkl-x86-64-v4-mpich-latest -> so-pmpm-py310-mkl-x86-64-v4-mpich-20231215
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 12 13:59 so-pmpm-py310-mkl-x86-64-v4-openmpi-20231212
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 13 01:11 so-pmpm-py310-mkl-x86-64-v4-openmpi-20231213
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 14 01:11 so-pmpm-py310-mkl-x86-64-v4-openmpi-20231214
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 15 01:12 so-pmpm-py310-mkl-x86-64-v4-openmpi-20231215
lrwxrwxrwx  1 northgridsgm cvmfsvos 44 Dec 15 11:48 so-pmpm-py310-mkl-x86-64-v4-openmpi-latest -> so-pmpm-py310-mkl-x86-64-v4-openmpi-20231215
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 12 13:59 so-pmpm-py310-nomkl-x86-64-v3-mpich-20231212
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 13 01:11 so-pmpm-py310-nomkl-x86-64-v3-mpich-20231213
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 14 01:10 so-pmpm-py310-nomkl-x86-64-v3-mpich-20231214
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 15 01:12 so-pmpm-py310-nomkl-x86-64-v3-mpich-20231215
lrwxrwxrwx  1 northgridsgm cvmfsvos 44 Dec 15 11:49 so-pmpm-py310-nomkl-x86-64-v3-mpich-latest -> so-pmpm-py310-nomkl-x86-64-v3-mpich-20231215
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 12 13:59 so-pmpm-py310-nomkl-x86-64-v3-openmpi-20231212
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 13 01:11 so-pmpm-py310-nomkl-x86-64-v3-openmpi-20231213
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 14 01:10 so-pmpm-py310-nomkl-x86-64-v3-openmpi-20231214
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 15 01:12 so-pmpm-py310-nomkl-x86-64-v3-openmpi-20231215
lrwxrwxrwx  1 northgridsgm cvmfsvos 46 Dec 15 11:49 so-pmpm-py310-nomkl-x86-64-v3-openmpi-latest -> so-pmpm-py310-nomkl-x86-64-v3-openmpi-20231215
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 12 13:59 so-pmpm-py310-nomkl-x86-64-v4-mpich-20231212
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 13 01:11 so-pmpm-py310-nomkl-x86-64-v4-mpich-20231213
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 14 01:10 so-pmpm-py310-nomkl-x86-64-v4-mpich-20231214
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 15 01:12 so-pmpm-py310-nomkl-x86-64-v4-mpich-20231215
lrwxrwxrwx  1 northgridsgm cvmfsvos 44 Dec 15 11:50 so-pmpm-py310-nomkl-x86-64-v4-mpich-latest -> so-pmpm-py310-nomkl-x86-64-v4-mpich-20231215
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 12 13:59 so-pmpm-py310-nomkl-x86-64-v4-openmpi-20231212
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 13 01:11 so-pmpm-py310-nomkl-x86-64-v4-openmpi-20231213
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 14 01:10 so-pmpm-py310-nomkl-x86-64-v4-openmpi-20231214
drwxr-xr-x 27 northgridsgm cvmfsvos 30 Dec 15 01:12 so-pmpm-py310-nomkl-x86-64-v4-openmpi-20231215
lrwxrwxrwx  1 northgridsgm cvmfsvos 46 Dec 15 11:50 so-pmpm-py310-nomkl-x86-64-v4-openmpi-latest -> so-pmpm-py310-nomkl-x86-64-v4-openmpi-20231215

C.f. #27

ickc commented 7 months ago

I figured out that the synchronization should be happening every 2 hours, some time between 49:43 to 50:42. And then seems like another synchronization happened between 09:57 to 10:56, which is 0.5 hour later.

So it seems the total lag will be somewhere between 2.5 to 3 hours.