script fails

[mrnicegyu11]

Running this on master06 and master07 (GPU nodes) fails with the following logs reproducibly:

docker run --rm --pull always --volume /docker/volumes/:/var/lib/docker/volumes/ --volume /var/run/docker.sock:/var/run/docker.sock itisfoundation/osparc-ops-dy-volumes-cleanup dyvc s3.amazonaws.com ACCESS SECRET BUCKET AWS --s3-region eu-central-1

Skipped in use docker volume: 'dyv_5c9199aa-3f38-468e-8319-9b3f1e81728f_78986540-dbca-50ea-adda-58d68fb06870_erots-derahs_'
['rclone', '--config', '/tmp/rclone_config.ini', '--low-level-retries', '3', '--retries', '3', '--transfers', '5', 'sync', '/docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-7132-45c5-88df-eb2da83c7a58_stupni_krow_ums_emoh_/_data', 'dst:/BUCKET/master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206-7132-45c5-88df-eb2da83c7a58/651cc0b5-909d-4ae6-b21c-687f1f385c87', '-P', '--exclude', '.hidden_do_not_remove', '--exclude', 'key_values.json']
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/lib/python3.10/site-packages/dy_volumes_cleanup/cli.py:74 in main │
│                                                                              │
│   71 │   │   FILES_TO_EXCLUDE, help="Files to ignore when syncing to s3"     │
│   72 │   ),                                                                  │
│   73 ):                                                                      │
│ ❱ 74 │   asyncio.get_event_loop().run_until_complete(                        │
│   75 │   │   async_entrypoint(                                               │
│   76 │   │   │   s3_endpoint=s3_endpoint,                                    │
│   77 │   │   │   s3_access_key=s3_access_key,                                │
│                                                                              │
│ ╭─────────────────────────── locals ────────────────────────────╮            │
│ │  exclude_files = ['.hidden_do_not_remove', 'key_values.json'] │            │
│ │  s3_access_key = 'XXX'                       │            │
│ │      s3_bucket = 'XXX'            │            │
│ │    s3_endpoint = 's3.amazonaws.com'                           │            │
│ │ s3_parallelism = 5                                            │            │
│ │    s3_provider = <S3Provider.AWS: 'AWS'>                      │            │
│ │      s3_region = 'eu-central-1'                               │            │
│ │     s3_retries = 3                                            │            │
│ │  s3_secret_key = 'XXX'   │            │
│ ╰───────────────────────────────────────────────────────────────╯            │
│                                                                              │
│ /usr/local/lib/python3.10/asyncio/base_events.py:646 in run_until_complete   │
│                                                                              │
│    643 │   │   if not future.done():                                         │
│    644 │   │   │   raise RuntimeError('Event loop stopped before Future comp │
│    645 │   │                                                                 │
│ ❱  646 │   │   return future.result()                                        │
│    647 │                                                                     │
│    648 │   def stop(self):                                                   │
│    649 │   │   """Stop running the event loop.                               │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │   future = <Task finished name='Task-1' coro=<async_entrypoint() done,   │ │
│ │            defined at                                                    │ │
│ │            /usr/local/lib/python3.10/site-packages/dy_volumes_cleanup/c… │ │
│ │            exception=RuntimeError('Shell subprocesses yielded nonzero    │ │
│ │            error code 3  for command rclone --config                     │ │
│ │            /tmp/rclone_config.ini --low-level-retries 3 --retries 3      │ │
│ │            --transfers 5 sync                                            │ │
│ │            /docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d1… │ │
│ │            dst:/master-simcore-encrypted-backup/master-simcore/fa40b3a0… │ │
│ │            -P --exclude .hidden_do_not_remove --exclude                  │ │
│ │            key_values.json\n\n2022-10-07 14:02:35 ERROR : Local file     │ │
│ │            system at                                                     │ │
│ │            /docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d1… │ │
│ │            error reading source root directory: directory not            │ │
│ │            found\nTransferred:   \t          0 B / 0 B, -, 0 B/s, ETA    │ │
│ │            -\nElapsed time:         0.1s\n2022-10-07 14:02:35 ERROR : S3 │ │
│ │            bucket master-simcore-encrypted-backup path                   │ │
│ │            master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206… │ │
│ │            not deleting files as there were IO errors\nTransferred:   \t │ │
│ │            0 B / 0 B, -, 0 B/s, ETA -\nErrors:                 1         │ │
│ │            (retrying may help)\nElapsed time:         0.1s\n2022-10-07   │ │
│ │            14:02:35 ERROR : S3 bucket master-simcore-encrypted-backup    │ │
│ │            path                                                          │ │
│ │            master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206… │ │
│ │            not deleting directories as there were IO                     │ │
│ │            errors\nTransferred:   \t          0 B / 0 B, -, 0 B/s, ETA   │ │
│ │            -\nErrors:                 1 (retrying may help)\nElapsed     │ │
│ │            time:         0.1s\n2022-10-07 14:02:35 ERROR : Attempt 1/3   │ │
│ │            failed with 1 errors and: directory not found\nTransferred:   │ │
│ │            \t          0 B / 0 B, -, 0 B/s, ETA -\nErrors:               │ │
│ │            1 (retrying may help)\nElapsed time:         0.1s\n2022-10-07 │ │
│ │            14:02:35 ERROR : Local file system at                         │ │
│ │            /docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d1… │ │
│ │            error reading source root directory: directory not            │ │
│ │            found\nTransferred:   \t          0 B / 0 B, -, 0 B/s, ETA    │ │
│ │            -\nElapsed time:         0.1s\n2022-10-07 14:02:35 ERROR : S3 │ │
│ │            bucket master-simcore-encrypted-backup path                   │ │
│ │            master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206… │ │
│ │            not deleting files as there were IO errors\nTransferred:   \t │ │
│ │            0 B / 0 B, -, 0 B/s, ETA -\nErrors:                 1         │ │
│ │            (retrying may help)\nElapsed time:         0.1s\n2022-10-07   │ │
│ │            14:02:35 ERROR : S3 bucket master-simcore-encrypted-backup    │ │
│ │            path                                                          │ │
│ │            master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206… │ │
│ │            not deleting directories as there were IO                     │ │
│ │            errors\nTransferred:   \t          0 B / 0 B, -, 0 B/s, ETA   │ │
│ │            -\nErrors:                 1 (retrying may help)\nElapsed     │ │
│ │            time:         0.1s\n2022-10-07 14:02:35 ERROR : Attempt 2/3   │ │
│ │            failed with 1 errors and: directory not found\nTransferred:   │ │
│ │            \t          0 B / 0 B, -, 0 B/s, ETA -\nErrors:               │ │
│ │            1 (retrying may help)\nElapsed time:         0.1s\n2022-10-07 │ │
│ │            14:02:35 ERROR : Local file system at                         │ │
│ │            /docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d1… │ │
│ │            error reading source root directory: directory not            │ │
│ │            found\nTransferred:   \t          0 B / 0 B, -, 0 B/s, ETA    │ │
│ │            -\nElapsed time:         0.1s\n2022-10-07 14:02:35 ERROR : S3 │ │
│ │            bucket master-simcore-encrypted-backup path                   │ │
│ │            master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206… │ │
│ │            not deleting files as there were IO errors\nTransferred:   \t │ │
│ │            0 B / 0 B, -, 0 B/s, ETA -\nErrors:                 1         │ │
│ │            (retrying may help)\nElapsed time:         0.1s\n2022-10-07   │ │
│ │            14:02:35 ERROR : S3 bucket master-simcore-encrypted-backup    │ │
│ │            path                                                          │ │
│ │            master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206… │ │
│ │            not deleting directories as there were IO                     │ │
│ │            errors\nTransferred:   \t          0 B / 0 B, -, 0 B/s, ETA   │ │
│ │            -\nErrors:                 1 (retrying may help)\nElapsed     │ │
│ │            time:         0.1s\n2022-10-07 14:02:35 ERROR : Attempt 3/3   │ │
│ │            failed with 1 errors and: directory not found\nTransferred:   │ │
│ │            \t          0 B / 0 B, -, 0 B/s, ETA -\nErrors:               │ │
│ │            1 (retrying may help)\nElapsed time:         0.1sTransferred: │ │
│ │            \t          0 B / 0 B, -, 0 B/s, ETA -\nErrors:               │ │
│ │            1 (retrying may help)\nElapsed time:         0.1s\n2022/10/07 │ │
│ │            14:02:35 Failed to sync: directory not found\n')>             │ │
│ │ new_task = True                                                          │ │
│ │     self = <_UnixSelectorEventLoop running=False closed=False            │ │
│ │            debug=False>                                                  │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/dy_volumes_cleanup/cli.py:39 in      │
│ async_entrypoint                                                             │
│                                                                              │
│   36 │   │   │   │   typer.echo(f"Skipped in use docker volume: '{volume_nam │
│   37 │   │   │   │   continue                                                │
│   38 │   │   │                                                               │
│ ❱ 39 │   │   │   await store_to_s3(                                          │
│   40 │   │   │   │   dyv_volume=dyv_volume,                                  │
│   41 │   │   │   │   s3_endpoint=s3_endpoint,                                │
│   42 │   │   │   │   s3_access_key=s3_access_key,                            │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │         client = <aiodocker.docker.Docker object at 0x7f121d4062c0>      │ │
│ │     dyv_volume = {                                                       │ │
│ │                  │   'CreatedAt': '2022-09-29T08:21:09Z',                │ │
│ │                  │   'Driver': 'local',                                  │ │
│ │                  │   'Labels': {                                         │ │
│ │                  │   │   'node_uuid':                                    │ │
│ │                  '1d18a206-7132-45c5-88df-eb2da83c7a58',                 │ │
│ │                  │   │   'run_id':                                       │ │
│ │                  '651cc0b5-909d-4ae6-b21c-687f1f385c87',                 │ │
│ │                  │   │   'source':                                       │ │
│ │                  'dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-713… │ │
│ │                  │   │   'study_id':                                     │ │
│ │                  'fa40b3a0-3fce-11ed-811a-02420a18751d',                 │ │
│ │                  │   │   'swarm_stack_name': 'master-simcore',           │ │
│ │                  │   │   'user_id': '5'                                  │ │
│ │                  │   },                                                  │ │
│ │                  │   'Mountpoint':                                       │ │
│ │                  '/docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385… │ │
│ │                  │   'Name':                                             │ │
│ │                  'dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-713… │ │
│ │                  │   'Options': None,                                    │ │
│ │                  │   'Scope': 'local'                                    │ │
│ │                  }                                                       │ │
│ │    dyv_volumes = [                                                       │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-30T11:39:11Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  '78986540-dbca-50ea-adda-58d68fb06870',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  '5c9199aa-3f38-468e-8319-9b3f1e81728f',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_5c9199aa-3f38-468e-8319-9b3f1e81728f_78986540-dbc… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  '6c741ae4-40b4-11ed-8db3-02420a18756e',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '726'                            │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_5c9199aa-3f38-468e-8319-9b3f1e817… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_5c9199aa-3f38-468e-8319-9b3f1e81728f_78986540-dbc… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-29T08:21:09Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  '1d18a206-7132-45c5-88df-eb2da83c7a58',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  '651cc0b5-909d-4ae6-b21c-687f1f385c87',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-713… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  'fa40b3a0-3fce-11ed-811a-02420a18751d',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '5'                              │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-713… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-29T08:37:41Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  'a0f20579-7c0e-582f-9dc9-b6337478a868',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  'c9043283-6e00-40f2-a208-25a0e69c4c5c',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_c9043283-6e00-40f2-a208-25a0e69c4c5c_a0f20579-7c0… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  '43e6284e-3fd1-11ed-9b26-02420a1875fa',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '713'                            │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_c9043283-6e00-40f2-a208-25a0e69c4… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_c9043283-6e00-40f2-a208-25a0e69c4c5c_a0f20579-7c0… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-29T11:36:53Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  '10b3e12e-f123-5858-a490-e1837a27e1be',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  '9e505458-198f-4155-a7a3-5657deb79f42',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_9e505458-198f-4155-a7a3-5657deb79f42_10b3e12e-f12… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  'ea4faed0-3fea-11ed-9138-02420a187550',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '727'                            │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_9e505458-198f-4155-a7a3-5657deb79… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_9e505458-198f-4155-a7a3-5657deb79f42_10b3e12e-f12… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-29T11:37:18Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  '5432f8a4-1cf7-5244-9889-c1f4c28bc218',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  '50039992-acb6-4cdc-b4e9-3cc5dc8d8f20',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_50039992-acb6-4cdc-b4e9-3cc5dc8d8f20_5432f8a4-1cf… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  'f1e41a1e-3fea-11ed-9138-02420a187550',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '729'                            │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_50039992-acb6-4cdc-b4e9-3cc5dc8d8… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_50039992-acb6-4cdc-b4e9-3cc5dc8d8f20_5432f8a4-1cf… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-30T11:39:18Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  'aa11c008-bea9-5c75-9c87-f81cf114559a',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  '3b537086-55b2-48c5-a7f9-99aa6fd96b49',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_3b537086-55b2-48c5-a7f9-99aa6fd96b49_aa11c008-bea… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  '6cf6b03a-40b4-11ed-8ee5-02420a18756f',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '725'                            │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_3b537086-55b2-48c5-a7f9-99aa6fd96… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_3b537086-55b2-48c5-a7f9-99aa6fd96b49_aa11c008-bea… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-29T08:38:02Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  'f3a590ab-2b02-5266-84da-2c07396bd1e8',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  'e78f20ab-bd3e-444a-a405-86e684b90f70',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_e78f20ab-bd3e-444a-a405-86e684b90f70_f3a590ab-2b0… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  '4667fbf6-3fd1-11ed-88d1-02420a1875f9',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '722'                            │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_e78f20ab-bd3e-444a-a405-86e684b90… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_e78f20ab-bd3e-444a-a405-86e684b90f70_f3a590ab-2b0… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-29T08:37:41Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  'a0f20579-7c0e-582f-9dc9-b6337478a868',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  'c9043283-6e00-40f2-a208-25a0e69c4c5c',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_c9043283-6e00-40f2-a208-25a0e69c4c5c_a0f20579-7c0… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  '43e6284e-3fd1-11ed-9b26-02420a1875fa',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '713'                            │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_c9043283-6e00-40f2-a208-25a0e69c4… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_c9043283-6e00-40f2-a208-25a0e69c4c5c_a0f20579-7c0… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-29T11:38:02Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  '10b3e12e-f123-5858-a490-e1837a27e1be',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  '9e505458-198f-4155-a7a3-5657deb79f42',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_9e505458-198f-4155-a7a3-5657deb79f42_10b3e12e-f12… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  'ea4faed0-3fea-11ed-9138-02420a187550',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '727'                            │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_9e505458-198f-4155-a7a3-5657deb79… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_9e505458-198f-4155-a7a3-5657deb79f42_10b3e12e-f12… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   {                                                   │ │
│ │                  │   │   'CreatedAt': '2022-09-29T15:39:59Z',            │ │
│ │                  │   │   'Driver': 'local',                              │ │
│ │                  │   │   'Labels': {                                     │ │
│ │                  │   │   │   'node_uuid':                                │ │
│ │                  '36b72452-7cb9-5feb-b3d7-ce3bff1f2ea2',                 │ │
│ │                  │   │   │   'run_id':                                   │ │
│ │                  '71ce33d5-297b-474c-8f27-7acc8020b674',                 │ │
│ │                  │   │   │   'source':                                   │ │
│ │                  'dyv_71ce33d5-297b-474c-8f27-7acc8020b674_36b72452-7cb… │ │
│ │                  │   │   │   'study_id':                                 │ │
│ │                  'bc6c187e-400c-11ed-8db3-02420a18756e',                 │ │
│ │                  │   │   │   'swarm_stack_name': 'master-simcore',       │ │
│ │                  │   │   │   'user_id': '730'                            │ │
│ │                  │   │   },                                              │ │
│ │                  │   │   'Mountpoint':                                   │ │
│ │                  '/docker/volumes/dyv_71ce33d5-297b-474c-8f27-7acc8020b… │ │
│ │                  │   │   'Name':                                         │ │
│ │                  'dyv_71ce33d5-297b-474c-8f27-7acc8020b674_36b72452-7cb… │ │
│ │                  │   │   'Options': None,                                │ │
│ │                  │   │   'Scope': 'local'                                │ │
│ │                  │   },                                                  │ │
│ │                  │   ... +195                                            │ │
│ │                  ]                                                       │ │
│ │  exclude_files = ['.hidden_do_not_remove', 'key_values.json']            │ │
│ │  s3_access_key = 'XXX'                                  │ │
│ │      s3_bucket = 'XXX'                       │ │
│ │    s3_endpoint = 's3.amazonaws.com'                                      │ │
│ │ s3_parallelism = 5                                                       │ │
│ │    s3_provider = <S3Provider.AWS: 'AWS'>                                 │ │
│ │      s3_region = 'eu-central-1'                                          │ │
│ │     s3_retries = 3                                                       │ │
│ │  s3_secret_key = 'XXX'              │ │
│ │    volume_name = 'dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-713… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /usr/local/lib/python3.10/site-packages/dy_volumes_cleanup/_s3.py:115 in     │
│ store_to_s3                                                                  │
│                                                                              │
│   112 │   stdout, _ = await process.communicate()                            │
│   113 │                                                                      │
│   114 │   if process.returncode != 0:                                        │
│ ❱ 115 │   │   raise RuntimeError(                                            │
│   116 │   │   │   f"Shell subprocesses yielded nonzero error code {process.r │
│   117 │   │   │   f" for command {str_r_clone_command}\n{stdout.decode()}"   │
│   118 │   │   )                                                              │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                   _ = None                                               │ │
│ │    config_file_path = PosixPath('/tmp/rclone_config.ini')                │ │
│ │          dyv_volume = {                                                  │ │
│ │                       │   'CreatedAt': '2022-09-29T08:21:09Z',           │ │
│ │                       │   'Driver': 'local',                             │ │
│ │                       │   'Labels': {                                    │ │
│ │                       │   │   'node_uuid':                               │ │
│ │                       '1d18a206-7132-45c5-88df-eb2da83c7a58',            │ │
│ │                       │   │   'run_id':                                  │ │
│ │                       '651cc0b5-909d-4ae6-b21c-687f1f385c87',            │ │
│ │                       │   │   'source':                                  │ │
│ │                       'dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a20… │ │
│ │                       │   │   'study_id':                                │ │
│ │                       'fa40b3a0-3fce-11ed-811a-02420a18751d',            │ │
│ │                       │   │   'swarm_stack_name': 'master-simcore',      │ │
│ │                       │   │   'user_id': '5'                             │ │
│ │                       │   },                                             │ │
│ │                       │   'Mountpoint':                                  │ │
│ │                       '/docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f… │ │
│ │                       │   'Name':                                        │ │
│ │                       'dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a20… │ │
│ │                       │   'Options': None,                               │ │
│ │                       │   'Scope': 'local'                               │ │
│ │                       }                                                  │ │
│ │       exclude_files = ['.hidden_do_not_remove', 'key_values.json']       │ │
│ │             process = <Process 6>                                        │ │
│ │     r_clone_command = [                                                  │ │
│ │                       │   'rclone',                                      │ │
│ │                       │   '--config',                                    │ │
│ │                       │   '/tmp/rclone_config.ini',                      │ │
│ │                       │   '--low-level-retries',                         │ │
│ │                       │   '3',                                           │ │
│ │                       │   '--retries',                                   │ │
│ │                       │   '3',                                           │ │
│ │                       │   '--transfers',                                 │ │
│ │                       │   '5',                                           │ │
│ │                       │   'sync',                                        │ │
│ │                       │   ... +7                                         │ │
│ │                       ]                                                  │ │
│ │       s3_access_key = 'XXX'                             │ │
│ │           s3_bucket = 'XXX'                  │ │
│ │         s3_endpoint = 's3.amazonaws.com'                                 │ │
│ │      s3_parallelism = 5                                                  │ │
│ │             s3_path = PosixPath('/master-simcore-encrypted-backup/maste… │ │
│ │         s3_provider = <S3Provider.AWS: 'AWS'>                            │ │
│ │           s3_region = 'eu-central-1'                                     │ │
│ │          s3_retries = 3                                                  │ │
│ │       s3_secret_key = 'XXX'         │ │
│ │          source_dir = '/docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f… │ │
│ │              stdout = b'\n2022-10-07 14:02:35 ERROR : Local file system  │ │
│ │                       at /docker/volumes/dyv_651cc0b5-9'+3909            │ │
│ │ str_r_clone_command = 'rclone --config /tmp/rclone_config.ini            │ │
│ │                       --low-level-retries 3 --retries 3 --trans'+357     │ │
│ │          to_exclude = 'key_values.json'                                  │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Shell subprocesses yielded nonzero error code 3  for command
rclone --config /tmp/rclone_config.ini --low-level-retries 3 --retries 3
--transfers 5 sync
/docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-7132-45c5-88df
-eb2da83c7a58_stupni_krow_ums_emoh_/_data
dst:/master-simcore-encrypted-backup/master-simcore/fa40b3a0-3fce-11ed-811a-0242
0a18751d/1d18a206-7132-45c5-88df-eb2da83c7a58/651cc0b5-909d-4ae6-b21c-687f1f385c
87 -P --exclude .hidden_do_not_remove --exclude key_values.json

2022-10-07 14:02:35 ERROR : Local file system at
/docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-7132-45c5-88df
-eb2da83c7a58_stupni_krow_ums_emoh_/_data: error reading source root directory:
directory not found
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : S3 bucket master-simcore-encrypted-backup path
master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206-7132-45c5-88df-eb2d
a83c7a58/651cc0b5-909d-4ae6-b21c-687f1f385c87: not deleting files as there were
IO errors
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : S3 bucket master-simcore-encrypted-backup path
master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206-7132-45c5-88df-eb2d
a83c7a58/651cc0b5-909d-4ae6-b21c-687f1f385c87: not deleting directories as there
were IO errors
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : Attempt 1/3 failed with 1 errors and: directory not
found
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : Local file system at
/docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-7132-45c5-88df
-eb2da83c7a58_stupni_krow_ums_emoh_/_data: error reading source root directory:
directory not found
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : S3 bucket master-simcore-encrypted-backup path
master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206-7132-45c5-88df-eb2d
a83c7a58/651cc0b5-909d-4ae6-b21c-687f1f385c87: not deleting files as there were
IO errors
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : S3 bucket master-simcore-encrypted-backup path
master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206-7132-45c5-88df-eb2d
a83c7a58/651cc0b5-909d-4ae6-b21c-687f1f385c87: not deleting directories as there
were IO errors
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : Attempt 2/3 failed with 1 errors and: directory not
found
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : Local file system at
/docker/volumes/dyv_651cc0b5-909d-4ae6-b21c-687f1f385c87_1d18a206-7132-45c5-88df
-eb2da83c7a58_stupni_krow_ums_emoh_/_data: error reading source root directory:
directory not found
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : S3 bucket master-simcore-encrypted-backup path
master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206-7132-45c5-88df-eb2d
a83c7a58/651cc0b5-909d-4ae6-b21c-687f1f385c87: not deleting files as there were
IO errors
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : S3 bucket master-simcore-encrypted-backup path
master-simcore/fa40b3a0-3fce-11ed-811a-02420a18751d/1d18a206-7132-45c5-88df-eb2d
a83c7a58/651cc0b5-909d-4ae6-b21c-687f1f385c87: not deleting directories as there
were IO errors
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s
2022-10-07 14:02:35 ERROR : Attempt 3/3 failed with 1 errors and: directory not
found
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1sTransferred:                    0 B / 0 B, -, 0 B/s,
ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.1s
2022/10/07 14:02:35 Failed to sync: directory not found

For further information, on master07 this is

df -h 
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  225G   18G  196G   9% /
/dev/nvme0n1p2                     2.0G  155M  1.7G   9% /boot
/dev/nvme0n1p1                     1.1G  5.3M  1.1G   1% /boot/efi
/dev/sda1                          2.7T  138G  2.5T   6% /docker
...

and some selected lines from docker volume ls

local     dyv_f1c12281-cbbd-4c14-85b3-a6200f03690f_fe9144ba-fc17-556e-ad6f-06c9470df8eb_ecapskrow_krow_ums_emoh_
local     dyv_f1c12281-cbbd-4c14-85b3-a6200f03690f_fe9144ba-fc17-556e-ad6f-06c9470df8eb_erots-derahs_
local     dyv_f1c12281-cbbd-4c14-85b3-a6200f03690f_fe9144ba-fc17-556e-ad6f-06c9470df8eb_stupni_krow_ums_emoh_
local     dyv_f1c12281-cbbd-4c14-85b3-a6200f03690f_fe9144ba-fc17-556e-ad6f-06c9470df8eb_stuptuo_krow_ums_emoh_
local     dyv_f8887b67-51fa-4a48-8b19-699f68c03c00_81a6183c-f655-5266-b3dc-c02d40ea44a0_ecapskrow_krow_ums_emoh_
local     dyv_f8887b67-51fa-4a48-8b19-699f68c03c00_81a6183c-f655-5266-b3dc-c02d40ea44a0_erots-derahs_
local     dyv_f8887b67-51fa-4a48-8b19-699f68c03c00_81a6183c-f655-5266-b3dc-c02d40ea44a0_stupni_krow_ums_emoh_
local     dyv_f8887b67-51fa-4a48-8b19-699f68c03c00_81a6183c-f655-5266-b3dc-c02d40ea44a0_stuptuo_krow_ums_emoh_
local     dyv_fcb4c1ec-f8a4-4f62-a2dd-07b064e2c298_0fa85017-1dee-531e-9bf7-9c4dedd8eff9_ecapskrow_krow_ums_emoh_
local     dyv_fcb4c1ec-f8a4-4f62-a2dd-07b064e2c298_0fa85017-1dee-531e-9bf7-9c4dedd8eff9_erots-derahs_
local     dyv_fcb4c1ec-f8a4-4f62-a2dd-07b064e2c298_0fa85017-1dee-531e-9bf7-9c4dedd8eff9_stupni_krow_ums_emoh_
local     dyv_fcb4c1ec-f8a4-4f62-a2dd-07b064e2c298_0fa85017-1dee-531e-9bf7-9c4dedd8eff9_stuptuo_krow_ums_emoh_
local     dyv_fec04be2-97cf-4308-a475-c45a22b723b7_42ca839e-9a80-53c1-8e44-08a34a3e081c_ecapskrow_krow_nayvoj_emoh_
local     dyv_fec04be2-97cf-4308-a475-c45a22b723b7_42ca839e-9a80-53c1-8e44-08a34a3e081c_erots-derahs_
local     dyv_fec04be2-97cf-4308-a475-c45a22b723b7_42ca839e-9a80-53c1-8e44-08a34a3e081c_stupni_krow_nayvoj_emoh_
local     dyv_fec04be2-97cf-4308-a475-c45a22b723b7_42ca839e-9a80-53c1-8e44-08a34a3e081c_stuptuo_krow_nayvoj_emoh_

[GitHK]

Here is your issue: --volume /docker/volumes/:/docker/volumes/ give it a try with this. There is also a note in the README for it. You need to make sure that the the same paths as on the host disk are viewed from the container.

docker run --rm --pull always --volume /docker/volumes/:/docker/volumes/ --volume /var/run/docker.sock:/var/run/docker.sock itisfoundation/osparc-ops-dy-volumes-cleanup dyvc s3.amazonaws.com ACCESS SECRET BUCKET AWS --s3-region eu-central-1

[mrnicegyu11]

This solves it, merci