mutagen-io / mutagen

Fast file synchronization and network forwarding for remote development
https://mutagen.io
Other
3.47k stars 153 forks source link

Max staging file size does not work as expected #502

Open Pigrenok opened 5 months ago

Pigrenok commented 5 months ago

If you use --max-staging-file-size=<size> you would expect that files larger than the given limit will not be synced.

Yet, I set 10 kB limit for the session (see below the mutagen sync list -l output), and yet it try to syncronise over 800 MB file.

--------------------------------------------------------------------------------
Name: test-split
Identifier: sync_tDEywqaNnSwijl8m0BTTWxn2xfxi9tsfVj5KLv755y3
Configuration:
    Synchronization mode: Two Way Safe
    Hashing algorithm: Default (SHA-1)
    Maximum allowed entry count: Default (2⁶⁴−1)
    Maximum staging file size: 10000 (10 kB)
    Symbolic link mode: Default (Portable)
    Ignore VCS mode: Default (Propagate)
    Ignores: None
    Permissions mode: Default (Portable)
Alpha:
    URL: /home/pigrenok/YandexDisk/QueenMary/Bioinformatics/ONT/test_split
    Configuration:
        Watch mode: Default (Portable)
        Watch polling interval: Default (10 seconds)
        Probe mode: Default (Probe)
        Scan mode: Default (Accelerated)
        Stage mode: Default (Mutagen Data Directory)
        File mode: Default (0600)
        Directory mode: Default (0700)
        Default file/directory owner: Default
        Default file/directory group: Default
    Connected: Yes
    Synchronizable contents:
        1 directory
        19 files (52 kB)
        0 symbolic links
    Transition problems:
        basecalls.bam: unable to swap file: unable to set staged file permissions: unable to set permission bits: chmod /home/pigrenok/.mutagen/staging/sync_tDEywqaNnSwijl8m0BTTWxn2xfxi9tsfVj5KLv755y3-alpha/a3/a318044517dcb210de625ce510a7d7a877251fdd2c2b98be3af4b85c040455245ae37a93: no such file or directory
        basecalls.bam.bai: unable to create file: unable to set staged file permissions: unable to set permission bits: chmod /home/pigrenok/.mutagen/staging/sync_tDEywqaNnSwijl8m0BTTWxn2xfxi9tsfVj5KLv755y3-alpha/31/313da2412c24a065816f72c054a3b96555a30f79ad56dc98edb105cab72e5e4b7f355e9a: no such file or directory
        report.PycoQC.html: unable to create file: unable to set staged file permissions: unable to set permission bits: chmod /home/pigrenok/.mutagen/staging/sync_tDEywqaNnSwijl8m0BTTWxn2xfxi9tsfVj5KLv755y3-alpha/e7/e70abea6c337802f43edb507242f0a9c8409af38401efbb1304489d15a2e3ee000ad1853: no such file or directory
Beta:
    URL: apocrita:/data/scratch/bty316/Turtles_test
    Configuration:
        Watch mode: Default (Portable)
        Watch polling interval: Default (10 seconds)
        Probe mode: Default (Probe)
        Scan mode: Default (Accelerated)
        Stage mode: Default (Mutagen Data Directory)
        File mode: Default (0600)
        Directory mode: Default (0700)
        Default file/directory owner: Default
        Default file/directory group: Default
        Compression: Default (DEFLATE)
    Connected: Yes
    Synchronizable contents:
        1 directory
        21 files (887 MB)
        0 symbolic links
Status: Staging files on alpha
Staging progress: 2/3 - 645 MB - 67%
Current file: basecalls.bam (639 MB/881 MB)
--------------------------------------------------------------------------------

Do I misunderstand the purpose of this option or is it not working correctly? It is binary file and any touching of it (even without any real changes) triggers full transfer of the file. If this option is not working, is there another reliable way to limit what files/dirs are being synchronized and which are not. I understand that scripts can be moved somewhere else, but that is not preferred way of doing in some cases.

I am aware of the issue #391 but that talks specifically about pre-rejection. I do not mind if first 10 kB (or whatever the limit) is transferred before the file is rejected for transfer, but it does not seem to work that way either.

Thanks.

jd7h commented 47 minutes ago

I have the same problem! maxStagingFileSize is set to 1 MB, but Mutagen is happily syncing large files according to mutagen sync monitor.