Every album re-copied on each update

loafofpiecrust commented 6 months ago

I just started using this plugin, and the first time I ran beets alt update ipod after setting things up worked just fine. It took all night, as expected. However, after I added a new album to my library and then ran beets alt update ipod again, it first printed + lines with the new tracks. Then it appeared to print all of my tracks and it looks like it's also going to take hours. I'm guessing it's re-copying over some files each time, which defeats the purpose of this for me coming from using a simple rsync script. Maybe I made a mistake at some point? Or maybe this is an interaction with the extrafiles plugin? Here's my config...

config.yaml

```yaml directory: /run/media/snead/icepack/music library: ~/music/beets-library.db original_date: yes format_album: $albumartist - [$original_year/$year] $album format_item: $artist - $album - $disc.$track. $title sort_album: albumartist+ year+ album+ per_disc_numbering: yes import: autotag: no copy: no timid: yes languages: en incremental: yes incremental_skip_later: yes match: preferred: countries: ['US', 'GB|UK', 'JP'] media: ['CD', 'DigitalMedia|File', 'Vinyl'] original_year: yes musicbrainz: extra_tags: [year] paths: albumtype:soundtrack: Soundtracks/[$realyear] $album/%if{$multidisc,Disc %right{$disc,1}/}$track. $title comp: %first{$genre}/[$realyear] $album/%if{$multidisc,Disc %right{$disc,1}%if{$grouping,. $grouping}/}$track. $title albumtype:single: %the{$albumartist}/Singles/%if{$subartist,[$subartist]}[$realyear] $album/$track. $title default: %the{$albumartist}/[$realyear]%if{$subartist,[$subartist]} $album%ifdef{atypes, $atypes}/%if{$multidisc,Disc %right{$disc,1}%if{$grouping,. $grouping}/}$track. $title item_fields: multidisc: 1 if disctotal > 1 else 0 # MF Doom is something special, with all of his alternate names subartist: albumartist_credit if albumartist.lower() != albumartist_credit.lower() else '' realyear: original_year if original_year != 0 and original_year < year else year plugins: the badfiles albumtypes missing rewrite fromfilename inline extrafiles fetchart alternatives convert duplicates # Automatically resample all hi-fi FLACs down to CD quality on import convert: auto: no embed: no max_bitrate: 1411 never_convert_lossy_files: yes format: flac formats: flac: ffmpeg -y -i $source -acodec flac -af aresample=osf=s16:dither_method=triangular:resampler=soxr:out_sample_rate=44100 $dest fetchart: auto: no cautious: yes cover_names: cover front folder album art minwidth: 1000 enforce_ratio: 2% sources: filesystem coverart itunes deinterlace: yes extrafiles: patterns: images: '*.(jpg|jpeg|png)' albumtypes: types: - ep: 'EP' - single: 'Single' - live: 'Live' - compilation: 'Comp' ignore_va: compilation bracket: '()' rewrite: albumartist DANGERDOOM: MF DOOM albumartist Viktor Vaughn: MF DOOM albumartist The Jimi Hendrix Experience: Jimi Hendrix albumartist Mychael Danna, DeVotchKa: DeVotchKa albumartist Adrian Orange: Thanksgiving albumartist The Velvet Underground & Nico: The Velvet Underground albumartist kidkanevil & Daisuke Tanabe: Daisuke Tanabe albumartist David Byrne & St. Vincent: David Byrne albumartist filous & Daði Freyr: Daði Freyr albumartist J\.U\.F: Gogol Bordello albumartist Gorillaz feat.*: Gorillaz albumartist Spacemonkeyz vs. Gorillaz: Gorillaz albumartist Justin Timberlake & .*: Justin Timberlake albumartist The Management: MGMT albumartist Urban Thermo Dynamics: Mos Def albumartist The Flaming Lips .*: The Flaming Lips albumartist Prince and The Revolution: Prince alternatives: ipod: directory: /run/media/snead/SHELBY_S IP/music-beets query: "onplayer:true" removable: true replace: '[\?]$': '' '[\\/]': "-" '^\.': _ '<\|': '◀' '\|>': '▶' '[\x00-\x1f]': _ ':': ';' '"': "'" '[<>\?\*\|]': _ '\.$': '' '\s+$': '' '^\s+': '' '^-': _ '‐': '-' duplicates: album: yes keys: [albumartist, album, year] ```

geigerzaehler commented 6 months ago

Hi @loafofpiecrust, thanks for reporting this. The config looks good to me and the second update should not try to copy the files again. You’re probably right that one of the plugins (or maybe some custom configuration) is causing the problem.

Could you try to isolate the issue by disabling plugins and removing custom configuration? Then I can try to identify the problem with a simpler configuration.

loafofpiecrust commented 6 months ago

Thanks for the quick follow-up! I removed most of my plugins, reduced the config a little. I tested on my local BTRFS filesystem and things worked as expected. Then I tested again with my iPod which has a FAT32 filesystem and the issue recurred, but only with certain files. At first I added some FLACs and it worked fine. Then I added a few more albums and the issue happened with those files (MP3 and FLAC). Must be related to FAT32 filesystem.

loafofpiecrust commented 6 months ago

Okay sorry I forgot something. It seems like I made a mistake. The issue is that when I set my onplayer=true tag I used -W to avoid writing the tag to disk. This is what caused those specific files to be copied over during every single update. I just tested removing that tag entirely, then re-writing it without -W and I got the right behavior with the same files. I didn't want this tag in my files, but so it goes! This might be worth adding to the docs for this plugin though. Interestingly, using -W was still only an issue on FAT32.

EDIT: I've since tested with my whole library again after letting it write onplayer to the files, and still running into the same issue.

geigerzaehler commented 6 months ago

Thanks for the update. I think I’ve found the problem and it has to do with modification times on FAT32 only having second resolution. I’ve pushed a patch and you can try it out to see if that fixes your issue:

pip install --user git+https://github.com/geigerzaehler/beets-alternatives.git@fat32-mtime

geigerzaehler commented 6 months ago

@loafofpiecrust, did you have a chance to try out my branch and see if it fixes the problem?

loafofpiecrust commented 6 months ago

I figured out how to build your branch to include it in my beets build, and tested it out. It does not appear to fix my issue, and it looks like your code change would avoid unnecessary WRITE actions. Actually, I'm getting unnecessary ADD actions on the same files every time I run beet alt update ipod. The destination files have a brand new mtime way later than the source mtime, yet it keeps copying them over. It must have something to do with the destination path itself?

loafofpiecrust commented 6 months ago

@geigerzaehler There doesn't seem to be a pattern to which files are getting copied every time. Some are whole albums, some individual tracks. Perhaps more non-alphanumeric characters? I'm not sure. I do have several extra replace entries in my config to replace problematic characters... Including a few here in case that's helpful.

+/run/media/snead/SHELBY_S IP/Music/4 Non Blondes/[1992] Bigger, Better, Faster, More!/03. What's Up¿.flac
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2007] Collaborations (Comp)/01. Ayo Technology.mp3
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2003] Justin & Christina (EP)(Comp)/01. That's What Love Can Do.mp3
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2003] Justin & Christina (EP)(Comp)/03. Beautiful (Valentin club mix).mp3
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2003] Justin & Christina (EP)(Comp)/05. Fighter (Hellraiser remix).mp3
+/run/media/snead/SHELBY_S IP/Music/Anamanaguchi/[2006] Power Supply (EP)/06. Flora⁄Fauna.mp3
+/run/media/snead/SHELBY_S IP/Music/Anderson .Paak/[2016] Malibu/04. The Season ⁄ Carry Me.mp3
+/run/media/snead/SHELBY_S IP/Music/Nujabes/[2003][Various Artists] Hydeout Productions; First Collection (Comp)/03. Strive.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/01. The Narrator.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/02. Tidal Wave.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/03. High Tide.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/04. Green Machine.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/05. Winter Must Be Cold.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/06. She's Just Like Me ⁄ Taking Time.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/07. Glowworm.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/08. Dots 1-2-3.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/09. Lucky Charm.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/10. Innerspace.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/11. Show the World.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/12. Love You Alice ⁄ D.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/13. Pine Away.mp3
+/run/media/snead/SHELBY_S IP/Music/Arcade Fire/[2007] Neon Bible/05. Black Wave⁄Bad Vibrations.flac
+/run/media/snead/SHELBY_S IP/Music/Arctic Monkeys/[2013] AM/01. Do I Wanna Know¿.mp3
+/run/media/snead/SHELBY_S IP/Music/Arctic Monkeys/[2013] AM/02. R U Mine¿.mp3
+/run/media/snead/SHELBY_S IP/Music/Arctic Monkeys/[2013] AM/09. Why'd You Only Call Me When You're High¿.mp3
+/run/media/snead/SHELBY_S IP/Music/Grizzly Bear/[2007] Friend (EP)/09. Knife.mp3

loafofpiecrust commented 6 months ago

I had to make a few more changes to use existing untracked destination files imported by my rsync script on this branch (EDIT: I also did remove art embedding since I use cover files). It seems to work fine now with yours and my changes together, updating metadata on my destination when necessary. Much more robust to manual renames/moves. As an aside, it does seem quite slow to finish when there are zero updates needed (4 minutes on my library!).

geigerzaehler commented 4 months ago

it looks like your code change would avoid unnecessary WRITE actions. Actually, I'm getting unnecessary ADD actions on the same files every time I run beet alt update ipod.

Yes, you’re right! I didn’t realize that the problem was that files were added.

It seems to work fine now with yours and my changes together, updating metadata on my destination when necessary.

I took a look at your branch and I couldn’t quite figure out what the code you changed does. But to me it looks like it breaks things because we’re effectively doing self.set_path(item, self.destination(item)) which mixes library paths with paths in the alt collection.

I think it works for you at the moment because your basically skipping adding a file when the path is not found which seems to suggest that the paths don’t actually exist or os.path.lexists(syspath(path)) behaves strangely on fat32.

One thing you could do is debug the code in matched_item_actions to figure out why the files do not exist.

wisp3rwind commented 2 months ago

I haven't been following this issue up to now; and I'm not sure that I understand correctly which steps are required to trigger the extraneous ADD actions. Maybe you could clarify in terms of the actual beet commands? From what I gather, it should be something like:

configure an alternative ipod on a FAT32 filesystem (required, other filesystems don't exhibit the issue) which includes tracks according to the onplayer:true query
beet modify -W <some query> onplayer=true (is the -W required? you used it initially, but in the end, the issue occurred without, right?)
beet alt update ipod
beet import <some new music>
beet modify -W <query for new music> onplayer=true
beet alt update ipod (should add only the new music, but in fact also adds the old tracks again)

Some questions to ponder:

Does the same issue occur on a plain FAT32 thumb drive (or a FAT32 loop device) (i.e. is there a chance that the iPod does something weird with mtimes or file name normalization? Although mtimes shouldn't be the issue, since they would lead to WRITEs.)
Could we add a FAT32 test case to CI by creating a FAT32 partition on a loop device and running the tests there?
Could you repeat the steps and report the files' mtimes after each of them?
As @geigerzaehler said:

One thing you could do is debug the code in matched_item_actions to figure out why the files do not exist.

geigerzaehler / beets-alternatives

Every album re-copied on each update #74