Speed up write? - Githubissues

catap commented 11 months ago

Proposed solution

Speed up write speed via network.

Right now beets writes the same file twice (tested on .flac but probably it's true for all format). The first time it had happened when files is moved into library. It works as expected via temporarry name. Anyway, when it edits tags, it reuploads the whole file with edited tags.

I see this behaviour quite well when upload heavy .flac into library via WebDAV by investigating .davfs.* files in the distination folder via SSH to the NAS.

Rework editing of tags from save full file, into append or truncate and replacing only tag part at the end of the file.

wisp3rwind commented 11 months ago

Right now beets writes the same file twice (tested on .flac but probably it's true for all format). The first time it had happened when files is moved into library. It works as expected via temporarry name. Anyway, when it edits tags, it reuploads the whole file with edited tags.

It is quite unclear to me which commands you're running here. Could you specify the actual beets commands where you run into this issue? I guess that would be required as a starting point to track down why exactly this happens.

catap commented 11 months ago

@wisp3rwind I simple import an album via very slow internet and started investigation how it works :)

sampsyo commented 11 months ago

Indeed; it would be great to know when/how these writes happen… is there any chance it's a consequence of your configuration or plugins you're using? Understanding a bit more about when the writes happen, and where they arise in the beets code, could help a lot.

catap commented 11 months ago

@sampsyo what information is required? list of plugins / configuration seems to be useless for now, because it means to check everything.

Maybe a kind of debug log which includes each possible write? Or... I may do it another way: copy file on NAS side and compare it O:-)

sampsyo commented 11 months ago

It's sort of an open-ended recommendation, to be honest! In general, to understand how to proceed here, we will need to understand where those two writes are happening on your machine. A few options could include:

try disabling plugins successively until it goes away; the last one disabled was the culprit
if disabling all plugins still exhibits the problem, similarly try turning off config options
if the problem still happens with a "vanilla" import, yeah, maybe something like strace or similar could reveal the I/O behavior

arogl commented 11 months ago

I see similar issues while importing and moving files on the same drive.

It seems beet imports the files, into the DB as temporary files in current location, does some work on some tags and then moves the files, updates the DB, if replaygain enabled, now on the moved files run that, then update tags and database. I am not sure if the RG update is direct or via temporary files in the same folder.

I'll try to do more testing later and provide some output

catap commented 11 months ago

@arogl this is almost the same things that I feel / saw.

@sampsyo / @wisp3rwind I'll try to investigate in this year

arogl commented 11 months ago

Evaluating 33 candidates.

I:\New_Music\import\2WEI\(2022) Mad World (1 items)
Sending event: import_task_before_choice
Sending event: before_choose_candidate

  Match (75.7%):
  2WEI, Tommee Profitt & Fleurie - Mad World
  ≠ artist
  MusicBrainz, Digital Media, 2022, XW, Position Music, None, None
  https://musicbrainz.org/release/9a87a277-5bb1-4f56-be96-e41dd3c1d9a0
  ≠ Artist: 2WEI -> 2WEI, Tommee Profitt & Fleurie
  * Album: Mad World
     * (#1) Mad World (4:15)
➜ [A]pply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort, eDit, edit Candidates? a
Sending event: import_task_choice
Sending event: import_task_apply
0 of 1 items replaced
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
fetchart: trying source bandcamp for album 2WEI, Tommee Profitt & Fleurie - Mad World
bandcamp: Not fetching art for a non-bandcamp album URL
fetchart: trying source filesystem for album 2WEI, Tommee Profitt & Fleurie - Mad World
fetchart: using well-named art file cover.jpg
fetchart: using local image I:\New_Music\import\2WEI\(2022) Mad World\cover.jpg
replaygain: analyzing 2WEI, Tommee Profitt & Fleurie - (2022) - Mad World
replaygain: analyzing 2WEI, Tommee Profitt & Fleurie - Mad World - Mad World
replaygain: executing ffmpeg -nostats -hide_banner -i I:\New_Music\import\2WEI\(2022) Mad World\01 - Mad World.mp3 -map a:0 -filter ebur128=peak=true -f null -
replaygain: 2WEI, Tommee Profitt & Fleurie - Mad World - Mad World: 2168 blocks over -18.7 LUFS
replaygain: 2WEI, Tommee Profitt & Fleurie - Mad World - Mad World: gain -10.0 LU, peak 1.3489628825916535
replaygain: 2WEI, Tommee Profitt & Fleurie - (2022) - Mad World: gain -10.0 LU, peak 1.3489628825916535
Sending event: database_change
replaygain: applied track gain -10.0 LU, peak 1.3489628825916535 of FS
Sending event: database_change
replaygain: applied album gain -10.0 LU, peak 1.3489628825916535 of FS
replaygain: done analyzing 2WEI, Tommee Profitt & Fleurie - Mad World - Mad World
lastgenre: added last.fm album genre (None):
Sending event: database_change
Sending event: database_change
Parsed query: AndQuery([NoneQuery('album_id', True)])
Parsed sort: NullSort()
Parsed query: AndQuery([SubstringQuery('albumtype', 'soundtrack', fast=True)])
Parsed sort: NullSort()
Parsed query: AndQuery([SubstringQuery('albumartist', '“Weird Al”', fast=True), AnyFieldQuery('Yankovic', ('artist', 'title', 'comments', 'album', 'albumartist', 'genre'), SubstringQuery)])
Parsed sort: NullSort()
Parsed query: AndQuery([SubstringQuery('albumartist', 'various artists', fast=True)])
Parsed sort: NullSort()
Parsed query: AndQuery([SubstringQuery('albumartist', 'various', fast=True)])
Parsed sort: NullSort()
Parsed query: AndQuery([BooleanQuery('comp', 1, fast=True)])
Parsed sort: NullSort()
Parsed query: AndQuery([SubstringQuery('ext', '.pdf', fast=False)])
Parsed sort: NullSort()
Parsed query: AndQuery([SubstringQuery('paired_ext', '.lrc', fast=False)])
Parsed sort: NullSort()
Parsed query: AndQuery([SubstringQuery('paired_ext', '.txt', fast=False)])
Parsed sort: NullSort()
Sending event: before_item_moved
importadded: Recorded mtime 1702539820.8176394 for item 'I:\Music\2\2WEI\(2022) Mad World\Mad World - 01 - 2WEI, Tommee Profitt & Fleurie - Mad World.mp3' imported from 'I:\New_Music\import\2WEI\(2022) Mad World\01 - Mad World.mp3'
Sending event: item_moved
Sending event: database_change
Sending event: database_change
Sending event: write
popularity: Mad World - 2WEI, Tommee Profitt & Fleurie not found
Sending event: after_write
importadded: Write of item 'I:\Music\2\2WEI\(2022) Mad World\Mad World - 01 - 2WEI, Tommee Profitt & Fleurie - Mad World.mp3', selected item.added=1702973066.8870106
Sending event: database_change
Sending event: import_task_files
Sending event: art_set
embedart: Embedding album art into 2WEI, Tommee Profitt & Fleurie - (2022) - Mad World
embedart: embedding I:\Music\2\2WEI\(2022) Mad World\cover.jpg
Sending event: write
popularity: Mad World - 2WEI, Tommee Profitt & Fleurie not found
Sending event: after_write
importadded: Write of item 'I:\Music\2\2WEI\(2022) Mad World\Mad World - 01 - 2WEI, Tommee Profitt & Fleurie - Mad World.mp3', selected item.added=1702973066.8870106
Sending event: database_change
Sending event: database_change
scrub: auto-scrubbing I:\Music\2\2WEI\(2022) Mad World\Mad World - 01 - 2WEI, Tommee Profitt & Fleurie - Mad World.mp3
scrub: writing new tags after scrub
Sending event: write
popularity: Mad World - 2WEI, Tommee Profitt & Fleurie not found
Sending event: after_write
importadded: Write of item 'I:\Music\2\2WEI\(2022) Mad World\Mad World - 01 - 2WEI, Tommee Profitt & Fleurie - Mad World.mp3', selected item.added=1702973066.8870106
scrub: restoring art
Sending event: album_imported
Sending event: database_change
importadded: Import of album 'Mad World', selected album.added=1702539820.8176394 from item file mtimes.

OK, verbose log from importing a 1 track album AFTER finding the matches. @catap I was wrong, it moves the file(s) and THEN runs all operations/updates/plugins on the moved file.

We need to see if we can move to a temporary location, perform all operations and then finally move to the library destination

catap commented 11 months ago

@arogl as I pointed in the beginning: edit of tags means save the whole file, not just edit / append at the end of it ;)

catap commented 11 months ago

Seems that this is behaviour of mutagen: https://github.com/quodlibet/mutagen/blob/main/mutagen/id3/_file.py#L223-L274

I feel a kind of dejavu. I have reported near the similar "issue" here or in mutagen or somewhere else BTW

beetbox / beets

Speed up write? #5029

Proposed solution