LMS's rescan operation seems unreliable processing metadata changes and detecting changed files

[audiomuze]

I've been doing a lot of work of late cleaning up metadata across my library to ensure artist names are consistent and avoid two instances of an artist because of variations in their name, case differences etc.

I make changes to a database table then write the changes back to the underlying files. I've been doing this for years, and I know the import/export code works, inspecting tags verifies this.

Tonight I removed 'Bruce Springsteen' as an albumartist from all albums where the albumartist includes 'Bruce Springsteen & the E Street Band'. I wrote the changes back to the underlying files and exported the file list to a text file which is then processed by a bash script which adds 1 second to the last modified date of every file so that LMS picks up the changes.

#!/bin/bash

filelist=./affected_files.csv

echo "Files to process:  " $(wc -l $filelist)
while [ -s $filelist ] ; do

    file=$(head -1 $filelist)
    touch -c -m -r "$file" -d '+1 second' "$file"

    # remove current file from list
    sed -i.prev '1d' $filelist
done

This should all just work, except LMS picked up only 19% of the changed files on a rescan.

In addition, I've also noticed that LMS doesn't get rid of orphans on recanning - so every one of those rescanned albums will still have Bruce Springsteen appearing as albumartist on those albums because (I presume) the scan logic doesn't keep track of and delete old contributor associations when it's rescanned a file.

@mherger, have you come across this failure to detect changes before?

Also, I'm not sure how the rescan logic works, but would it not be worthwhile writing all data related to changed files to staging tables then deleting the associated records from the main tables and updating the main tables from the staging tables? Whatever is presently done leaves orphaned associations that basically require a trashing of the database to dispense with them.

[michaelherger]

See https://forums.slimdevices.com/forum/user-forums/logitech-media-server/1666272-is-lms-s-rescan-operation-reliable-at-detecting-metadata-changes-and-changed-files.

[schnillerman]

I can confirm in a different way: I removed certain tags from ~3.000 files completely, did a rescan (scanner picked up the correct amount of changed files), but the information stored in the tags was still available in the LMS library after the scan.

I restart my LMS docker every day.

Lyrion Music Server Version: 9.0.0 - 1715811455 @ Thu May 16 02:29:13 CEST 2024 Hostname: DISKSTATION01 IP-Adresse des Servers: 192.168.1.11 Server-HTTP-Portnummer: 9000 Betriebssystem: Debian (Docker) - DE - utf8 Plattformarchitektur: x86_64-linux Datenbankversion: SQLite

Perl-Version: 5.32.1 - x86_64-linux-gnu-thread-multi Audio::Scan: 1.06 DBD::SQLite: 1.58 (sqlite 3.22.0) IO::Socket::SSL: 2.069 Mozilla::CA: Net::SSLeay: 1.88 - OpenSSL 1.1.1w 11 Sep 2023

[michaelherger]

Can you give more details please? Like what tags you changed?

[schnillerman]

Among others:

• provider
• work
• endcodersettings
• encoded-by
• comment
• copyright

[michaelherger]

Most of those tags aren't even read by LMS. Where would you see provider, copyright, etc.?

[schnillerman]

So it seems to be the work tag at least? Other than that I can just say that a random word stored in these tags yields search results after a rescan even if the tags are deleted.

Also, today, I had a classic error: I corrected albumartist, artist and album tags because of wrong capitalization: from "Nypc" to "NYPC" in all 3 tags, the album is titled like the artist which is also the albumartist.

A rescan caused the album title to be correct ("NYPC"), but the artist still read "Nypc" although there is no other files by this artist except the corrected ones. Also, the album art was suddenly missing.

Moving the album directory out of the library, rescan, and moving it back inside, rescan again, fixed it.

[michaelherger]

The casing I thought we had fixed a while back. But that probably was for albums only...

About the other tags: the fact that you list tags which aren't supported by LMS makes me wonder how you figure out that those tags would not be picked up? Where do you even see them?

[schnillerman]

If I click on "More Info > Show Tags"

Unfortunately, I already cleaned them all and did a complete rescan but tag WORK was definitely listed and also searchable.

[mherger]

Well, then this has nothing to do with the scanner: the data shown there is the tags as they are stored in the file. Please double check them in eg. mp3tag. Sometimes there are multiple versions of ID3 tags stored in the same file, which can lead to this confusing kind of situation.