Upgrading foobar2000 from 1.x to 2.x ruins lyrics saved in tag for all songs that are in playlists during the upgrade

[ZYinMD]

When upgrading from foobar2000 v1.x to v2.x, lyrics stored in tags of all songs in all playlists are turned into . (see screenshot) even though the files on disk aren't actually modified and the data in tag is unchanged.

Steps to reproduce

1. Download foobar2000 latest 1.x installer (1.6.17), run it, choose "portable installation". (it's probably the 32bit version, I didn't find x64 version for 1.x)
2. Launch portable v1.6.17, install OpenLyrics 1.8.
3. Add some songs into some playlists. Play those songs, let OpenLyrics download lyrics and persist them in tag. (by default, the UNSYNCED LYRICS tag is used, can also customize the tag name)
4. Exit
5. Backup the portable folder by making a copy of it
6. Download installer of latest foobar2000 (v2.1.2). Either 32 or 64 bit, both can reproduce.
7. Run new installer. Choose "Portable Installation". Install on top of the old version (by choosing the same location)
8. Now the portable foobar2000 has been upgraded to v2.1.2
9. Launch. settings, UI layouts, and playlists are preserved. (but if you upgraded from x86 to x64, OpenLyrics will become uninstalled)
10. Check the playlists created in step 3. Realize all lyric tags of those songs have turned into .
11. If OpenLyrics became uninstalled, install it again.
12. Play those songs. Realize the lyric tags are still . and the lyric panel also shows . as lyric content.
13. Find the backup folder created in step 5 containing 1.6.17 , launch it. (you can run the 2 instances side by side), realize the tags aren't actually modified, they still work totally fine in the 1.6.17 instance.

What can fix the tags in the 2.1.2 instance:

1. right click song > properties > Tools > Reload info
2. If you locate the problematic song on disk, make a copy of the file, and drag the copy into the 2.1.2 instance, the copy has correct tag, meanwhile the original doesn't.
3. If you drag in other songs that weren't in a playlist during the upgrade, their lyric tags are fine (if they already had the tag to begin with)

What doesn't fix the tags in the 2.1.2 instance:

If you remove a problematic song from the playlist, then add it back, the tag won't be fixed.

Expected behavior

1. Upgrading fb2k from 1.x to 2.x shouldn't turn lyric tag into .
2. If fb2k has somehow turned lyric tag into ., OpenLyrics should still be able to parse it

Versions

foobar2000 version: 1.6.17 (can only find x86) upgrading into 2.1.2 (either x86 or x64 will reproduce) OpenLyrics version: 1.8

Debug logs

First launch after upgrade (without OpenLyrics installed): nothing notable in log

Relaunch after installing OpenLyrics: nothing notable in log

When playing a song with . as lyric:

Opening track for playback: "C:\Users\Zhi\Desktop\Heart Of Glass - Blondie.mp3"
INFO-OpenLyrics: LyricPanel::compute_background_image took 1461us
INFO-OpenLyrics: No search avoidance info available for track
INFO-OpenLyrics: Searching for lyrics for artist='Blondie', album='Parallel Lines', title='Heart Of Glass'...
INFO-OpenLyrics: Searching for lyrics in tag: 'LYRICS'
INFO-OpenLyrics: Found lyrics in tag: 'LYRICS'
INFO-OpenLyrics: Searching for lyrics in tag: 'SYNCEDLYRICS'
INFO-OpenLyrics: Searching for lyrics in tag: 'UNSYNCEDLYRICS'
INFO-OpenLyrics: Searching for lyrics in tag: 'UNSYNCED LYRICS'
INFO-OpenLyrics: Successfully retrieved lyrics from source: Metadata tags
INFO-OpenLyrics: Parsing lyrics text...
INFO-OpenLyrics: Loaded lyrics already form a valid UTF-8 sequence
INFO-OpenLyrics: Parsing LRC lyric text...
INFO-OpenLyrics: Lyric loading complete
INFO-OpenLyrics: Skipping lyric save. Type: 1, Local: yes, Timestamped: no, Autosave: 1
Device: Speakers (Sound Blaster Play! 3)
Mix format: 48000 Hz / 32-bit float / 2 channels (0x3)
Sending stream: 48000 Hz / 32-bit float / 2 channels (0x3)

Things seem normal, but Lyric panel just shows .

[marc2k3]

Custom components should remain in place if you installed them correctly.

https://wiki.hydrogenaud.io/index.php?title=Foobar2000:How_to_install_a_component

If yours got removed, I suspect you might have installed them manually in the main fb2k folder\components. If that's incorrect and you think you've done nothing wrong, report it to the fb2k dev on hydrogenaud.io

As for this metadata issue, not sure what's going on there. People can try force reloading all tags by using the properties dialog>Tools button>Reload info.

[ZYinMD]

I'm pretty sure with the 64 bit version of foobar2000, custom components are deleted when upgrading from version 1.x to 2.x. In my test, a clean new portable 1.6.17 → install OpenLyrics → upgrade to 2.1.2 → OpenLyrics are gone.

32bit may be different, but I've always been on 64bit The reason seems to be upgrading from 32 to 64 bit, and I don't know how to test otherwise, because https://www.foobar2000.org/old only has x86 for v1.x, and x64 for v2.x.

[marc2k3]

Probably not deleted but ignored. 32bit uses a folder named user-components inside the profile folder of a portable install. 64bit fb2k uses user-components-x64.

And you obviously can't move the components. They are completely incompatible You need to grab all the updated 64bit versions -if they even exist - yourself.

[ZYinMD]

@marc2k3 thanks for the info, yes you're right, they're ignored. I was mistaken in my in my previous comment, now fixed.

But issue in OP stays the same, with clear steps to reproduce. I also tested what would happen if I stay 32bit, result is the same.

Updated OP to make it more explicit.

[marc2k3]

Did you try my suggestion from my original post?

People can try force reloading all tags by using the properties dialog>Tools button>Reload info.

[jacquesh]

It's not clear that this is actually an openlyrics issue and not a fb2k issue. Openlyrics just reads the tags as reported by fb2k. If fb2k is reporting no lyrics (or just '.') in its own UI, then that's what openlyrics is going to display.

If you can confirm that fb2k shows the correct lyrics in the tag between steps 3 & 4, but shows '.' at step 10 (as you did in your original screenshot) then the problem is with fb2k and you should ask on the hydrogenaudio forums as @marc2k3 suggested (thanks for the help btw!). You can/should actually test this without getting openlyrics involved at all: You can just edit the tags directly in the Properties window shown in your screenshot.

Since this is about saving and loading tag values, component installation shouldn't make any difference. You should be able to verify the existence of lyrics (or the lack thereof) using just fb2k and/or some 3rd party program like kid3 or VLC or whatever.

On the other hand if fb2k shows the correct tag value in the Properties, but openlyrics isn't loading them correctly, then that would probably be an openlyrics issue.

[ZYinMD]

@marc2k3 Yes "Reload info" does work. I added it as the first item in the "what fixes the issue" in OP, but didn't notify you. Thanks

@jacquesh thanks for the super insightful comment. I tested manually created tags, without involving OpenLyrics, and had very interesting findings.

I first created these random tags:

Then did the upgrade. Here's what happened after the upgrade:

So the 2 tags LYRICS and UNSYNED LYRICS got special treatments! Mind blowing...

I will file report on hydrogenaud.io, but first I want to see if you have more comments about my findings.

I also want to ask you about this:

Here, you mentioned that UNSYNED LYRICS are strongly recommended as the tag to store lyrics, even for synced ones. Did you say that because UNSYNED LYRICS is the only one in this table? Do you think this table is still up-to-date?

[marc2k3]

There is some history about period characters being used for lyrics in foobar2000 1.x. Before 1.3.0, fb2k would load all metadata from files and these would be cached in the library/playlist files for faster performance. But with larger collections and what are consider to be oversized tags, this would lead to increased memory usage and would become unusable for some.

So 1.3.0 introduced a large change and would not not cache them and they would be unavailable via standard title formatting functions. The period was used to represent that the tag existed but more work had to be done to actually display it in full.

The 1.3 SDK contained new methods which component devlopers were encouraged to implement where oversized tags would be displayed in full. I'm not searching but this component must implement these.

If people relied on old components that were never going to updated with newer APIs in the 1.3 SDK, they could edit the content of a file named LargeFieldsConfig.txt inside their profile folder. This would revert to old behaviour. If you check the contents of that file on a 1.6.x install, it will show the tag/size limits.

foobar2000 v2 uses a complete new sqlite backend for caching and oversized tags are once again displayed in full and those APIs for 1.3.x and later are no longer necessary. But obviously they continue to work and are necessary if you're supporting 1.6.x and 2.0+ at the same time.

[jacquesh]

@marc2k3 oh that's interesting, I didn't know about LargeFieldsConfig.txt. Awesome, thanks! In general openlyrics tries to use the new API and falls back to the older API that loads all fields for exactly this reason.

@ZYinMD Correct, UNSYNCED LYRICS is strongly recommended because in mp3 files its the only one that gets stored in a dedicated frame. Some other media players will only look for that USLT frame, meaning that if you stored your lyrics elsewhere they would not be detected. I assume that table is indeed up-to-date.

[ZYinMD]

@jacquesh thanks, so, the 2 things below are both true, but they're apples and oranges, unrelated to each other, right?

1. UNSYNCED LYRICS is stored in a dedicated frame, while LYRICS is not
2. both UNSYNCED LYRICS and LYRICS are somehow turned into . , but the other tags are not (for instance SYNCED LYRICS)

[jacquesh]

I'm much less sure about the 2nd point, but yes it would seem so.

[marc2k3]

but the other tags are not (for instance SYNCED LYRICS)

Well that's no mystery. I did say to check the contents of LargeFieldsConfig.txt.

# Any changes in this file will take effect on next foobar2000 startup.
# Note, size limit values are in UTF-8 bytes, not characters!
# Note, field names can be enclosed in quotation marks to include trailing spaces etc.
# Size limit for meta fields that appear in the basic fields list
basicMetaMax=2000
# Size limit for meta fields that do not appear in either list
defaultMetaMax=1000
# Size limit for tech info fields
infoMax=200

# List of basic meta fields - essential info such as artist & title
fieldBasic=album
fieldBasic=album artist
fieldBasic=artist
fieldBasic=composer
fieldBasic=conductor
fieldBasic=date
fieldBasic=discnumber
fieldBasic=genre
fieldBasic=keywords
fieldBasic=performer
fieldBasic=producer
fieldBasic=style
fieldBasic=title
fieldBasic=totaldiscs
fieldBasic=totaltracks
fieldBasic=tracknumber

# List of spam meta fields - rarely useful stuff that *never* gets cached
fieldSpam=accurate rip
fieldSpam=aucdtect
fieldSpam=biography
fieldSpam=cuesheet
fieldSpam=eac logfile
fieldSpam=itunes_cddb_1
fieldSpam=itunmovi
fieldSpam=log
fieldSpam=logfile
fieldSpam=lyrics
fieldSpam=unsynced lyrics

The last 2 are lyrics related. Any other variant gets a free pass.