Provider mappings stuck with `available = 0` in the database

[chatziko]

What version of Music Assistant has the issue?

2.1.0

What version of the Home Assistant Integration have you got installed?

2024.6.2

Have you tried everything in the Troubleshooting FAQ and reviewed the Open and Closed Issues and Discussions to resolve this yourself?

• [X] Yes

The problem

For the past two days, 1/3 of the artists in my spotify library have become "unavailable", meaning that they appear grey in the artist page and they are not clickable. After some digging I found that both:

• the available json field in the artists.provider_mappings column, and
• the provider_mappings.available column

are 0 for these artists. Disabling and enabling the spotify provider does not solve the issue.

What's interesting, looking at the code, is that the available field of the ProviderMapping dataclass seems to be modified only by ProviderMapping::__post_init__:

    def __post_init__(self) -> None:
        ...
        if not available_providers.intersection({self.provider_domain, self.provider_instance}):
            self.available = False

Moreover, the code only changes the flag from True to False if theprovider itself is not avaible, but never the opposite! If for any reason it becomes False in the database, the code will never reset it to True (unless I miss something, I'm not really familiar with the code).

How to reproduce

I cannot reproduce the exact problem where 1/3 of the artists are unavailable, but I can reproduce a similar issue that clearly shows the problem. The following are run in my development environment (server running in vscode container).

• Delete the ~/.musicassistant dir to start fresh
• Start the server and add the spotify provider
• Check that the db entries are saved and the available flags are 1 by running the following query in sqlite3:

  SELECT name, json_extract(a.provider_mappings,'$[0].available'), pm.available
  FROM artists a LEFT JOIN provider_mappings pm ON pm.item_id = a.item_id AND media_type = 'artist'
  ORDER BY name ;

• Disable (witout deleting) the spotify provider
• What until the artists appear grey (disabled) in the UI, and run the same query above. The flags should still be 1 although the entries appear disabled. That is, the flag in the db is 1, but in memory the flag is 0 (changed by ProviderMapping::__post_init__).
• Now restart the server. On restart the MetaDataController::metadata_scanner operation is executed, which updates the artist entry in the database, so the 0 flag which up to now was in memory is now written in the database.
• Rerun the query to verify that the flags in the db are now 0.
• Re-enable the spotify provider
• Although we expect the artists to appear enabled again, they stay disabled. The reason seems to be that the flag is still 0 in the db, and ProviderMapping::__post_init__ only changes it from 1 to 0, never from 0 to 1.

Music Providers

Only tried with spotify.

Player Providers

The issue is only about the library, the players are not concerned.

Full log output

No response

Additional information

As a workaround (hacky, use at your own risk), the available flags can be restored to 1 with the following queries:

update artists   set provider_mappings = json_replace(provider_mappings, '$[0].available',json('true')) ;
update albums    set provider_mappings = json_replace(provider_mappings, '$[0].available',json('true')) ;
update playlists set provider_mappings = json_replace(provider_mappings, '$[0].available',json('true')) ;
update provider_mappings set available = 1;

What version of Home Assistant Core are your running

2024.7.3

What type of installation are you running?

Home Assistant OS

On what type of hardware are you running?

Generic x86-64 (e.g. Intel NUC)

[OzGav]

@marcelveldt please see above

[marcelveldt]

Can you recheck it with commenting out the cleanup logic ? Because we need to find what is causing the provider to be disabled

[chatziko]

I updated the description after some more tries, the cleanup has nothing to do with the issue, the MetaDataController::metadata_scanner is the operation that triggers writing the 0 flag in the database.

Clarification: the provider is disabled by me (just to reproduce the issue). The problem is that when a provider is disabled the code does not update the available flag in the database, the flag stays 1 in the db but is 0 in memory (modified by ProviderMapping::__post_init__). This inconsistency between memory and the db is a time-bomb, cause an unrelated operation (metadata scanner) can later trigger an update and the flag becomes 0 in the db.

Finally when it becomes 0 in the db it will stay 0 forever, even if I re-enable the provider, because ProviderMapping::__post_init__ only changes it from 1 to 0, never from 0 to 1.

IMHO if available is a global flag of the provider, not a per-entry flag, it shouldn't be in the db entries at all, and only set to 0 or 1 by ProviderMapping::__post_init__ .

[marcelveldt]

available is NOT a global flag but an indication if the item is available - streaming providers are annoying and sometimes switch their content due to licensing and that can make an item to become unavailable, which we resolve with track linking.

Anyways, setting the available flag from provider availability on the item's available flag is a hack. It is just for api consumers and should not merge with internal logic, hence if we move it to the serialize logic it will be fixed.

BTW: I now understand why you are the only one reporting this so far. You manually disabled the provider.

Thanks for the insight. I'll provide a fix soon (together with the custom client id)

[chatziko]

BTW: I now understand why you are the only one reporting this so far. You manually disabled the provider.

Well, I manually disabled the provider in my development setup trying to reproduce the problem.

In my real setup I'm still not sure what happened. Maybe I also disabled the provider there (not sure, I was experimenting with lots of things). But even if I did, it's not clear why only 1/3 of the artists appear disabled and not all of them.

In any case let's fix the case we understand, the other will hopefully be related.

[marcelveldt]

Forgot to let you know that I patched this issue.

[chatziko]

Thanks! I tried the same procedure again, the problem is partially fixed but not fully. Here's what happens now:

• I add spotify on a fresh install. Intially all artists are available and both artists.provider_mappings[0].available (json) and provider_mappings.available are 1.
• Now I disable spotify, the artists are shown as unavailalbe (grey) but in the db both flags are still 1.
• I restart MA, now only for 29 (out of 41) artists, artists.a.provider_mappings[0].available becomes 0, while provider_mappings.available is 1 (and for the rest of the artists both db flags are 1).
• I re-enable spotify, the db flags stay the same, and the artists are still shown as unavailable (and in fact, at some point the number of unavailable artists became 30 from 29, not sure when exactly).

So it's actually more similar now to what happened in my production setup in which only part of the artists became unavailable.

I'll try to pinpoint exactly the code that changes the db flags at startup.

[OzGav]

I am assuming this is what I am seeing since Spotify came back online. I have a album in my Spotify library that is now greyed out in MA and I can't do anything with it?

[chatziko]

Run in sqlite3:

SELECT name, json_extract(a.provider_mappings,'$[0].available'), pm.available
 FROM albums a LEFT JOIN provider_mappings pm ON pm.item_id = a.item_id AND media_type = 'artist'
 ORDER BY name ;

If you see a 0 flag that's probably it.

You can manually restore the flags as a workaround (see the queries at the end of the description).

[chatziko]

I found what's happening, the weirdest thing was the inconsistency between the artists.provider_mappings json field and the provider_mappings table, these two should be kept in sync!

I guess the intention of __post_serialize__ was to set available = false only when sending the data to the frontend. But __post_serialize__ is also called when updating the json column in:

        await self.mass.music.database.update(
            self.db_table,
            {"item_id": db_id},
            {"provider_mappings": serialize_to_json(provider_mappings)},
        )

cause serialize_to_json calls to_dict.

So the available flag becomes false in the json column (and can never change to true again), although it's still true in the provider_mappings table.

[chatziko]

Just wanted to report that after upgrading to 2.1.1 exactly one artist in my library became unavailalbe. So the issue can happen without manually disabling the provider. Maybe it temporarily becomes disabled while loading and metadata_scanner might run at the same time?

Moreover for that single artist, artists.a.provider_mappings[0].available became 0 while provider_mappings.available is still 1, so this must be caused by the __post_serialize__ issue described above. Fixing this will likely fix the problem, even though we don't fully understand why the provider becomes disabled.

[agdamsbo]

I believe I have this same issue. I only have Spotify as provider on 2.1.1. Everything is greyed out and unavailable.

What Ive gathered is that if I remove an artist from the library and re-adds, the albums/tracks for that given artist is available again. I have tried removing Spotify and have tried adding a ClientID with no effect. Also clearing cache and restarting HA.

[marcelveldt]

this one is fixed a while ago

[chatziko]

this one is fixed a while ago

You mean the initial fix in 2.1.1 or another one later?

The initial fix had the issue that __post_serialize__ is called when updating the json column (see the comments above).

[marcelveldt]

Hmmmm but why does your provider become unavailable?