Tidal playlist are not loaded in Mopidy unless refreshed manually

[fmarzocca]

I have just installed this plugin and authorized. It is working (in Iris I can Browse my Tidal).

Now, I have a program which lists all Mopidy's playlists (from Mopidy API, with method "core.playlists.as_list") but Tidal's playlists are not there. What should I do? Thanks

[tehkillerbee]

Good to hear it is working for you. For Tidal playlists, you should look in "Tidal | My Playlists" and it should display all the playlists you have added.

[fmarzocca]

For Tidal playlists, you should look in "Tidal | My Playlists" and it should display all the playlists you have added.

Yes, but that's not my point. I am using NodeRed to manage Mopidy, and when I ask to list all my playlists (with method "core.playlists.as_list"), it returns all the playlists (Local, Spotify) expect those from Tidal. Why Mopidy doesn't list Tidal's playlists?

[fmarzocca]

Sorry, fixed by restarting mopidy!

[fmarzocca]

Sorry, but I have to re-open this. I have set several playlists in my Tidal account. They are immediately showing up in Iris, but they are not listed from API method core.playlists.as_list until after a couple of hours or more… In other words, managing directly from Mopidy, it looks like the new Tidal playlists are very slow to get shown in the list.

[tehkillerbee]

@fmarzocca Interesting.. As far as I know, mopidy clients including Iris use a similar way to get the playlists.. Does this happen with any playlist or only specific playlist (eg. large playlists).

Can you describe the basic steps needed to replicate this issue?

[fmarzocca]

Hi Johannes, thanks for answering. This is how to replicate the issue:

• From Tidal GUI select and add a new playlist to yours;
• run this code from terminal: curl -d '{"jsonrpc": "2.0","id": 1,"method": "core.playlists.as_list"}' -H 'Content-Type: application/json' http://localhost:6680/mopidy/rpc
• You will see that the new playlist has not been added to mopidy (until your restart the server)

(with Spotify, playlists are immediately available in mopidy's playlists)

[fmarzocca]

Do you have any progress on this? I am not using any frontend, as I am handling mopidy from NodeRed, through Mopidy API RPC. Each time I add a playlist from Tidal I have to restart the server.

[tehkillerbee]

@fmarzocca Unfortunately I have not had time to test this. Perhaps, the playlists has to be refreshed beforehand.

Does this issue happen with any playlist or only specific playlists (eg. large playlists?)

[fmarzocca]

It happens on EVERY playlist I add. And it is really boring because if I add a playlist, I have to refresh ALL the others and it can take several minutes before starting playing. This was not happening with spotify.

[fmarzocca]

The title is not correct. NodeRed is not any part of this. The playlists are not loaded in Mopidy (NodeRed reads the playlists from Mopidy)

[fmarzocca]

I believe this is related with the "cache" issue you are handling on other tickets. Indeed if I ask Mopidy to "refresh" Tidal's playlist, they appear in the list, but the refresh process is quite time-demanding.

[tehkillerbee]

Yes, I figured it would be a cache refresh issue. Perhaps, we can force a refresh when requesting the playlist. With the new cache improvements in #60, it should be less of an issue to refresh, unless a full refresh is always performed

Regarding the title of this issue; Tidal playlists are loaded in Mopidy (otherwise they would not be shown in the clients). Since this appears to be a cache refresh problem, I would like the title to reflect this. I will leave Node red out of the title, however.

[fmarzocca]

Perhaps, we can force a refresh when requesting the playlist

Careful here... if the playlist contains hundred of tracks, it can take a long time. This is an excerpt of a long playlist refresh:

2022-05-24 10:56:28,454 INFO [4855:TidalBackend-3] mopidy_tidal.library: Returning 1 tracks 2022-05-24 10:56:28,456 INFO [4855:TidalBackend-3] mopidy_tidal.library: Lookup uris 'tidal:track:5147321:8919 4643:89194647' 2022-05-24 10:56:28,457 INFO [4855:TidalBackend-3] mopidy_tidal.library: Returning 1 tracks 2022-05-24 10:56:28,458 INFO [4855:TidalBackend-3] mopidy_tidal.library: Lookup uris 'tidal:track:25389291:904 22809:90422810' 2022-05-24 10:56:28,459 INFO [4855:TidalBackend-3] mopidy_tidal.library: Returning 1 tracks 2022-05-24 10:56:28,460 INFO [4855:TidalBackend-3] mopidy_tidal.library: Lookup uris 'tidal:track:7543394:8896 5581:88965586' 2022-05-24 10:56:28,461 INFO [4855:TidalBackend-3] mopidy_tidal.library: Returning 1 tracks 2022-05-24 10:56:28,462 INFO [4855:TidalBackend-3] mopidy_tidal.library: Lookup uris 'tidal:track:5806445:8537 8233:85378234' ....

[blacklight]

Careful here... if the playlist contains hundred of tracks, it can take a long time.

This should be addressed (or at least mitigated) in https://github.com/tehkillerbee/mopidy-tidal/pull/60.

I have some playlists with ~2500 items. Loading them the first time takes ~15 minutes, but any successive load request resolves in <10 seconds - even after mopidy is restarted. Simple playlist lookups (without expanding the tracklist) should instead resolve in <1 second.

Feel free to test the PR and let me know if it solves your issue. Also, note that if you want to load collections with >1000 items you may want to use my PR for python-tidal https://github.com/tamland/python-tidal/pull/99, since the current version has a hard ceiling on 999 items and it doesn't support pagination.

p.s. @tehkillerbee caching will surely solve many of these issues, but the first load of a playlist will still be annoyingly slow. I've noticed that the current library lookup logic scans a request ID by ID, and performs the lookups sequentially. Maybe if the TIDAL API supports bulk lookups we could leverage that, and if it doesn't then at least we could run a couple of lookup workers to speed things up?

[fmarzocca]

I've noticed that the current library lookup logic [scans a request ID by]

you got the point. The sequential scan ID by ID is really a pain…

[felipesaezreyes]

@BlackLight is there a way of using https://github.com/tamland/python-tidal/pull/99 with mopidy-tidal? I installed the specific branch and I'm getting

  Disabled extension tidal: Dependency The 'tidalapi<0.7.0,>=0.6.9' distribution was not found and is required by the application not found

from mopidy. Am I missing something?

[blacklight]

@felipesaezreyes how did you install python-tidal? The most recently released version on PyPI is 0.6.10, so pip install tidalapi should suffice.

Note however that pagination support and proper playlist cache refresh upon update are so far only available on my fork of python-tidal. That's because PRs on 0.6.x are currently blocked and 0.7.x will be a major rewrite that will require some changes on mopidy-tidal as well.

@tehkillerbee btw, according to the Github backlog only one feature is missing before python-tidal 0.7.0 (support for mixes, which I would argue is quite optional anyway). The developer has confirmed that the 0.7.x should already be in a usable state. Maybe it's worth migrating this project to the new version? So far some features (such as pagination and support of updatedAt on playlists) only work on my fork, which includes some patchy workarounds to get things to work before 0.7.

[felipesaezreyes]

@BlackLight I installed it by adding it to my Pipfile with this line

tidalapi = {git = "https://github.com/BlackLight/python-tidal.git", editable = true, ref = "pagination-support"}

and also tried with pip install git+https://github.com/BlackLight/python-tidal.git@pagination-support#egg=tidalapi

When installing the dependency is correctly resolved (or else mopidy-tidal would install the pip version), but it seems like mopidy is having issues loading the plugin when any git version is installed. I tried with https://github.com/tamland/python-tidal as well and I'm getting the same results, so I wonder if anything needs to be done so tidalapi is loaded. Thanks in advance

[felipesaezreyes]

@BlackLight nevermind, it was an issue with pypa. If I install using pip after the pipfile install it works. I'm able to load bigger playlists now, thanks!

[fmarzocca]

I am sorry, but my mopidy is become each day more unusable. Each time I fire a core.playlists.as_list() command to get the list of the playlists, mopidy-tidal start refreshing the tracks in the playlists and I can wait even 15 minutes before I can play...

I see you all are working hard on the cache issue. Is there an expected release day? Thanks

[blacklight]

@fmarzocca if you install the branch from my PR (https://github.com/tehkillerbee/mopidy-tidal/pull/60), as well as my python-tidalfork for pagination/playlist updates support, everything should already work fine (at least after the first load).

The best way to speed things up on the cache PR is to get more testers and reviewers.

Things will still be a bit slow the first time you load a Playlist though, but I'm thinking of speeding them up with a bunch of workers in parallel instead of doing sequential API calls - I just need to fine tune it to avoid hitting Tidal's API throttling limits.

[fmarzocca]

Ciao @BlackLight, I never installed a test branch. It should not be a great problem, but can you pls confirm procedure for that? Thanks

[blacklight]

@fmarzocca for caching support on mopidy-tidal:

git clone https://github.com/BlackLight/mopidy-tidal.git
cd mopidy-tidal
git checkout better-cache
[sudo] pip install .

For pagination/playlist updates support on the Tidal API:

git clone https://github.com/BlackLight/python-tidal.git
cd python-tidal
git checkout pagination-support
[sudo] pip install .

[tehkillerbee]

Closing for now, as this issue is fixed in PR #60