Import Playcounts and Loved Tracks from Last.fm/Libre.fm

[sten0]

Is your feature request related to a problem? Please describe. One barrier to switching between music library managers (so much more than an audio player) is that the user loses playcount and loved track statistics. This was why I originally started using last.fm eg: external repository of such things. It also provides a centralised location that one can sync from. eg: a users' cellphone and cheap rockboxed mp3 player (for gym) scrobble to last.fm or libre.fm, and Strawberry syncs the changes from one of these services to its own database.

Describe the solution you'd like I'd be happy with a well-maintained script. A button would be a plus.

Describe alternatives you've considered @kadaimx clementine-player/Clementine/issues/90 https://www.linux-apps.com/content/show.php/Last+Sync?content=65784 https://github.com/werdeil/clementine-last-export

[jonaski]

This would be nice and more user friendly to have directly included in the player instead of a script. API documentation: https://www.last.fm/api/show/track.getInfo

[sten0]

Jonas Kvinge notifications@github.com writes:

This would be nice and more user friendly to have directly included in the player instead of a script. API documentation: https://www.last.fm/api/show/track.getInfo

Agreed! I just wanted to set the bar as low as possible, in the hopes someone would see the issue and think "oh, that's easy, I can do that" ;-)

[UndeadKernel]

This would indeed be an excellent addition to strawberry. However, how could we deal with the rate limit of Last.fm? I have about 10,000 songs in my library, I'm sure sending 10k requests would not work. Are there any bulk fetching mechanisms we could use?

[NucleaPeon]

@UndeadKernel One thought I have is that it would just fetch statistics on the current playing tracks and store those (simplest option). If getting all tracks are necessary, perhaps a background job that does a slow increment through the user's collection. That, or simply go through as many songs as possible before rate limiting occurs (error 29) and then queue them until later, but that might interfere with other functions such as "Loving" a track. Forgive me if I'm being ignorant (unable to find the information), but I'm not seeing what the rate limits are.

[UndeadKernel]

Doing it every time a track is played sounds like a reasonable way to do it without going above the rate limit (which, indeed, is not clearly specified). How would you cope with the possibility of having songs with higher play times than what is stored in last.fm? Would you just ignore then what comes back? Does the API provide a way to update track statistics?

[sten0]

UndeadKernel notifications@github.com writes:

This would indeed be an excellent addition to strawberry. However, how could we deal with the rate limit of Last.fm? I have about 10,000 songs in my library, I'm sure sending 10k requests would not work. Are there any bulk fetching mechanisms we could use?

I'm not sure what approach clementine-last-export (https://github.com/werdeil/clementine-last-export) uses, except that it's slow (several hours for my similarly sized library), and that it works, so long as the local ID3 tags are sufficiently similar to the last.fm ones. The API provides a method to get corrected track names, so for maximum accuracy ones could build a list of tracks that didn't return a playcount and then try this corrected track name method in a 2nd run. Beyond that I suspect the law of diminishing returns is in effect for the outliers...maybe the "no playcounts found after 2nd run" list could be dumped into a playlist for user review?

The API documentation frequently mentions 50 in combination with any batch operations (eg: https://www.last.fm/api/show/track.scrobble?lang=pt ), so I wonder if that's the limit? I also wonder if one of the reasons it's not well-documented is because they made some of the old v1 features "subscriber-only".

§ 4.4 at https://www.last.fm/api/tos gives "partners [at] last [dot] fm" as a contact for the API limits.

There's an example article about downloading 10 years of scrobbles here: https://www.r-bloggers.com/10-years-of-playback-history-on-last-fm-just-sit-back-and-listen/

This project looks alive, and maybe we could consult them about this? https://github.com/pylast/pylast

Oh, and here is what seems like a high-quality quick reference: https://rapidapi.com/dimashirokov/api/LastFm/details

[sten0]

Dann notifications@github.com writes:

@UndeadKernel One thought I have is that it would just fetch statistics on the current playing tracks and store those (simplest option). If getting all tracks are necessary, perhaps a background job that does a slow increment through the user's collection. That, or simply go through as many songs as possible before rate limiting occurs (error 29) and then queue them until later, but that might interfere with other functions such as "Loving" a track.

Here's the project for a GPL-3 Android scrobbler. Its handling of batching pending scrobbles and love status is reliable when moving between a place with cellular service and a métro/tube/subway. https://github.com/kawaiiDango/pScrobbler

From what I've been able to find out, queuing things for future scrobbling needs a start date+time and possibly and end date+time. I believe "loving" a track can be similarly deferred.

Forgive me if I'm being ignorant (unable to find the information), but I'm not seeing what the rate limits are.

I'm guessing the limit is 50...see previous message.

[sten0]

UndeadKernel notifications@github.com writes:

Doing it every time a track is played sounds like a reasonable way to do it without going above the rate limit (which, indeed, is not clearly specified).

Unfortunately this doesn't fulfill the motivation for filing this bug, namely bootstrapping the playcounts from last.fm (I've logged playcounts here for most of my adult life), and using last.fm (or libre.fm!) as a synchronisation point between devices (namely a computer and android phone for most users, but possibly a rockboxed mp3 player too).

For anyone with a large collection, this is particularly useful for rediscovering albums with low playcount stats (eg: an album you listened to half a dozen times nine years ago and then forgot about). In Amarok 1.x and Clementine I had/have a bunch of dynamic playlists that do interesting things with these stats. It's a killer feature that instantly sold me on the last.fm/libre.fm + Amarok 1.x/Clementine/Strawberry combo.

How would you cope with the possibility of having songs with higher play times than what is stored in last.fm? Would you just ignore then what comes back? Does the API provide a way to update track statistics?

I would just ignore what comes back when it's lower. The only methods I'm aware of to update the track stats are LastFm.scrobbleTracks and LastFm.scrobbleSingleTrack, but it should be mentioned that V1 of the API was abused to do this, so they made some changes for V2, such as denying scrobble for requests that are too far in the past. It's also worth mentioning that synthesizing a fake listening queue that is scrobbled when the player is paused will mess up a user's charts.

[jonaski]

I started looking into this. There is the user.getRecentTracks method: https://www.last.fm/api/show/user.getRecentTracks, we can get last played time from there. Rate limiting isn't a huge problem with that since we can get 1000 results for each page. We can also date it back as far as we want. However, for some strange reason, it does not include play counts so we need to use another method for that. To get play counts we have to use user.getTopTracks: https://www.last.fm/api/show/user.getTopTracks, looks like there is a restriction on how long you can date it back, but there is "overall" for period, does that mean you can get all tracks ever listened, or is it limited how long it dates back?

[esalgado]

Hello, I can see that "Last played" and "Play Count" is in the menu, I can successfully import "Play Count" with 12700 songs, but importing "Last played" crashes randomly (20%, 59%), while importing 51000 tracks. It succeeded after some tries... but I see that last played is set to 1/1/70 for all them.

What I would love to have "loved songs" matched to the 5 star rating (that is different from Play Count). DO you think this is possible?

https://www.last.fm/api/show/user.getLovedTracks

Thanks, I just discovered this player, and I am in love :+1:

[jonaski]

It might be because the date/time returns in a different format based on your regional settings. If you know how to compile from source try removing << Param("lang", QLocale().name().left(2).toLower()) in lastfmimport.cpp

[jonaski]

I think there is bug in QDateTime::fromString() in Qt 5.15.1, don't know if other Qt versions are affected too. But it works with Qt 6 (dev branch). Example: QDateTime::fromString("09 Sep 2020, 00:17", "dd MMM yyyy, HH:mm")

[esalgado]

Hello! I have installed qt5-base 5.15.2-1, I am installing now QT6, since commenting out that line from lastfmimport and compiling resulted in the dialog fetching initial data, but hanging at 0% receiving data :( (for 51169 themes). It is just a matter of installing QT6, and compiling? It will recognize I have both QT5 and QT6 and use the later?

Edit: nevermind, it looks quite straighforward, in the Readme:

`To compile with Qt 6 use:

cmake .. -DBUILD_WITH_QT6=ON`

Thanks :)

[esalgado]

Hello!

Sorry for my late reply... I have finally compiled using QT6, and the import last playcount made it to the end :+1:

But I think that instead of this being the last played date, it is the first played date:

Can it be that the function is continously updating the play date for a particular song, in the order that the Last.fm API sends them? All those 51k plays, is all my play history since 2005, and maybe it is being sent by Last.Fm in reverse chronological order, or maybe random?

I don't know if I have to open a bug for this?

Thanks for the app, I love it :)

[undergroundwires]

Clementine refugee here 😄 I came for this feature. How can I use it? When I show column "play count" it does not show the stats from last.fm.

[esalgado]

Hello!! You just need to use the "Tools" - - > "Import from Last.fm" menu option 👍