Spotify json import no longer working

[lylebrown]

It seems like Spotify has changed the format of their exports.

Example of old file: Filename: StreamingHistory0.json

[
  {
    "endTime" : "2021-04-21 05:30",
    "artistName" : "Fall Out Boy",
    "trackName" : "Sugar, We're Goin Down",
    "msPlayed" : 215710
  },
  {
    "endTime" : "2021-04-23 23:27",
    "artistName" : "Arrested Youth",
    "trackName" : "98 Degrees",
    "msPlayed" : 167126
  },
  {
    "endTime" : "2021-04-23 23:30",
    "artistName" : "Mystery Skulls",
    "trackName" : "Live Forever",
    "msPlayed" : 171226
  },

Example of new file: Filename: Streaming_History_Audio_2019_0.json

[{"ts":"2019-02-11T05:03:20Z","username":"**REDACTED**","platform":"Windows 10 (10.0.17134; x64)","ms_played":47980,"conn_country":"US","ip_addr_decrypted":"**REDACTED**","user_agent_decrypted":"unknown","master_metadata_track_name":"Resonance","master_metadata_album_artist_name":"Home","master_metadata_album_album_name":"Odyssey","spotify_track_uri":"spotify:track:1TuopWDIuDi1553081zvuU","episode_name":null,"episode_show_name":null,"spotify_episode_uri":null,"reason_start":"clickrow","reason_end":"endplay","shuffle":false,"skipped":null,"offline":false,"offline_timestamp":1549861353656,"incognito_mode":false},{"ts":"2019-02-11T05:04:12Z","username":"**REDACTED**","platform":"Windows 10 (10.0.17134; x64)","ms_played":16550,"conn_country":"US","ip_addr_decrypted":"**REDACTED**","user_agent_decrypted":"unknown","master_metadata_track_name":"Come Back Down","master_metadata_album_artist_name":"Home","master_metadata_album_album_name":"Odyssey","spotify_track_uri":"spotify:track:0Z5r4cmTjEyA62tfSbAbFD","episode_name":null,"episode_show_name":null,"spotify_episode_uri":null,"reason_start":"clickrow","reason_end":"endplay","shuffle":false,"skipped":null,"offline":false,"offline_timestamp":1549861401124,"incognito_mode":false},{"ts":"2019-02-11T05:05:05Z","username":"**REDACTED**","platform":"Windows 10 (10.0.17134; x64)","ms_played":22970,"conn_country":"US","ip_addr_decrypted":"**REDACTED**","user_agent_decrypted":"unknown","master_metadata_track_name":"Outside the Realm","master_metadata_album_artist_name":"Big Giant Circles","master_metadata_album_album_name":"There Came an Echo","spotify_track_uri":"spotify:track:21uFfgSfkwvDwJA26GbmWl","episode_name":null,"episode_show_name":null,"spotify_episode_uri":null,"reason_start":"clickrow","reason_end":"endplay","shuffle":false,"skipped":null,"offline":false,"offline_timestamp":1549861453146,"incognito_mode":false},{"ts":"2019-02-11T05:06:20Z","username":"**REDACTED**","platform":"Windows 10 (10.0.17134; x64)","ms_played":49620,"conn_country":"US","ip_addr_decrypted":"**REDACTED**","user_agent_decrypted":"unknown","master_metadata_track_name":"Seabirds","master_metadata_album_artist_name":"pizzagirl","master_metadata_album_album_name":"Seabirds","spotify_track_uri":"spotify:track:5I0iOfX9YLacuhLspYnTaY","episode_name":null,"episode_show_name":null,"spotify_episode_uri":null,"reason_start":"clickrow","reason_end":"endplay","shuffle":false,"skipped":null,"offline":false,"offline_timestamp":1549861505969,"incognito_mode":false},

Let me know if I can help with any other info.

[FoxxMD]

Exports? Import? Multi-scrobbler does not import anything from file. Should this issue have been made on the Maloja repository?

[lylebrown]

Exports? Import? Multi-scrobbler does not import anything from file. Should this issue have been made on the Maloja repository?

I thought I was in the right place? Am I going crazy?

[FoxxMD]

OOPS i am the one going crazy :sweat_smile: I thought this was my repo. Too early in the morning to be answering issues. Sorry

[krateng]

Did some initial parsing, it looks like these new files don't contain information about the track artist, is this correct? I'm using the album artist for now, but this is of course far from ideal.

[lylebrown]

It looks like you're right. While it would be time intensive on the server during the import, the Spotipy library can get track information from that ID.

[Dbhardwaj99]

Hey guys, Can I try working on this Issue? Will be my first Open Source Contribution

[lylebrown]

I'll make a suggestion for whoever works on this - if you do end up looking up each track by ID, it'd probably be a good idea to cache those IDs in the process, because it's highly likely to see repeats in listening history.

I'd be interested on working on this as well, but my time is very limited right now, so it'd likely be months before I could start.

[Kappawaii]

the fix mentionned in #228 does not work, just tried building it on bleeding-edge and it still gives the same error : File StreamingHistory_music_0.json could not be identified as a valid import source.

[carharv]

Just worked for me using the latest docker image with a fresh install. Normally I use a compose file, but I deleted the container and launched Maloja in shell mode with the snippet on the readme. I had a folder of 8 spotify json files in my data directory that were all detected and imported in one go. After that I deleted the container I started in shell mode and used my compose file like usual.