refactor request: import system

ohitstom commented 7 months ago

The import system is really bad, not to sound horrible.

Issues:

it doesnt update live and you have to reload
you cant import zips directly and you have to choose the files manually
it doesnt auto retry failed imports (it should auto retry and if it imports more after the retry to continue the retry loop, if it retries and nothing new gets added it should fail - this should apply to api timeouts too, if there is an api timeout there should be no reason to fail just keep trying until the api doesnt timeout)
its very slow
sometimes it gets stuck at 0 when imported from mobile devices
constant failing
if you start a new import and the last one had failed it should autocleanup and warn the user it will

So far in trying to import 26906 tracks, ive only got up to 10669 after 2 DAYS of manually retrying every 5 minutes, this is extremely painful and i dread to think how long it will take, if it will be possible at all, to import my extended user data when i receive it.

a potential fix to the api problem would be to allow the user to expose internal spotify apis server side using spicetify, you can receive all the same web-api stuff required with ZERO timeout or fetch limit.

Yooooomi commented 7 months ago

Hello! The basic privacy data does not allow one to import fast because there is no id for each item. I must search for the text on Spotify, find it, get the album and the artist, everything done within the respect of rate limits. The thing is meant to be slow. Also I don't see the point of having it real time if it's this slow. Uploading the takeout zip would upload unnecessary data that can grow very large in the future for whatever reason. You should try requesting extended streaming history. It'll be faster and more accurate. Thanks for your feedback

Edit: when mentioning fails or crashes, please join the associated logs otherwise there really isn't anything I can do.

ohitstom commented 7 months ago

Hello! The basic privacy data does not allow one to import fast because there is no id for each item. I must search for the text on Spotify, find it, get the album and the artist, everything done within the respect of rate limits. The thing is meant to be slow. Also I don't see the point of having it real time if it's this slow. Uploading the takeout zip would upload unnecessary data that can grow very large in the future for whatever reason. You should try requesting extended streaming history. It'll be faster and more accurate. Thanks for your feedback

Edit: when mentioning fails or crashes, please join the associated logs otherwise there really isn't anything I can do.

does this not mean they can be auto retried? and if it auto cleans the zip wont matte, real time is necessary because i cant see if its failed until i reload which i cant sit and do constantly!

Yooooomi commented 7 months ago

Could you join the errors you get when you try to import? Many thanks :)

ohitstom commented 7 months ago

Could you join the errors you get when you try to import? Many thanks :)

i dont think its an error its like a timeout error, how can i check though?

Yooooomi / your_spotify

refactor request: import system #315