search

trakt-tools / universal-trakt-scrobbler

MIT License
551 stars 54 forks source link

Error scanning netflix history #250

Closed Kellett closed 1 year ago

Kellett commented 1 year ago

I saw there was a patch recently to fix some a scan problem with netflix. I downloaded the release version (The way you guys do releases is confusing as you seem to be updating the old release tag. I think it's the release up-to commit 185e85ce360c097703f20ef68590eb0dd5406ddc).

I load this into FireFox (108.0.2 (64-bit)) as a temp extension

I run the history sync for netflix and it runs for a little while then throws an error.

From the devtools console: [UTS] Failed to load history. Error: {"request":{"url":"https://www.netflix.com/api/shakti/mre/viewingactivity?languages=en-US&authURL=1673274687910.ftGMVLddQcQMvIYknk\\x2Ffb6LDfnM\\x3D&pg=12","method":"GET","withHeaders":{},"withRateLimit":{"id":"default","maxRPS":{"*":2}}},"status":0,"text":"","isCanceled":false} r moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 n moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 p moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 O moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 k moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 l moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 value moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 p moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 O moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 k moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 value moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:205 p moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 O moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 k moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 value moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:205 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:75 p moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 O moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 k moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 value moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:75 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:205 p moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 O moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 k moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 k moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:205 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:205 p moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 O moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 k moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 t moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 k moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:205 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:205 p moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 O moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 k moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 s moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 promise callback*e moz-extension://532a46a6-fe94-4f40-9b77-635b47424eba/history.js:2 history.js:2:363359

And here is the json that I assume it was trying to read at the time {"codeName":"S-Icarus-6.Jump!","profileInfo":{"profileName":"JK","avatarImages":{"32":"https://occ-0-1175-1168.1.nflxso.net/dnm/api/v6/K6hjPJd6cR6FpVELC5Pd6ovHRSk/AAAABdJLDomY0pjvbIKej3gt6O-5QxS_n10ybY8Ks58bkjpqeet8sfcBJtdYz2rjKmBzDQZr61nqQF2oOaNnr-U2XyBXXdQIEUY.png?r=5f6","50":"https://occ-0-1175-1168.1.nflxso.net/dnm/api/v6/K6hjPJd6cR6FpVELC5Pd6ovHRSk/AAAABUptvR6LljQ3K5HT_zfbYxjOuBShMLlHpe1kLkFDWwMRtbT_XVxExe2nE8PF90_tKAiR3RPKCH00pkkCy2pp7DVnRg2q_tg.png?r=5f6","64":"https://occ-0-1175-1168.1.nflxso.net/dnm/api/v6/K6hjPJd6cR6FpVELC5Pd6ovHRSk/AAAABe-QnHUVABn71vp4S5yTNzvOGbkUa_BwAwt5R5DET06HcyXkNFjEU8w3Oti2wrzFfZ4PlYYx4XHWc336gjGxKCwPLwD5zJU.png?r=5f6","80":"https://occ-0-1175-1168.1.nflxso.net/dnm/api/v6/K6hjPJd6cR6FpVELC5Pd6ovHRSk/AAAABW1VkxDxANPNCfJt5bNaJaeD2qRyJll7bmFnOH5rim6k0yRV1RyYB_zXzX6rzLreUVrS9Uyt6iM1_zKIIhwauxZGKLMhj7A.png?r=5f6","100":"https://occ-0-1175-1168.1.nflxso.net/dnm/api/v6/K6hjPJd6cR6FpVELC5Pd6ovHRSk/AAAABZNCEnmmu5k_Em_1k4QL4M2RImerU9JhLFdEspJ0yGY-hr-VN0qpmXwRaYWGQn7o1T4z4PZ7z25GG2BWoftocOJQ_Ia1aaw.png?r=5f6","112":"https://occ-0-1175-1168.1.nflxso.net/dnm/api/v6/K6hjPJd6cR6FpVELC5Pd6ovHRSk/AAAABfcB6hDEmmBmUmr1YkWg98hx0UY-xdv6DHKgFdGdR-IIxBJsn-3LGZogPemq4VFLqX0eh93rca0tZY7ywpNymJcw-sGCb1g.png?r=5f6","160":"https://occ-0-1175-1168.1.nflxso.net/dnm/api/v6/K6hjPJd6cR6FpVELC5Pd6ovHRSk/AAAABah9aZ6vXXFOvDAjLzyK_qA7qfbzRQGOnMUNdS9ePfIXFYE7xwTaMEepLj1BbfZA-OLiffeU2jS-1v6SyKIoLDCeYP-z2MEs_w.png?r=5f6","200":"https://occ-0-1175-1168.1.nflxso.net/dnm/api/v6/K6hjPJd6cR6FpVELC5Pd6ovHRSk/AAAABX1xGCcp7ypx9VMpHQwdAk9FJV8cOHRhYGMU7LvvqrqGFUZECimlrbWApoX6I_9ig2ZltmnEum28ko9eHvk0c25OgTDeRGLEJA.png?r=5f6","320":"https://occ-0-1175-1168.1.nflxso.net/dnm/api/v6/K6hjPJd6cR6FpVELC5Pd6ovHRSk/AAAABcI7whK3xReOzqR8bFF9aMaBfGyT8s0Zt0OsHgDnV_3O3oMfd3wu84Je53vJN6Zkall_roci9_6fsNbcy8qZNJN6n5WkQvfUlv70.png?r=5f6"},"guid":"QYJ2LOCK5RFIBIMI643W3WYQCM","isAccountOwner":false},"viewedItems":[{"title":"Season 1: \"BoJack Horseman: The BoJack Horseman Story, Chapter One\"","videoTitle":"BoJack Horseman: The BoJack Horseman Story, Chapter One","movieID":70298930,"country":"GB","bookmark":1464,"duration":1539,"date":1661773187478,"deviceType":2233,"dateStr":"8/29/22","index":240,"topNodeId":"70300800","rating":"50","series":70300800,"seriesTitle":"BoJack Horseman","seasonDescriptor":"Season 1","episodeTitle":"BoJack Horseman: The BoJack Horseman Story, Chapter One"},{"title":"\"Hamoodi\"","videoTitle":"Hamoodi","movieID":81187222,"country":"GB","bookmark":1301,"duration":1421,"date":1661771709676,"deviceType":2233,"dateStr":"8/29/22","index":241,"topNodeId":"81134264","estRating":"23","series":81134264,"seriesTitle":"Mo","seasonDescriptor":"Mo","episodeTitle":"Hamoodi"},{"title":"Season 1: \"The Apartment\"","videoTitle":"The Apartment","movieID":80004178,"country":"GB","bookmark":1291,"duration":1303,"date":1661722944303,"deviceType":1376,"dateStr":"8/28/22","index":242,"topNodeId":"70281562","estRating":"49","series":70281562,"seriesTitle":"Brooklyn Nine-Nine","seasonDescriptor":"Season 1","episodeTitle":"The Apartment"},{"title":"Season 1: \"Full Boyle\"","videoTitle":"Full Boyle","movieID":80004177,"country":"GB","bookmark":1296,"duration":1305,"date":1661721654439,"deviceType":1376,"dateStr":"8/28/22","index":243,"topNodeId":"70281562","series":70281562,"seriesTitle":"Brooklyn Nine-Nine","seasonDescriptor":"Season 1","episodeTitle":"Full Boyle","estRating":"49"},{"title":"Season 1: \"My Mother, the Car\"","videoTitle":"My Mother, the Car","movieID":70133680,"country":"GB","bookmark":1266,"duration":1303,"date":1661720395786,"deviceType":1376,"dateStr":"8/28/22","index":244,"topNodeId":"70140358","rating":"50","series":70140358,"seriesTitle":"Arrested Development","seasonDescriptor":"Season 1","episodeTitle":"My Mother, the Car"},{"title":"Season 1: \"In God We Trust\"","videoTitle":"In God We Trust","movieID":70133679,"country":"GB","bookmark":1271,"duration":1302,"date":1661718495021,"deviceType":1376,"dateStr":"8/28/22","index":245,"topNodeId":"70140358","rating":"50","series":70140358,"seriesTitle":"Arrested Development","seasonDescriptor":"Season 1","episodeTitle":"In God We Trust"},{"title":"Season 7: \"The Mustard Tiger\"","videoTitle":"The Mustard Tiger","movieID":70171791,"country":"GB","bookmark":1334,"duration":1357,"date":1661598410179,"deviceType":1376,"dateStr":"8/27/22","index":246,"topNodeId":"70153385","rating":"50","series":70153385,"seriesTitle":"Trailer Park Boys","seasonDescriptor":"Season 7","episodeTitle":"The Mustard Tiger"},{"title":"Season 1: \"The Party\"","videoTitle":"The Party","movieID":80004176,"country":"GB","bookmark":1283,"duration":1289,"date":1661597132197,"deviceType":1376,"dateStr":"8/27/22","index":247,"topNodeId":"70281562","estRating":"49","series":70281562,"seriesTitle":"Brooklyn Nine-Nine","seasonDescriptor":"Season 1","episodeTitle":"The Party"},{"title":"Season 1: \"Five-O\"","videoTitle":"Five-O","movieID":80021961,"country":"GB","bookmark":2515,"duration":2562,"date":1661594341540,"deviceType":1376,"dateStr":"8/27/22","index":248,"topNodeId":"80021955","estRating":"47","series":80021955,"seriesTitle":"Better Call Saul","seasonDescriptor":"Season 1","episodeTitle":"Five-O"},{"title":"Season 4: \"Who Pooped the Bed?\"","videoTitle":"Who Pooped the Bed?","movieID":70224522,"country":"GB","bookmark":1221,"duration":1265,"date":1661593111318,"deviceType":1376,"dateStr":"8/27/22","index":249,"topNodeId":"70136141","rating":"50","series":70136141,"seriesTitle":"It's Always Sunny in Philadelphia","seasonDescriptor":"Season 4","episodeTitle":"Who Pooped the Bed?"},{"title":"Season 4: \"Mac and Charlie Die: Part 2\"","videoTitle":"Mac and Charlie Die: Part 2","movieID":70224521,"country":"GB","bookmark":1221,"duration":1246,"date":1661591185464,"deviceType":1376,"dateStr":"8/27/22","index":250,"topNodeId":"70136141","rating":"50","series":70136141,"seriesTitle":"It's Always Sunny in Philadelphia","seasonDescriptor":"Season 4","episodeTitle":"Mac and Charlie Die: Part 2"},{"title":"Season 2: \"Under Pressure\"","videoTitle":"Under Pressure","movieID":81519684,"country":"GB","bookmark":2507,"duration":2580,"date":1661588630845,"deviceType":1376,"dateStr":"8/27/22","index":251,"topNodeId":"81349493","estRating":"36","series":81349493,"seriesTitle":"Big Timber","seasonDescriptor":"Season 2","episodeTitle":"Under Pressure"},{"title":"Season 1: \"Visiting Ours\"","videoTitle":"Visiting Ours","movieID":70133676,"country":"GB","bookmark":1269,"duration":1304,"date":1661587367304,"deviceType":1376,"dateStr":"8/27/22","index":252,"topNodeId":"70140358","rating":"50","series":70140358,"seriesTitle":"Arrested Development","seasonDescriptor":"Season 1","episodeTitle":"Visiting Ours"},{"title":"Season 1: \"Charity Drive\"","videoTitle":"Charity Drive","movieID":70133678,"country":"GB","bookmark":1271,"duration":1303,"date":1661586091945,"deviceType":1376,"dateStr":"8/27/22","index":253,"topNodeId":"70140358","rating":"50","series":70140358,"seriesTitle":"Arrested Development","seasonDescriptor":"Season 1","episodeTitle":"Charity Drive"},{"title":"Season 1: \"Key Decisions\"","videoTitle":"Key Decisions","movieID":70133677,"country":"GB","bookmark":1270,"duration":1305,"date":1661533207054,"deviceType":1376,"dateStr":"8/26/22","index":254,"topNodeId":"70140358","rating":"50","series":70140358,"seriesTitle":"Arrested Development","seasonDescriptor":"Season 1","episodeTitle":"Key Decisions"},{"title":"Season 1: \"Alpine Shepherd Boy\"","videoTitle":"Alpine Shepherd Boy","movieID":80021960,"country":"GB","bookmark":2618,"duration":2674,"date":1661524707627,"deviceType":1376,"dateStr":"8/26/22","index":255,"topNodeId":"80021955","estRating":"47","series":80021955,"seriesTitle":"Better Call Saul","seasonDescriptor":"Season 1","episodeTitle":"Alpine Shepherd Boy"},{"title":"Season 1: \"Hero\"","videoTitle":"Hero","movieID":80021959,"country":"GB","bookmark":2810,"duration":2838,"date":1661521900965,"deviceType":1376,"dateStr":"8/26/22","index":256,"topNodeId":"80021955","estRating":"47","series":80021955,"seriesTitle":"Better Call Saul","seasonDescriptor":"Season 1","episodeTitle":"Hero"},{"title":"Season 2: \"Between the Devil and the Deep Blue Sea\"","videoTitle":"Between the Devil and the Deep Blue Sea","movieID":81519683,"country":"GB","bookmark":2540,"duration":2616,"date":1661416947383,"deviceType":1376,"dateStr":"8/25/22","index":257,"topNodeId":"81349493","series":81349493,"seriesTitle":"Big Timber","seasonDescriptor":"Season 2","episodeTitle":"Between the Devil and the Deep Blue Sea","estRating":"36"},{"title":"Season 1: \"Bringing Up Buster\"","videoTitle":"Bringing Up Buster","movieID":70133675,"country":"GB","bookmark":1304,"duration":1330,"date":1661378359408,"deviceType":1376,"dateStr":"8/24/22","index":258,"topNodeId":"70140358","rating":"50","series":70140358,"seriesTitle":"Arrested Development","seasonDescriptor":"Season 1","episodeTitle":"Bringing Up Buster"},{"title":"Season 1: \"Top Banana\"","videoTitle":"Top Banana","movieID":70133674,"country":"GB","bookmark":1290,"duration":1324,"date":1661376275164,"deviceType":1376,"dateStr":"8/24/22","index":259,"topNodeId":"70140358","rating":"50","series":70140358,"seriesTitle":"Arrested Development","seasonDescriptor":"Season 1","episodeTitle":"Top Banana"}],"vhSize":2787,"trkid":200250783,"page":12,"size":20,"tz":"GMT"}

This seems to be reproducible every time. So if you need me to provide extra info or anything let me know.

downey-lv commented 1 year ago

I don't see anything wrong with the JSON itself, the format looks exactly like mine. I copied it to my instance manually and the history loaded perfectly fine.

image

Since it gets to Page 12 of your history and then has an issue out of nowhere, I was wondering if maybe it has something to do with the rate limiter, but I was unable to replicate the issue as the API requests are quite sparse and I can load way past Page 12.

From the error, it seems like it's a network request issue, as it doesn't even get an HTTP status code; so fetching the page fails completely. Could you check what the Network tab for the extension looks like when it fails? Do you see the request for &pg=12 and is it successful (what's the status and payload)?

image

Filter on viewingactivity here: image

Kellett commented 1 year ago

Hey, sorry I forgot I'd tried this and logged this issue.

Pulled the latest code, debugged, ran fine this time. Managed to do like 4 years of history. I don't know if I was hitting a rate limit or if something has changed it the code but its working fine for me now.