Trakt updated shows API is now hard limited to 100 results per page meaning shows aren't updating

[Js41637]

Not sure when this change happened but I noticed recently that the calendar hasn't been updating, this issue #1272 is also probably related.

When DuckieTV checks for updated shows it calls https://api-v2launch.trakt.tv/shows/updates/{date}?limit=10000, since this was added it there was never a limit on the amount of results that could be returned but it is now capped at 100 which means there is basically 0 chance any of your shows will be getting updated.

Changing the update period also isn't likely to help as the changes we request are only yyyy-mm-dd which means you'll only be getting changes for the whole day so even if you check every hour, you'll still only be getting all changes for that day and if there is already over 100 then you're not going to get any updates.

Only way to fix this is the code is going to have to change to load every page of changes while the total results returned is less than 100, unless there is another way to get changes.

Only work around for now is to go into Settings > Backup > Refresh All Series

Unforunately this also means all old versions of DuckieTV now suffer this issue.

[garfield69]

Oh thank you Trakt.tv for this pain in the a** pointless change. So now instead of a single query we have to potentially make 10 multiple queries to get all the available info. ~~, and still potentially miss out on updates in situations where there may be more than 1000 results since the last time we did a fetch. Users that don't use Dtv daily are particularly going to get hit with this one. :-(~~

Will you have a go at coding the page loop fetching?

[AlienProber]

Oh thank you Trakt.tv for this pain in the a** pointless change. So now instead of a single query we have to potentially make 10 queries to get all the available info, and still potentially miss out on updates in situations where there may be more than 1000 results since the last time we did a fetch. Users that don't use Dtv daily are particularly going to get hit with this one. :-(

Will you have a go at coding the page loop fetching?

OK so garfield69 in plain English how fucked are we. Enough to actually put a refresh all button on the main DTV interface?

[Js41637]

It's fixable but it just means every old version of DuckieTV won't be able to update shows properly anymore which is particularly bad if you use any of the chrome versions as we can't really update them anymore.

[AlienProber]

You can't be talking about the nightly versions. If you are then there is no legitimate reason to even do the nightlys. I download a fresh nightly about once a month and load it and remove the old one once the new one is setup. Now garfield69 mentioned using the refresh all, is it possible to add that to the leftside of the main interface?

[garfield69]

in plain English how fucked are we?

Too early to say.

We first implement fetching the up-to-10 multiple pages of updates ~~, and then we monitor for a week and see just how many updates trakt.tv sends each us day. As long as there are less than 1000 we are good.~~

But if trakt.tv regularly sends exactly 1000, then that suggests that there may well be more than 1000 shows being updated on a daily basis, in which case it is likely that a few of the shows that Dtv users may have in their DB may not be getting updates on that particular day.

However, since the updates list is in chronological order rather than trakt_id or show_name order, then its likely that shows that miss out on an update one day may well get updated the next day if they are popular or current.

So my first reaction may well have been an over-reaction and the impact may not be that significant.

First things first, lets try code to fetch up to 1000 rather than the now default 100 updates.

@Js41637 I did not see a response to "Will you have a go at coding the page loop fetching?", are you taking on this coding task?

[Js41637]

You keep mentioning 1000 but I don't think there is a limit on how many updates there can be? The page size is limited to 100 but there isn't a limit on how many pages you can request. It definitely isn't going to be ideal if there are potentially thousands of updates per day as it will make fetching them all kind of annoying and requires a lot of requests.

Otherwise the other option we have is instead of fetching updates for all shows we just try and request updates for all the shows a user currently has added and we store the time each show was updated last to see whether we should update the whole show. This could basically still require just as many requests if a user has a lot of shows. Also as far as I can tell there is no easy way to see when a show was last updated apart from calling the for the full show details with ?extended=full. Might still be quicker and more efficient to just call the updates endpoint but I will see.

I can maybe look into it today.

You can't be talking about the nightly versions. If you are then there is no legitimate reason to even do the nightlys. I download a fresh nightly about once a month and load it and remove the old one once the new one is setup. Now garfield69 mentioned using the refresh all, is it possible to add that to the leftside of the main interface?

Not sure why you're mentioning nightly specifically also just because you update once a month does not mean others do. Up until now basically every version of DuckieTV still works somewhat, this basically prevents all older versions from working properly which is not ideal. I personally use the Chrome versions so I won't even be able to get this fixed because they can't be updated at the moment.

Also i don't think we'll be adding a refresh all button to the main sidebar. The way refresh works is really not ideal and should not be used repeatedly.

[AlienProber]

So in a nutshell the desktop version is more up to date than the browser versions directly from you github page. I would think both would be the same.

[garfield69]

You keep mentioning 1000 but I don't think there is a limit on how many updates there can be? The page size is limited to 100 but there isn't a limit on how many pages you can request.

@Js41637 sorry, I misread https://trakt.docs.apiary.io/#introduction/pagination and the bit about 1 page of 10

I should have read further, and now comprehend it was talking about the default, 1 page of 10 items Duh!

[garfield69]

So in a nutshell the desktop version is more up to date than the browser versions directly from you github page. I would think both would be the same.

incorrect. There is only one source, and the build bot publishes on GitHub the standalone and the chrome extensions builds. What is not up to date is that same Nightly chrome extension code on the Chrome browser Web Store, because the Web store is not accepting our build ATM

[AlienProber]

OK thanks that's what I wanted clarification on, cause I don't use the standalone cause I don't have a system wide VPN.

[garfield69]

I personally use the Chrome versions so I won't even be able to get this fixed because they can't be updated at the moment.

@Js41637 there is nothing stopping you downloading the Nightly chrome extension from the Nightly GitHub, and installing it on your Chorme Browser manually via the extensions page load-unpacked button. I use this regularly for testing, although for personal use I prefer the standalone.

[Js41637]

Chrome constantly pesters you about using dev extensions that aren't from the store and it's pretty annoying.

[garfield69]

Chrome constantly pesters you about using dev extensions that aren't from the store and it's pretty annoying.

Oh that sucks. I guess I don't notice because when I'm not testing I keep the extensions I'm not using disabled.

[AlienProber]

Chrome constantly pesters you about using dev extensions that aren't from the store and it's pretty annoying.

There is something wrong with your Chrome then, cause that is not normal. I maintain my computer and a friends and on both Vivaldi, Vivaldi-Snapshot, and Chrome. All of them I load DTV manually and he nor I have never had a message about it.

[Js41637]

Well I sent a ticket to Trakt asking about this issue and whether there is something they can do, 100 limit on this kind of endpoint is kind of ridiculous, if we were only to update once a day there can easily be thousands of changes which is upwards of 50 requests which is just stupid. If they say they're not going to change anything or if they just ignore me, i'll probably have to look into fetching a last updated time per show.

[treebsd]

howdy, other than hitting the refresh button for each show, is there a plan action or work-around in the works?

[garfield69]

@treebsd https://github.com/SchizoDuckie/DuckieTV/issues/1273#issuecomment-694166898

[treebsd]

@treebsd #1273 (comment)

thank you, i should go sleep.. missed that completely!

[AlienProber]

So what is the progress on this? Do we have a nightly download with a fix / work around yet? Thanks

[ilike2burnthing]

@Js41637 ping

[ianwday]

I'm wondering if there's any update on this issue. It's rendering DuckieTV less and less useful. Is there some sort of work-around? Is there anything that paying VIP users of Trakt.tv can do?

[AlienProber]

For now just go to settings, backup, refresh all series now.

[garfield69]

@Js41637 can you post the link to the trakt.tv ticket you raised please? Thanks.

[Js41637]

it's posted here but trakts api support is garbage and they don't care, judging from other issues they'll also be adding rate limits which will also break DuckieTV as it basically spams hundreds of requests at once, especially if you do a full refresh.

https://github.com/trakt/api-help/issues/206

[uncovery]

what's the way to work around this issue? Can duckieTV do a trickle-request and just refresh every hour one show?

Also is there a field for shows that have ended? Those could be polled only once a month to reduce traffic?

Or something of that sort. Sorry if those are stupid, just looking for a solution.

[ilike2burnthing]

A trickle-request like that would really limit how many shows could be refreshed (e.g. only 24 if you ran DTV for a full day, which I doubt many do), and would severely screw those who use DTV briefly to find their shows for the day, download them, and then exit - they would get 1 or 2 shows updated a day.

The way it would have to be done is either running individual updates for each show in full, separated by X seconds, or requesting when each show was last updated, and only pulling those needed, but both are more wasteful, and would likely cause more traffic for them (basically it's a stupid decision by them).

[uncovery]

only 24 if you ran DTV for a full day

this would be only true if you restart the cycle every day instead of cycling until all shows are done and then restarting.

which I doubt many do

fair enough, the issue of not running the PC 24/7 is an issue. Still the API limit is at 60 seconds with 100 requests, If you truly queue all requests and make 100 every minute, you can get through a lot of shows in 2-3 hours.

I would think that a lot of people who connect this tool to bittorrent do have it running for longer though. Finally BT does not download everything within seconds.

requesting when each show was last updated, and only pulling those needed

That's what I meant to do anyhow.

but both are more wasteful

More than what's happening now? How so?

[ilike2burnthing]

Previously, if you had 1000 shows, and 10% were updated, you only had to pull the updated information for 100 shows.

You would now have to either do: 1) 1000 individual full requests 2) 1000 individual requests for last update time, and then 100 individual full requests

[uncovery]

I guess the point needs to be that not every show should be checked if there is new data at trakt every 24 hours, but only a fraction of them every minute instead. If there is no other way than otherwise breaking the whole project, I guess this is still an option, even if people who have their PC on only for a couple of minutes a day - despite having hundreds of shows listed might have to look to another way to do this.

[ghost]

I leave mine always open on my media server, is this not the use case of Duckie? Are you supposed to close it?

[garfield69]

@ampped101 its a personal choice, use Dtv in what ever way suits your needs. For comparison, I usually start my Dtv first thing in the morning and fetch the outstanding episodes broadcast the previous evening. While waiting for DL to complete, any shows I viewed last night I set the watched status, and then shut Dtv down again when I've got my new episodes. Total daily up time is probably 5 to 10 minutes at the most.

[RustyRiley]

Have to say a big thank you, guys -- my first posting so please forgive if I'm intruding where I shouldn't. Was really worried this had been abandoned, what with things not updating, new series not coming up -- Ugh!! But there's reason to hope. It was great in the past, so here's hoping.

[AlienProber]

@RustyRiley new shows do post you might just have to go to settings, backups ans refresh all but they do come up and get updated still.

[Js41637]

You cant really use the refresh all anymore as the API is rate limited and it'll start erroring after processing a few shows, i'm going to quickly rewrite it to make it do one at a time and to stop spamming requests.

Also i'm going to write it in ES6. ES5 just makes promises and stuff too complicated, especially when trying to only do one thing at a time. ES6 has plenty of support these days but i don't know how any of the build tooling works? It should just work.

[Js41637]

well, pushed some stuff that might fix the issue? Kinda hard to test so let me know?

[Js41637]

Well I think it works, an issue I see though is we kind of need a progress monitor as updates can take quite a bit longer to complete now and there isn't any visual indication that the update is still in progress. Exiting before it completes isn't really a big issue as it doesn't update ones that have already been updated so it's not going to have to process them every time if it didn't finish however it does still need to make the initial request to see when it was last updated.

[ilike2burnthing]

Happy to do some testing if I can get an installer, or even just the modified file(s) for it.

[garfield69]

@ilike2burnthing your'e on windows right? and using a standalone nightly?

[ilike2burnthing]

Yep

[garfield69]

[edit9] the Nightly builds are back in action at https://github.com/DuckieTV/Nightlies/releases This covers the compete Standalone suite. However the Web Store Chrome Extensions will not be updated, as we cannot satisfy the security rules Google put in place a while back that prevent the nightly push to the store.

Outstanding issue is the Backup imports, which still need fixing.

[ilike2burnthing]

Changed dl=0 to dl=1 and got it. Will give it a shot now. Ta!

[garfield69]

he he, my 264 series update took 14 minutes and 16 seconds. Just as well I'm not in any hurry ;-D

[ilike2burnthing]

Lets see how long 443 takes then :P Currently sitting at 142 checked, with the devtools spamming me with rate limited! trying again in 31000 :/

I exited while the update was running, on restart and rerunning the backup it happily sped through the first ~100, up to where it was before closing, and then returned to updating once every few seconds.

Since I'm just sitting here with the rate limited warning being repeated, I've closed and reopened again. Same behaviour as before. Unsurprisingly, closing it for a minute does not resolve the rate limited issue.

Half-tempted to throw my VPN on to bypass this lol Actually, I wonder if we could use X-Forwarded-For headers to fake a different IP making the requests.

Slight irk, closing the settings pane, reopening, and going to the Backup section, the Loading, please wait # of # - Name... is no longer visible.

Well, I was about to comment, and it thankfully started up again, so we're just past 200 now!

EDIT: spoke too soon lol trying again in 156000...

[Js41637]

Doing a refresh of everything is worse than the scheduled updater as the updater only makes one request to see if something needs an update whereas the refresh does the whole show every time. Also the rate limit is 1000 every 5 minutes. So exiting in the middle would be bad as just getting back to where you were would waste like 200 requests.

Ideally it would ignore things updated within the last few minutes but unfortunately there is no timestamp stored when a show was actually last updated. It only stores the last updated timestamp that trakt returns and I didn't want to mess with making DB changes.

[ilike2burnthing]

Just doing it for testing purposes, wouldn't normally.

It stopped again at 387 updated, so I closed it again, reopened, and reran the refresh. It had no issue checking those 387, and then stopped where I left off. So I'm not sure if the check alone is being counted as part of the rate limit.

Any thoughts on using X-Forwarded-For headers to bypass it?

[roshavagarga]

Tested with 277, Backup showed it being completed within a minute or so (I might have been fairly up-to-date, since I used the 'Refresh all shows' a few times the past few days) - looked up in devtools, but didn't notice anything weird minus a few 404's from trakt, though I might be looking in the wrong place.

[Js41637]

Trakt uses CloudFlare. It's highly unlikely you can send a header to fake your IP as they likely use the one CloudFlare returns

[ilike2burnthing]

Offline for a few hours, reopened DuckieTV, and the update ran smoothly in about a minute. Looks good!

[garfield69]

An unintended benefit of the slow Trakt.tv fetching for the show updates is that the fanart fetching (that occurs s apart of the same process) is now working without the familiar server-overloaded errors, so we are saving more picture urls that previously) ;-)

[VulcanTourist]

That is encouraging news about the fanart icon/banner fetching, since I have never at any time had more than about half of them populate at once, and the ones that do change at random!

On Tue, Nov 3, 2020, 9:33 AM garfield69 notifications@github.com wrote:

An unintended benefit of the slow Trakt.tv fetching for the show updates is that the fanart fetching (that occurs s apart of the same process) is now working without the familiar server-overloaded errors, so we are saving more picture urls that previously) ;-)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/SchizoDuckie/DuckieTV/issues/1273#issuecomment-721273802, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOIV7TSEBBCGP4N4BEFVXTSOA5FFANCNFSM4ROGMRQQ .