YouTube v2 API Deprecated - need to switch to v3 API

[Vankog]

Yesterday everything was quite fine. Today nothing works anymore. (Even though there was no update or version change...)

Tried with latest stable Chrome and IE: 1.a) already loaded videos (from yesterday) still show up 1.b) if going inkognito or open IE (I use it rarely), the video list stays empty after initialization 2) clicking on anything - video or menu items - or scrolling (so the update kicks in) always results to the initialization screen.

I first thought some chrome flags or extensions might have some influence on this. However, after checking some of them out and having the same result with IE, the origin seems to lie somewhere deeper in the guts.

[jtokoph]

It looks like the V2 API is deprecated (https://developers.google.com/youtube/2.0/developers_guide_protocol_subscriptions#Retrieving_new_subscription_videos)

It also looks like there isn't a corresponding V3 API: https://code.google.com/p/gdata-issues/issues/detail?id=3946

[daviddeutsch]

Yeah, this is the same as #136 - But dang it, either I completely misread their deprecation policy or they recently changed that recently.

Anyhow, doesn't change a thing, it's gone now and I'll have to fix it, soon. *sigh

[daviddeutsch]

Just a note here: Fuck it, let's go. I'm working on it.

[nickbrooking]

Probably not the right place to put this, but I just wanted to say, this tool is a life-saver for me. I would be completely unable to use youtube without it, so thank you very much for working on it :)

[daviddeutsch]

Nah, that's alright - thanks for helping me keep up the good spirits :wink:

[Vankog]

oh... ehm... doesn't "deprecated" mean it is still usable for some time? ^^ "most API functionality that was subject to that policy will continue to be available until April 20, 2015."

bugger...

[daviddeutsch]

@Vankog Yeah, pretty much all the functionality we need has been deprecated. Bugger indeed.

[AdvanceMancer]

Like neonshadow said, sanegrid is my go-to when I want to watch yt. Thanks for keeping the support, David!

I'll never understand why they keep placing so many barriers, so many clunky changes and try to kill anything that tries to interact with the site. Their priority should be having a robust api, instead of killing it for the sake of doing it. Maybe I'm biased because I don't like how google is handling yt these times, but I still can't understand what goes with their minds, really. /rant

[kalaspuffar]

I say the same. Love the application and I support with a small dollar amount on flattr because it's just that good. Even built a small android version for my own use when I'm on the go. Looking forward to a fix and please keep up the good work. An app being stable for these many months without a major hickup well that's almost Google quality :)

[daviddeutsch]

@Kanta- Me neither. The distinct feeling I get is that it's simply a matter of other things being more profitable. In the end, YT is a way to sell advertisement spots to advertisers, so you could argue that having a confusing site and API helps with that. May be too simplistic a thought, but then again, Occam's Razor and so forth.

@kalaspuffar Ooooh, feeling that burn :grin:

[Lebon14]

Since YT Startpage script has been totally out of commission, I've been here the whole time. Hopefully we can have it back in the next few days because as of right now, I'm using my twitter feed to watch videos.

Can we have an ETA soon, David?

[daviddeutsch]

I want to get this done over the weekend, but the current stopgap is wrestling the google singin API, so that's a "known unknown" that I cannot determine how long it will take with full accuracy. I'll try to get it done asap.

[daviddeutsch]

Ooh, just had a small breakthrough - authentication works now (although it's still a bit clunky). Now I pretty much just have to make it work with the new data formatting...

[Espyo]

Good luck! I'll be cheering on over here.

[Vankog]

First feedback for now: Authentication failed with:

Error: origin_mismatch

Application: YT Sanegrid

You can email the developer of this application at: skoremail@gmail.com

Request Details
immediate=false
response_type=token
scope=https://www.googleapis.com/auth/youtube
redirect_uri=postmessage
pageId=114110833026210133778
proxy=oauth2relay497076431
state=318767842|0.1381700968
origin=https://daviddeutsch.github.io
include_granted_scopes=true
client_id=950592637430.apps.googleusercontent.com
authuser=0

I've tried to login with my secondary user that is connected with my primary user. EDIT: However, the primary does not work neither. ;-)

[kalaspuffar]

Just a bit more feed back. Seems like my app stopped working as well so there is no turning back. Not sure I'll rewrite it seems like something I don't want to do.

[daviddeutsch]

Status update: I've rewritten large parts of how sanegrid is loaded due to the dependency of the google login api. I'm mixing in a bunch of refactors I had on my list for a while, so it's not all their fault, of course. For now, the login itself seems to work pretty well.

Retrieving the data will again be a tad different - the v2 API simply gave you a straight up list of videos in chronological order. With v3, you have to first get your channel list and then iterate through the channels loading the latest videos in through the activity stream.

As I mentioned in #136, the new API will make it possible to work on a bunch of longer standing suggestions, I'll list them here for further reference:

109, #104, #101, #84 (not sure about this, but possible), #83 (highly doubt this one, but it might be possible to at least share the view status), #67, #66, #64, #29, #12

All in all, I think once I'm done with this, I can finally call it a 1.0 :wink:

[Lebon14]

Oh boy... Good luck with all the work involved!

[SecondFlight]

So my list of "oh I'm totally going to watch those later" said me 5 months ago types of videos is back c:

claps

[thew78]

Good luck with the work! Waiting for a fix!

[Lebon14]

Any news?

[tordenflesk]

I get a "Select a account" popup, then nothing happens.

[daviddeutsch]

Right in the middle of the final push. Hold on to your seats :wink:

[daviddeutsch]

Another late night update here:

I have the connection to where it connects to your account and downloads channel and video data. I've also prepared localForage to stand in for the current localStorage based client side data storage.

Switching to localForage will make it possible to store far more data than we could so far. It will also give me a clean break because I do need to change a couple of data structures (the v3 API returns a lot more data in general). I also want to make the lists a little more performant, so instead of one huge list that stores all the videos, it will be a "new", "archive" and "trash" list in the future and you switch between them, if you want to. Clicking the video buttons will no longer just change the video state, but move it to another list.

Of course, there will be an option for migrating the existing data. I think finishing the video loading itself will only take another day or two. The functionality for migrating the data will take another day or two, but since we're opening up a new storage option, the old data isn't lost and can be migrated later on.

Thanks again for your patience, guys!

[AdvanceMancer]

Everything seems amazing! I can't wait David, thanks for keeping us updated ;)

[CJCon]

Ohhh I'm glad it's going well! And the new features (the 3 lists that you mentioned) sound really cool too.

Thanks for the status update and good luck!!! ^ ^

[daviddeutsch]

Another quick update - videos are now loading and the last remaining task is storing the data (including that three list thing that I mentioned). I will also do a little more cleanup and then evaluate how fast I can write a migration script.

[chhe]

Quick feedback. It seems you set the allowed origins for your your client id to "http://daviddeutsch.github.io" only which results in the following error message when i try to connect to "YT Sanegrid" my YouTube account using "https://daviddeutsch.github.io" (https instead of just http):

400. That’s an error.

Error: origin_mismatch

Application: YT Sanegrid

You can email the developer of this application at: skoremail@gmail.com

Request Details
immediate=false
response_type=token
scope=https://www.googleapis.com/auth/youtube
redirect_uri=postmessage
pageId=104721779687771398711
proxy=oauth2relay610818099
state=1236909458|0.3136622914
origin=https://daviddeutsch.github.io
include_granted_scopes=true
client_id=950592637430.apps.googleusercontent.com
authuser=0
That’s all we know.

[daviddeutsch]

A very good point indeed! I'll look into that.

[Vankog]

Ah, there lies the origin of the problem.

[daviddeutsch]

Should be working on https now.

[Vankog]

veriii nais!

Works now basically. However, many videos don't show up yet. I've made several site searches on sanegrid with the whole list visible, but the missing ones don't appear in the list anywhere. Also with ca. 450 loaded videos the list already goes back two years. This is way too few. In a good month I watch this amount in one month. (already around 150 subscriptions by now... I use YT as a TV substitute.)

[daviddeutsch]

Will have to check up on those videos. Are those just uploads on channels you subscribe to, though, or are they also "events" (like a channel liking a video in another channel etc.)

As for the loaded video amount: Well, of course it would eventually load more than 450 as new ones come in. That's just the ones it loads initially.

[nickbrooking]

Should it be actually doing something at this point? All I get is the main page, I click "Let's go!", it opens the OAuth window where I select my username, then the window closes and nothing else happens. Here is the console log in Chrome if that helps:

Refused to set unsafe header "Origin" jquery-1.10.1.js:8717
Refused to set unsafe header "Origin" jquery-1.10.1.js:8717
GET http://daviddeutsch.github.io/yt-sanegrid/%7B%7Bvideo.thumbnail.medium%7D%7D 404 (Not Found) jquery-1.10.1.js:6184
event.returnValue is deprecated. Please use the standard event.preventDefault() instead. jquery-1.10.1.js:5395

[daviddeutsch]

yeah, I keep breaking stuff left and right as I refactor things, so it's not really useful until this ticket here is closed :wink:

[nickbrooking]

Got it, thanks! Excuse us for being impatient, you are doing God's work over there so take your time :)

[daviddeutsch]

No problem at all, I understand - I'm as impatient as you are :wink:

[Vankog]

"Are those just uploads on channels you subscribe to, though, or are they also "events"" No, just uploads.

"As for the loaded video amount: Well, of course it would eventually load more than 450 as new ones come in. That's just the ones it loads initially." Sure, but that's not what I mean. The list is sorted by upload-date, right? It cannot really be that the videos at the bottom of the list (around position 400) are all already 1 to over 5 years old. The videos at these positions must be a max. of around 1-2 months old if they get loaded sequentially ordered by date. What I want to say is: there are tons of videos missing. ;-) Here's a screen of the last entries:

EDIT: btw, little side note: Because I just saw it. Have you realized that some durations show "undefined"? :-)

[daviddeutsch]

@Vankog What is actually happening is that I'm pulling videos from each channel individually. If a channel doesn't have that many uploads, it will return a couple of very old videos.

At least that's what I hope is happening, it might also me derping up the activities stream...

[Vankog]

(EDIT: As you just pointed out yourself...) OK, part of this is just related to the fact that there are only loaded so many videos per channel. For instance, I have the latest 20 by xdadevelopers. They post several times a week. But only the latest 5 from WheezyWaiter, who posts 5 times a week or so. On the other hand I have (only) the last 13 by VolkerStruebing. His main activity was years ago. So these are making up the old videos at the end of the list.

So I think this means two things:

1. The count of loaded videos per channel seems a bit arbitrary.
2. Many channels don't load at all. (e.g. Ralph Ruthe's English channel Ruthe Cartoons loads while his German channel koalakombat doesn't)

[daviddeutsch]

@Vankog Thanks, I will investigate further!

Alright, packing up for today. Videos load consistently now. Had a very weird problem where the YT API would return false page markers turning into an infinite loop. The odysee of the weird API continues, I suppose. I also think that maybe their own UI is so terrible... because it has to get their data through the very same API.

I have already sketched out a bunch of new providers that will finally enable me to pick apart the huge controller that currently governs the main video list. There are already a few speed improvements so far, but it will only get really interesting once I have things picked apart and in a more sensible configuration. (Do keep in mind that I programmed sanegrid to teach myself AngularJS - that does show, particularly now that I'm getting back to it after working on a bunch of "real" angular projects :wink: )

What I'm really satisfied with is the new list designations. Archive and Trash make a lot more sense than the watch and mute labels I had before. It will also make it possible to have dedicated filter rules for them, like "delete anything beyond 50 items in the trash" and the like.

I will either continue working on this tomorrow or the day after, I'm glad that I got it to a point where you can at least use Sanegrid to get a proper video list now :wink:

[Lebon14]

Seriously, thanks a bunch. It works enough for me to have a good list of videos to watch.

Archive and Trash... I don't like the term Trash, it seems like you want to want to remove from the Internet. "Remove" (from Grid) seem to be better term. Archive, though, is just fine.

Also, when you'll be finished fixing everything up, don't forget my feature request to create a black theme (I know I created a ticket where I requested it).

[daviddeutsch]

@Lebon14 The reason why I want those two terms is because they are a "place" that you move videos into. When you watch something, it then goes into your Archive. When you don't want to watch something, it goes in your Trash. Maybe stick with the names, but the hover text for the trashcan symbol would be 'Remove'?

And: Yeah, everything that is open on the tracker will definitely be done in the future. Just need time for it is all.

[G4M3R-729]

Some issues: Can't refresh the sub page, just stays at the spinning dots thing, using chrome. Some, not all, videos are missing - Seems random

TY

Edit: Also it seems that when you refresh the page, you have to authorise again, it doesn't stay "logged" in, unlike before.

[thew78]

Couple of problems:

I get the feeling that the subscription box isn't showing all my videos. With the V2 API I used to get 2 rows of videos a day (~18 videos) and now just half. Especially when scrolling down, only like 2 videos / day show up. Is this a bug or a feature?

Random videos keep popping up aswell quite some seconds later after a refresh is "done". (never realy seems to be done if I have to trust the refresh icon)

I don't realy like the fact that I have to select my Youtube account aswell. Is the Oauth window required for this app to work with API v3?

Thanks for all the hard work. I love this application! I can't see myself using the regular subscription box anymore.

And last of all, @Vankog : 150 is nothing, I'm subscribed to 262 ;)

[nickbrooking]

I'm still getting the same issue as I stated above. Can't get the videos page to load. After I select my youtube account in the OAuth window, the window just closes and the page does not change.

Edit: Sorry this was my fault, I had privacy badger running and it was blocking this behavior. It's working now, thanks.

[thew78]

After a few days I can confirm that not all videos seem to be showing up in my sanegrid.

[nickbrooking]

Same here, some do not show up. On May 26, 2014 9:21 AM, "thew78" notifications@github.com wrote:

After a few days I can confirm that not all videos seem to be showing up in my sanegrid.

— Reply to this email directly or view it on GitHubhttps://github.com/daviddeutsch/yt-sanegrid/issues/137#issuecomment-44188509 .

[Lebon14]

Looks like the API V3 do what the API V3's gonna do... This is why David sticked to V2 until the end.

[Vankog]

No, so far this seems to be channel related, not related to single videos.