Ocassionally missing video uploads

[XrXr]

New videos are sometimes omitted in the response of YouTube api.

Example (excerpt from log)

Fri May 23 2014 21:15:20 GMT-0400  Checking all subscriptions... 1400894120129
Fri May 23 2014 21:15:20 GMT-0400  {"kind":"youtube#searchListResponse","etag":"\"bvxF-DWHx1toJotsdJBeCm43SLs/Z7muB3yLsIIdSeQTKlRVObzSk8M\"","pageInfo":{"totalResults":0,"resultsPerPage":5},"items":[]}  1400874843070
Fri May 23 2014 21:15:20 GMT-0400  {"kind":"youtube#searchListResponse","etag":"\"bvxF-DWHx1toJotsdJBeCm43SLs/eETUVVWhTEG2Ngy1MOMzzNl2rd4\"","pageInfo":{"totalResults":1,"resultsPerPage":5},"items":[{"kind":"youtube#searchResult","etag":"\"bvxF-DWHx1toJotsdJBeCm43SLs/5XJVXFnHwbQaf9Ey4tpNd-FbI7g\"","id":{"kind":"youtube#video","videoId":"lGcLnJJOFGA"},"snippet":{"publishedAt":"2014-05-24T01:00:01.000Z","channelId":"UCsQnAt5I56M-qx4OgCoVmeA","title":"Hearthstone: Trump Cards - 103 - Part 1: Top 1-2% Deck (Mage Arena)","description":"Trump drafts one of his best decks ever so the start for this arena run is very promising especially considering that the Mage Hero Power is arguably one of the ...","thumbnails":{"default":{"url":"https://i.ytimg.com/vi/lGcLnJJOFGA/default.jpg"},"medium":{"url":"https://i.ytimg.com/vi/lGcLnJJOFGA/mqdefault.jpg"},"high":{"url":"https://i.ytimg.com/vi/lGcLnJJOFGA/hqdefault.jpg"}},"channelTitle":"TrumpSC","liveBroadcastContent":"none"}}]}  1400874843075
Fri May 23 2014 21:15:20 GMT-0400  {"kind":"youtube#searchListResponse","etag":"\"bvxF-DWHx1toJotsdJBeCm43SLs/Z7muB3yLsIIdSeQTKlRVObzSk8M\"","pageInfo":{"totalResults":0,"resultsPerPage":5},"items":[]}  1400874843080
Fri May 23 2014 21:15:20 GMT-0400  {"kind":"youtube#searchListResponse","etag":"\"bvxF-DWHx1toJotsdJBeCm43SLs/euNqEa6U-fxz3oeU0XcNBnARsYM\"","pageInfo":{"totalResults":1,"resultsPerPage":5},"items":[{"kind":"youtube#searchResult","etag":"\"bvxF-DWHx1toJotsdJBeCm43SLs/VCKRJHl1dI9uwAmI76q_A1sJOiY\"","id":{"kind":"youtube#video","videoId":"IR9Zy7eoajg"},"snippet":{"publishedAt":"2014-05-23T23:44:29.000Z","channelId":"UCXuqSBlHAE6Xw-yeJA0Tunw","title":"WE'RE LIVE! Tune into the WAN Show at the link in the video description","description":"http://twitch.tv/linustech OR http://linustechtips.com/main/page/media.html Don't worry if you can't watch live. I'll upload it later.","thumbnails":{"default":{"url":"https://i.ytimg.com/vi/IR9Zy7eoajg/default.jpg"},"medium":{"url":"https://i.ytimg.com/vi/IR9Zy7eoajg/mqdefault.jpg"},"high":{"url":"https://i.ytimg.com/vi/IR9Zy7eoajg/hqdefault.jpg"}},"channelTitle":"LinusTechTips","liveBroadcastContent":"none"}}]}  1400874843086
Fri May 23 2014 21:15:21 GMT-0400  {"kind":"youtube#searchListResponse","etag":"\"bvxF-DWHx1toJotsdJBeCm43SLs/Z7muB3yLsIIdSeQTKlRVObzSk8M\"","pageInfo":{"totalResults":0,"resultsPerPage":5},"items":[]}  1400874843091
Fri May 23 2014 21:15:21 GMT-0400  {"kind":"youtube#searchListResponse","etag":"\"bvxF-DWHx1toJotsdJBeCm43SLs/Z7muB3yLsIIdSeQTKlRVObzSk8M\"","pageInfo":{"totalResults":0,"resultsPerPage":5},"items":[]}  1400874843096

The third channel is UC3tNpTOHsTnkmbwztCs30sA. submitting a query to youtube/v3/search With parameters

{
order: "date"
channelId: "UC3tNpTOHsTnkmbwztCs30sA",
publishedBefore: (new Date(1400894120129)) .toISOString(), //2014-05-24T01:15:20.129Z, the time when the check occurs
publishedAfter: (new Date(1400874843080)).toISOString() //2014-05-23T19:54:03.080Z, the time last check occurred.
}

Will return a result that has one video in it. However, according to the log, at the time of checking, nothing was return. I think the source of the problem is when a video is uploaded, it may take some time before the api will return the proper result. A different way of checking for new uploads is desirable for better stability.

[XrXr]

Testing 906892940902732a0bbfecd479aef613a6a8683d. Hopefully it addresses the issue.

[XrXr]

While 906892940902732a0bbfecd479aef613a6a8683d makes the video check more stable, it still missed an upload during 3 days of testing. (detected one that would've been missed by the old method) The video it missed is simply not present in the search response, even after 2+ hours. A new checking method is needed. After some looking, a map filter based solution that relies on PlaylistItems seems more stable. (It successfully returns the video that cannot be found searching) Implementation of the method will be done soon.

[XrXr]

8d0a6e197a43e36e226d9492c90df5f1b85199d7 implements the new checking mechanism. New videos seem to appear in the playlist slower than they do in the search result, however they should always appear eventually. (The video the old checking mechanism missed still isn't in the search result, its been a day) The delay seem to be fairly long(30min? or longer), but this is good for now. For even more robust checking, (this method will miss old videos that appear later than new videos)a cache based approach can be used. However that doesn't seem to be necessary.

[XrXr]

An alternate checking method relying on activities that is similar to the old checking method might be the best. The update doesn't have any delay, and it doesn't seem to miss entries.

[XrXr]

Checking via the activities does seem to work better than checking playlistItems. However, it missed one upload today. I was lucky enough to notice it right away and saw the problem unravel in real time. The new upload appeared after another activity which happened later than the upload. The caching approach will have to be implemented. Since the other activity wasn't an upload, the video was detected as a new upload one cycle later. A situation where two uploads get updated on different time haven't been seen yet, so I will hold off on implementing the new strategy

[XrXr]

activities_checking seems to work the best. It has not missed a single video in a week and a half. Which means it is probably the way to go. This should no longer be a problem.