Improve handling of videos deleted from YouTube

[GoogleCodeExporter]

What steps will reproduce the problem?
The search page returns a video with no title or icon, with 'steve' listed as 
its describer. The video cannot be played.

Attempting to edit the video reveals its mediaId as IKH8vr0oM8g. 

Attempting to play that video directly from YouTube shows that YouTube deleted 
it for copyright violations.

YouDescribe should handle this situation more gracefully. It should not show 
any entry in the search results, and delete the video from its database. It 
should then notify the describer that YouTube has removed the video.

Original issue reported on code.google.com by keith...@gmail.com on 6 May 2013 at 6:31

[GoogleCodeExporter]

This is actually a broader issue; a video can be deleted, or marked as 
"Private", on YouTube, and it will no longer play in YouDescribe.

1/. We need to investigate how the caching of YouTube data (to speed up the 
loading of titles for the search page) affects this problem, and whether 
removing the caching actually significantly affects performance.

2/. We need a periodic task to run on DVX itself, which confirms each video in 
the database against YouTube, and hides descriptions of videos which have been 
made private or deleted. The same task needs to make the hidden descriptions 
visible again in the search list if that video subsequently is marked "Public". 
The task could run daily, and needs to be carefully designed so that temporary 
network or server outages do not cause all descriptions to become hidden!

3/. The Player page should detect that the video does not exist, and display a 
warning message rather than looking like it is buffering before playback 
forever. This is related to more general error handling in the player, which is 
a separate issue.

Original comment by Owen.R.E...@gmail.com on 26 Sep 2013 at 6:39

[GoogleCodeExporter]

Some information on the search.php page as it stands right now;

1/. With the YouTube cache, search.php loads in ~1.2 seconds
2/. Without the YouTube cache, search.php loads in ~9.1 seconds
3/. Without the cache, using the 'http://www.youtube.com/oembed' API instead of 
'https://gdata.youtube.com/feeds/api/videos/' to get titles, search.php loads 
in ~6.4 seconds

There might be other (more up-to-date) YouTube API we could use, but I just 
wanted to check what the OEmbed one gave, because it returns very little extra 
info.

Original comment by Owen.R.E...@gmail.com on 26 Sep 2013 at 10:24

[GoogleCodeExporter]

Okay, here's an option that addresses the problem until we make changes in the 
server:

* Ignore title information from the DVX database
* Always look in the cache for the title, and if it's not there, get it from 
YouTube and cache it
* Every time the search page is loaded, randomly delete ONE cache entry so that 
the next time the page is loaded, it will have to go to YouTube for the data 
(and will also randomly delete another cache entry). Over time each cache entry 
will be deleted and replaced.
* If no title is found, don't display the video in the search list UNLESS 
you're logged in AND you described that video (this logic was actually already 
in there, I just forgot to document it). So if you described a video but it was 
deleted or hidden, you still get shown that video but with no title or 
thumbnail. In the future, we will allow the description to be 'cloned' to 
another title.

The performance penalty of always looking in the cache (rather than the DVX 
database for title), and always having to fetch at least one title from 
YouTube, makes the page load in ~1.9 seconds.

Original comment by Owen.R.E...@gmail.com on 26 Sep 2013 at 11:57

• Changed state: Started

[GoogleCodeExporter]

Rather than search.php removing a cache entry randomly, the cache entry is now 
flushed and reloaded by player.php. So any attempt to play a deleted/private 
video will clear the cache entry, meaning it won't be shown in the search.php 
list again (unless it is subsequently made public)

Original comment by Owen.R.E...@gmail.com on 1 Nov 2013 at 3:44

• Changed state: CodeDone

[GoogleCodeExporter]

Fixed in release YouDescribe-release-20131220.

Original comment by Owen.R.E...@gmail.com on 21 Dec 2013 at 5:28

• Changed state: Verified