Support for downloading "non-new" trailers

aag / apple_trailer_downloader

A script to download HD trailers from the Apple Trailers website.

GNU General Public License v3.0

23 stars 5 forks source link

Support for downloading "non-new" trailers #8

Closed forthrin closed 8 years ago

forthrin commented 8 years ago

The current version of the script downloads the most recent trailers, which makes sense for continuously staying up-to-date with all released trailers.

However there are a lot of trailers which are hosted which are not listed as "most recent", some of which may be several years old, which also would be nice to download.

Is there any way to download "all" trailers hosted in some organised way?

aag commented 8 years ago

As far as I know, there's no way to get a complete listing of all the trailers hosted on the Apple site. There are a few more available under the "Genres" and "Studios" links on the website, but they appear to be just recent and upcoming releases.

forthrin commented 8 years ago

The approaches I can come up with are:

Perform a Google search for "site:trailers.apple.com/trailers" and parse the resulting HTML pages (or maybe better perform the searches through the Google API)
The JavaScript search on the Apple site performs searches when typing in just two letters. Searching all combinations of A-Z + A-Z would be 676 searches which is not completely outrageous. You could probably run this once and eliminate all combinations which return no results for future searches (assuming no movies with weird names would start showing up).

I would suggest that the searches are performed on a central server which makes available a list of URLs that the script talks to, and randomise/delay the searches to avoid potential detection/blocking.

aag commented 8 years ago

I'm not sure either of those methods would be reliable. You only get a maximum of 60 results for each search on the Apple trailers site, so that doesn't seem like a reliable way to crawl the catalog. Google also limits the number of hits for a single query to about 950 results, so I don't know how to get a complete listing out of it.

However, even if there were a reliable way to crawl the trailers site, I'm not particularly interested in downloading all of the trailers and it seems to me like it's out of the scope of this script. However, anyone else is free to create a separate project to get all of the trailer page URLs and either use this script as a Python library or call it with each page URL on the command line.