search

jaimeiniesta / metainspector

Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
https://github.com/metainspector/metainspector
MIT License
1.03k stars 165 forks source link

Issue HTTP HEAD request first to check content type before downloading? #221

Open ksouthworth opened 6 years ago

ksouthworth commented 6 years ago

First of all, thanks for the great gem!

It seems when inspecting pages that have a lot of links to big files like PDFs, that the entire file is downloaded (HTTP GET) before checking the Content-Type header to see that it's not an HTML file and then raising the MetaInspector::ParserError.

It would be really nice if there was an option or something to pass in so that MetaInspector first just does an HTTP HEAD request on the URL to get the response headers back from the webserver so it can check things like response HTTP Status Code and Content-Type more efficiently, without having to download the entire large PDF file first to do so.

Thanks!

jaimeiniesta commented 6 years ago

Thanks!

This sounds like a great idea to me. As maybe not all servers will reply to HEAD, it's fine that it's an option, off by default.

PRs welcome :)