search

mozilla-magnet / magnet-metadata-service

A service that returns metadata for given URLs
Other
8 stars 10 forks source link

204: No Content #54

Open kevinahuber opened 8 years ago

kevinahuber commented 8 years ago

Fantastic service! Very happy to see a node version of the PWS that is pushing the limits of what it can be. I really like your additional scraping services.

In Google's PWS, if a site returns 204 (no-content), the service bails and the url is not shown in scanners: https://github.com/google/physical-web/blob/master/web-service/helpers.py#L217

It would be great to follow that standard here- right now magnent scanners just show fetching content.. indefinitely.

arcturus commented 8 years ago

This can be an easy fix:

https://github.com/mozilla-magnet/magnet-metadata-service/blob/master/lib/routes/api/metadata/request.js#L123