Open JulienPalard opened 2 years ago
Hm, it seems like the page returns a "proper" 404 status code when the PDF isn't yet available.
I do wonder why the filter is run in this case. Have you maybe set things up to ignore 404 errors?
No, it they don't return a proper 404, in fact it really depends on the request:
$ curl -I https://donneespubliques.meteofrance.fr/donnees_libres/bulletins/BCM/202207.pdf
HTTP/1.1 404 Not Found
...
$ curl -i https://donneespubliques.meteofrance.fr/donnees_libres/bulletins/BCM/202207.pdf
HTTP/1.1 302 Found
Date: Tue, 19 Apr 2022 20:08:54 GMT
Location: https://donneespubliques.meteofrance.fr/?fond=donnee_indisponible
...
And in urlwatch, in the job retrieve function we're hitting the 2nd case:
(Pdb) p response
<Response [200]>
(Pdb) p response.url
'https://donneespubliques.meteofrance.fr/?fond=donnee_indisponible'
(Pdb) p response.history
[<Response [302]>]
I wanted to use urlwatch like this:
to wait for next bulletins, but when a bulletin is not available yet meteofrance.fr redirects to an HTML page, leading to an exeption:
maybe the pdf2text need some kind of error handling option, what do you think?