Closed Hywan closed 8 years ago
@krtek4 Are you OK with this one?
This is a first step... We should also raise a warning when the crawler cannot access an URL, it is fairly easy to catch those by registering the events fetch404
and fetcherror
, eventually fetchtimeout
Bug merge this as is and maybe open a new thing.
Done :-).
(git rebase master
)
Fix #29.