Open macwinnie opened 2 years ago
Hey, This will be possible in pywebcopy 7. Currently its only in source code on github repo.
If you want that functionality of domain exclusion, then you have to go through pywebcopy 7 manually.
How to configure this on pywebcopy 7?
so the Session object has an attribute .domain_blacklist
it is an set or list object which is empty by default. Now if you want to skip the downloading from a certain domain, then just put the domain in a string format to the list and that's all. from then onwards the crawler will skip any page or file from that domain.
If we use domain_blacklist, resources from blacklisted domains don't get downloaded, but html links points to local resources which were not downloaded.
<a class="navbar-item" href="../blacklisted-domain/style.css">
Blacklisting is done by session object separately, so it poses as an unreachable link while the localiser still makes them localised. I guess its not a bug its a feature. On a serious note blocking should be done in scheduler so that it prevents this from happening. Scheduler also needs a queue.
Hi,
is there a configuration possibility to skip other domains from being crawled, so I only copy the whole website-stack of
example.com
but won't download remote links likegoogle.com
,facebook.com
and so on?I want to follow the basic documentation to copy one complete website – as:
Thanks a lot! Best macwinnie