search

yasserg / crawler4j

Open Source Web Crawler for Java
Apache License 2.0
4.56k stars 1.93k forks source link

added disregarded protocols #446

Open mihalispap opened 4 years ago

mihalispap commented 4 years ago

expanded disregarded protocols to skip such links