google / corpuscrawler

Crawler for linguistic corpora
Other
192 stars 55 forks source link

[util/fetch_sitemap] Add subsitemap_filter option #2

Closed behnam closed 7 years ago

behnam commented 7 years ago

Allows faster initial fetches from websites with subsitemaps with language-based prefixes, like dw.com.

googlebot commented 7 years ago

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.


googlebot commented 7 years ago

CLAs look good, thanks!

behnam commented 7 years ago

Fixed: dropped the comment.