Closed adrianreber closed 8 years ago
:+1: easy and safe
I have tested the code in the crawler. I was not able to test the code changes in app.py. I hope they make sense.
Without testing I got it wrong... Of course. This version now actually works.
Looks fine to me :+1:
Thanks.
Some mirrors where failing to be scanned as a leading whitespace broke our protocol detection (.startswith('rsync')). This ignores whitespace in the URLs.