Open Neiby opened 2 years ago
Thanks for letting me know. Can you point me to a query/resource that originally triggers this error? I tried with the one that I see in your trace irr_hunt_routes(20473, 'AS-CHOOPA', None)
but it seems it gets completed with no error. I suspect garbage lines may be injected by the ipwhois
package, so knowing the exact resource that is not parsed correctly would be of great help.
That is the query that causes the problem for us, and it happens every time. Very interesting! If it matters, I'm running this from a Macbook Pro, so Mac OSX. I wonder if we're running different versions of the dependencies than you are.
I'm running v1.2.0 of ipwhois
and v1.4 of bgpq4
.
We think the problem might be with ipwhois
. For example, this is a quick test that always fails for us. And by "fail", I mean that the source
always has extraneous information in the string. We used the same dummy network and approach found in whois_proxy.py
.
>>> from ipwhois.net import Net
>>> from ipwhois.asn import ASNOrigin
>>> DUMMY_NET = "193.201.40.0"
>>> mynet = Net(DUMMY_NET)
>>> obj = ASNOrigin(mynet)
>>> lookup = obj.lookup('AS137')
>>> for net in lookup['nets']:
... print(net)
@vajra77 I think we've shown this problem exists in ipwhois
, not your code, so you can probably close out this issue. We haven't figured out root cause, but it seems the extraneous information in the source
field is coming from ipwhois
, or perhaps even the raw WHOIS data. I'm not sure yet.
@vajra77 I think we've shown this problem exists in ipwhois
, not your code, so you can probably close out this issue. We haven't figured out root cause, but it seems the extraneous information in the source
field is coming from ipwhois
, or perhaps even the raw WHOIS data. I'm not sure yet.
If you're interested, I submitted an issue for the ipwhois
problem here:
https://github.com/secynic/ipwhois/issues/317
It appears to be a problem with a regular expression.
Hi, thanks for submitting the ipwhois
issue, I have added a workaround in WhoisProxy.expand_as()
to strip away all the remarks strings that follow the source name in the output from ipwhois
, tested with records for AS137. Regarding AS20473,AS-CHOOPA I have tested it both on Mac OS X and Ubuntu 20.04, both with bgpq
1.4 and ipwhois
1.2.0.
Excellent! Thank you so much for your help.
Occasionally,
new_route.source
will have multi-line output in it that includes remarks appended. Here's an example of what I mean:When that happens, we get the following traceback:
The exception is in
irr_scan.py
in a function that checks to see if the source exists inself._source_weight.keys()
. I was looking through the code to see where the source info is originally acquired. It appears to be in theexpand_as
function inwhois_proxy.py
.Perhaps one solution is to check to see if there is whitespace in the source string and, if so, split it into a list and then take the first element. I was able to get a working solution using that method directly in
irr_scan.py
just as a test, but I don't think doing it there was very clean and I just did it as a test to verify that it solved this particular problem.