`jq: error (at <stdin>:1): Cannot iterate over null (null)`

From a Hotel Wifi I get:

$ git log --oneline --no-decorate -1
6cb4456 improved subnet detection, introduced ROU element in report. - many ISPs allocate small blocks for PtP networks (or customers allocations), and take the time to define them in RIPE. But Cymru/Pwhois don't index these blocks (since they're not routed directly but aggregated in larger routes). Now `asn` will try to identify these subnets even if they're routed within larger prefixes. - NET element refers now to the smaller inetnum, while ROU reports the target IP's route. - Examples:   - asn -n 188.152.136.18   - asn -n 195.103.16.76   - asn -n 217.212.125.124   - asn -n 2.228.17.105
$ ./asn -v www.molgen.mpg.de

────────────────────────────────────────────────────────────
            WARNING 

No IPQualityScore token found, so disabling in-depth threat 
analysis and IP reputation lookups. Please visit 
https://github.com/nitefood/asn#ip-reputation-api-token 
for instructions on how to enable it. 
────────────────────────────────────────────────────────────

╭──────────────────────────────────╮
│ ASN lookup for www.molgen.mpg.de │
╰──────────────────────────────────╯

- Resolving "www.molgen.mpg.de"... 1 IP address found:

[2022-11-10 19:58:14]   DEBUG   curl -s api64.ipify.org
[2022-11-10 19:58:16]   DEBUG   curl -m5 -s https://stat.ripe.net/data/abuse-contact-finder/data.json?resource=134.76.31.205&sourceapp=nitefood-asn
[2022-11-10 19:58:17]   DEBUG   curl -m4 -s https://ipmap.ripe.net/api/v1/locate/134.76.31.205/best
[2022-11-10 19:58:18]   DEBUG   curl -m4 -s http://ip-api.com/json/134.76.31.205?fields=status,message,country,countryCode,regionName,city,mobile,proxy,hosting
[2022-11-10 19:58:18]   DEBUG   curl -m2 -s https://api.incolumitas.com/datacenter?ip=134.76.31.205
[2022-11-10 19:58:18]   DEBUG   curl -m4 -s http://api.stopforumspam.org/api?json&ip=134.76.31.205
[2022-11-10 19:58:19]   DEBUG   curl -m5 -s https://api.greynoise.io/v3/community/134.76.31.205
[2022-11-10 19:58:20]   DEBUG   curl -m5 -s https://internetdb.shodan.io/134.76.31.205
[2022-11-10 19:58:21]   DEBUG   curl -s https://stat.ripe.net/data/rpki-validation/data.json?resource=680&prefix=134.76.0.0/16&sourceapp=nitefood-asn

 134.76.31.205 ┌PTR npsw-www.mpg.de
               ├ASN 680 (DFN Verein zur Foerderung eines Deutschen Forschungsnetzes e.V., DE)
               ├ORG GWD Goettingen
               ├NET 134.76.0.0/16 (GWDG)
               ├ABU abuse@gwdg.de / abuse@ripe.net
               ├ROA ✓ VALID (2 ROAs found)
               ├TYP  Mobile network IP 
               ├GEO Göttingen, Lower Saxony (DE)
               ├CPE [APP: apache:http_server] [APP: rubyonrails:rails] [APP: ruby-lang:ruby]
               ├POR Open ports: 80, 443
               └REP ✓ NONE

╭────────────────────────────╮
│ Trace to www.molgen.mpg.de │
╰────────────────────────────╯

[2022-11-10 19:58:22]   DEBUG   mtr -> 134.76.31.205 (5 rounds)
 Hop IP Address                                                                               Loss%      Ping avg     AS Information                
  1. _gateway (172.16.3.254)                                                                     0%        1.5 ms    BOGON  rfc1918 (Private Space)
  2. host18523696-201.telnaptelecom.pl (185.236.96.201)                                          0%        1.4 ms   [AS43372] TELNAP, PL
  3. ???                                                                                       100%             *   (No reply)
  4. host185186152-40.telnaptelecom.pl (185.186.152.40)                                          0%        7.9 ms   [AS43372] TELNAP, PL
  5. 82.177.247.209                                                                              0%        2.1 ms   [AS20804] ASN-TELENERGO ul. PERKUNA 47, WARSZAWA, PL
[2022-11-10 19:58:36]   DEBUG   curl -s https://www.peeringdb.com/api/ixpfx?prefix__startswith=88.220&protocol__in=IPv4
  6. 88.220.206.191                                                                              0%       21.5 ms   (EXATEL-NET)
[2022-11-10 19:58:38]   DEBUG   curl -s https://www.peeringdb.com/api/ixpfx?prefix__startswith=88.220&protocol__in=IPv4
  7. 88.220.204.181                                                                              0%       18.0 ms   (EXATEL-NET)
[2022-11-10 19:58:39]   DEBUG   curl -s https://www.peeringdb.com/api/ixpfx?prefix__startswith=88.220&protocol__in=IPv4
  8. 88.220.195.38                                                                               0%       18.1 ms   (EXATEL-NET / Connected by EXATEL S.A.)
[2022-11-10 19:58:41]   DEBUG   curl -s https://www.peeringdb.com/api/ixpfx?prefix__startswith=88.220&protocol__in=IPv4
jq: error (at <stdin>:1): Cannot iterate over null (null)
  9. 88.220.196.43                                                                               0%       16.3 ms   (EXATEL-NET / Connected by EXATEL S.A.)
 10. ???                                                                                       100%             *   (No reply)
 11. cr-han2-be6.x-win.dfn.de (188.1.144.134)                                                    0%       25.1 ms   [AS680] DFN Verein zur Foerderung eines Deutschen Forschungsnetzes e.V., DE
 12. kr-goe12-13.x-win.dfn.de (188.1.231.126)                                                    0%       27.9 ms   [AS680] DFN Verein zur Foerderung eines Deutschen Forschungsnetzes e.V., DE
C 13. gv-GWDGIR-gwdg.net.gwdg.de (134.76.147.181)                                                 0%       27.8 ms   [AS680] DFN Verein zur Foerderung eines Deutschen Forschungsnetzes e.V., DE
 14. npsw-www.mpg.de (134.76.31.205)                                                             0%       26.2 ms   [AS680] DFN Verein zur Foerderung eines Deutschen Forschungsnetzes e.V., DE

Trace completed in 22 seconds on 2022-11-10 19:58:44 CET

╭──────────────────────────────╮
│ AS path to www.molgen.mpg.de │
╰──────────────────────────────╯

  43372  TELNAP (Local AS)
 ╭╯
 ╰20804  ASN-TELENERGO ul. PERKUNA 47
 ╭╯
 ╰680    DFN Verein zur Foerderung eines Deutschen Forschungsnetzes e.V.

Hey @paulmenzel!

here's a run down of what I think is happening:

1) the path to your target crosses a bunch of hops (the 88.220.x.x ones) which belong to unannounced prefixes (likely used internally for a PNI/internal network handover between the "main" ASN-TELENERGO network and the EXATEL-NET one - which I assume is some subdivision of the former - not really familiar with Polish ISPs). Therefore Team Cymru has no data about the 80.220.x.x IPs. 2) Since 80.220.x.x is unannounced and unknown, asn tries to detect if it's an IXP prefix, by querying PeeringDB 3) PeeringDB has recently introduced throttling if you query them without an API key (which I honestly deem overkill), so while this should not happen very often during daily use (it's rare to find more unannounced subsequent IPs in a traceroute, they're more likely private IPs than unannounced public ones), in your scenario we're querying PeeringDB continuously. 4) After two/three queries in rapid succession, PeeringDB will fail on you with a "throttled" error - you can quickly verify this by running this command in a shell and repeating it for 3/4 times : curl -s "https://www.peeringdb.com/api/ixpfx?prefix__startswith=88.220&protocol__in=IPv4"

I think it's not worth it to introduce the burden of yet another API key just for PeeringDB (I strive to focus on free, public APIs that require no key - that's why I added shodan only after I found out about their "no API key" service aka InternetDB)

Nevertheless, it's an error and it has to be handled. I could either cache the previous queries so as to not run them more than once on the same subnet (i.e. it would query 88.220.x.x only once, and thus not get throttled), or simply choose to ignore the "throttled" json response, and get a clean output even though under the hood it's not really getting anything from PeeringDB anymore and can potentially miss a real IXP later in the trace. It's an edge case from what I can tell, so it may not be worth it, but I'm open for suggestions in case you had this happen on you more often - as I said I'm not familiar with how they "do it" in Poland, so I may have a wrong bias here :-)

Thanks for reporting!

nitefood / asn

`jq: error (at <stdin>:1): Cannot iterate over null (null)` #39