search

kellyjonbrazil / jc

CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.
MIT License
7.79k stars 198 forks source link

ifconfig parser fail to list multi line for inet6 addr #314

Closed matrixbx closed 1 year ago

matrixbx commented 1 year ago

Hi there,

am I missing something obvious ? $ ifconfig wlp1s0 | grep -c inet6 4 $ jc -p ifconfig wlp1s0 | grep -c ipv6_addr 1 $

Also didn't find an "ip" command parser yet.

jc version 1.17.3 on Ubuntu 22.04

kellyjonbrazil commented 1 year ago

Hi there! Thanks for reporting this. Could you copy/paste the ifconfig output without grep? I'd like to see if I can reproduce.

Also didn't find an "ip" command parser yet.

There are no plans to create an ip command parser since the ip command has JSON output built-in. (e.g. ip -j)

matrixbx commented 1 year ago

Hi there! Thanks for reporting this. Could you copy/paste the ifconfig output without grep? I'd like to see if I can reproduce.

Here it is. $ ifconfig wlp1s0

wlp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.11  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 2a01:cb19:171:3f00:2f3a:aca3:2e91:14c  prefixlen 64  scopeid 0x0<global>
        inet6 2a01:cb19:171:3f00:498e:4470:f88a:1e12  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::f48c:36b4:9e91:9c31  prefixlen 64  scopeid 0x20<link>
        ether f8:59:71:5c:ed:01  txqueuelen 1000  (Ethernet)
        RX packets 16053904  bytes 18526826729 (18.5 GB)
        RX errors 0  dropped 41590  overruns 0  frame 0
        TX packets 4773817  bytes 1369565952 (1.3 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
$

There are no plans to create an ip command parser since the ip command has JSON output built-in. (e.g. ip -j)

Obviously I didn't know it, I should really look the man more often, That's cool. Thank you.

kellyjonbrazil commented 1 year ago

Yep, was able to reproduce an issue here. I was expecting only the first IPv6 address to be parsed but got null instead. I'll have to see what's going on here. Thanks!

kellyjonbrazil commented 1 year ago

I'm using a third-party library to parse ifconfig into objects, so I'm not as familiar with this code, but I think I have found the issue has to do with the regular expression:

re_linux_ipv6 = re.compile(
    r"inet6 addr:\s+(?P<ipv6_addr>\S+)/(?P<ipv6_mask>[0-9]+)\s+Scope:(?P<ipv6_scope>Link|Host)",
    re.I)

The test files that I'm using are based on CentOS7, where the IPv6 line looks like this:

inet6 fe80::c1cb:715d:bc3e:b8a0  prefixlen 64  scopeid 0x20<link>

But your IPv6 line looks a little different (Ubuntu 22.04?)

inet6 2a01:cb19:171:3f00:2f3a:aca3:2e91:14c prefixlen 64 scopeid 0x0

The regex seems completely wrong so maybe it was built for older versions of linux and newer versions use more of a bsd-style, which is handled by some other regex's in the parser.

Doesn't seem like it will be difficult to fix.

Note that the ifconfig parser only pulls the first IPv4 and IPv6 address. Since the ip command has JSON output I haven't put too much effort into enhancing this parser, but I could look into grabbing all of the IPs into an array or something as an enhancement.

matrixbx commented 1 year ago

Oh, no, same format, but github hide the "<global>" at the end of the line (didn't noticed before). Edited previous post.

kellyjonbrazil commented 1 year ago

Oh, gotcha! Ok, so here's what's happening:

% echo 'wlp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.11  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 2a01:cb19:171:3f00:2f3a:aca3:2e91:14c  prefixlen 64  scopeid 0x0<global>
        inet6 2a01:cb19:171:3f00:498e:4470:f88a:1e12  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::f48c:36b4:9e91:9c31  prefixlen 64  scopeid 0x20<link>
        ether f8:59:71:5c:ed:01  txqueuelen 1000  (Ethernet)
        RX packets 16053904  bytes 18526826729 (18.5 GB)
        RX errors 0  dropped 41590  overruns 0  frame 0
        TX packets 4773817  bytes 1369565952 (1.3 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0' | jc --ifconfig -p
[
  {
    "name": "wlp1s0",
    "flags": 4163,
    "state": [
      "UP",
      "BROADCAST",
      "RUNNING",
      "MULTICAST"
    ],
    "mtu": 1500,
    "ipv4_addr": "192.168.1.11",
    "ipv4_mask": "255.255.255.0",
    "ipv4_bcast": "192.168.1.255",
    "ipv6_addr": "fe80::f48c:36b4:9e91:9c31",
    "ipv6_mask": 64,
    "ipv6_scope": "0x20",
    "mac_addr": "f8:59:71:5c:ed:01",
    "type": "Ethernet",
    "rx_packets": 16053904,
    "rx_bytes": 18526826729,
    "rx_errors": 0,
    "rx_dropped": 41590,
    "rx_overruns": 0,
    "rx_frame": 0,
    "tx_packets": 4773817,
    "tx_bytes": 1369565952,
    "tx_errors": 0,
    "tx_dropped": 0,
    "tx_overruns": 0,
    "tx_carrier": 0,
    "tx_collisions": 0,
    "metric": null
  }
]

Looks like this is working as designed and is a limitation of the ifconfig parser where it only pulls the last address of each type. Like I said, it's one of those things where the ip command gives better output and if you need that level then that's probably the way to go. If you'd prefer to use the ifconfig parser, I can look into enhancing it to put the interfaces in an array or something.

kellyjonbrazil commented 1 year ago

Maybe for now I'll add this to the ifconfig parser documentation and recommend people use the built-in JSON output functionality in the ip command if they need more detail.

matrixbx commented 1 year ago

I can go with ip on most hosts. But *BSDs don't have ip afaik.

kellyjonbrazil commented 1 year ago

I've added the limitations and recommendation to use ip in the documentation in the dev branch. I'll have to investigate to see the level of effort to include all interface IPs in the output. Thanks again for taking the time to report this!

kellyjonbrazil commented 1 year ago

Looks like I should be able to do a quick rewrite and add in multiple IPv4 and IPv6 IP address info per interface. This way I can also add more functionality to the parser and not need to rely on the 3rd party library.

matrixbx commented 1 year ago

That's a very good news! Thank you.

kellyjonbrazil commented 1 year ago

I think I have this working - just need to do a lot more testing to make sure I don't break backwards compatibility.

Basically I'm keeping the old IP fields for backwards compatibility but adding ipv4_info and ipv6_info fields that are lists of objects:

% echo 'wlp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.11  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 2a01:cb19:171:3f00:2f3a:aca3:2e91:14c  prefixlen 64  scopeid 0x0<global>
        inet6 2a01:cb19:171:3f00:498e:4470:f88a:1e12  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::f48c:36b4:9e91:9c31  prefixlen 64  scopeid 0x20<link>
        ether f8:59:71:5c:ed:01  txqueuelen 1000  (Ethernet)
        RX packets 16053904  bytes 18526826729 (18.5 GB)
        RX errors 0  dropped 41590  overruns 0  frame 0
        TX packets 4773817  bytes 1369565952 (1.3 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0' | jc --ifconfig2 -p
[
  {
    "name": "wlp1s0",
    "flags": 4163,
    "state": [
      "UP",
      "BROADCAST",
      "RUNNING",
      "MULTICAST"
    ],
    "mtu": 1500,
    "ipv4_addr": "192.168.1.11",
    "ipv4_mask": "255.255.255.0",
    "ipv4_bcast": "192.168.1.255",
    "ipv6_addr": "fe80::f48c:36b4:9e91:9c31",
    "ipv6_mask": 64,
    "ipv6_scope": "0x20",
    "type": "Ethernet",
    "mac_addr": "f8:59:71:5c:ed:01",
    "rx_packets": 16053904,
    "rx_bytes": 18526826729,
    "rx_errors": 0,
    "rx_dropped": 41590,
    "rx_overruns": 0,
    "rx_frame": 0,
    "tx_packets": 4773817,
    "tx_bytes": 1369565952,
    "tx_errors": 0,
    "tx_dropped": 0,
    "tx_overruns": 0,
    "tx_carrier": 0,
    "tx_collisions": 0,
    "ipv4_info": [
      {
        "ipv4_addr": "192.168.1.11",
        "ipv4_mask": "255.255.255.0",
        "ipv4_bcast": "192.168.1.255"
      }
    ],
    "ipv6_info": [
      {
        "ipv6_addr": "2a01:cb19:171:3f00:2f3a:aca3:2e91:14c",
        "ipv6_mask": 64,
        "ipv6_scope": "0x0",
        "type": "global"
      },
      {
        "ipv6_addr": "2a01:cb19:171:3f00:498e:4470:f88a:1e12",
        "ipv6_mask": 64,
        "ipv6_scope": "0x0",
        "type": "global"
      },
      {
        "ipv6_addr": "fe80::f48c:36b4:9e91:9c31",
        "ipv6_mask": 64,
        "ipv6_scope": "0x20",
        "type": "link"
      }
    ]
  }
]

Just noticed I can clean up the field names within ipv4_info and ipv6_info.

matrixbx commented 1 year ago

Looks good to me, thanks a lot.

kellyjonbrazil commented 1 year ago

I have a working version in the dev branch that can be used for testing: https://github.com/kellyjonbrazil/jc/blob/dev/jc/parsers/ifconfig.py

You can copy the file above and put it in your plugin directory:

Let me know if you run into any issues!

(note this version will only run with jc v1.22.1)

matrixbx commented 1 year ago

Hi, tried.

$ wget -q https://github.com/kellyjonbrazil/jc/releases/download/v1.22.1/jc-1.22.1-linux-x86_64.tar.gz
$ tar -xzf jc-1.22.1-linux-x86_64.tar.gz
$ mkdir -p ~/.local/share/jc/jcparsers
wget -q https://raw.githubusercontent.com/kellyjonbrazil/jc/dev/jc/parsers/ifconfig.py -O ~/.local/share/jc/jcparsers/ifconfig.py
$ ./jc -p ifconfig | jq '.[].ipv6[].address'
"::1"
"2a01:cb19:171:3f00:2fb0:eb5c:9de:edf3"
"fe80::f48c:36b4:9e91:9c31"
"2a01:cb19:171:3f00:498e:4470:f88a:1e12"
$

\o/

kellyjonbrazil commented 1 year ago

Nice!

kellyjonbrazil commented 1 year ago

Released in jc v1.22.2.