search

thombashi / tcconfig

A tc command wrapper. Make it easy to set up traffic control of network bandwidth/latency/packet-loss/packet-corruption/etc. to a network-interface/Docker-container(veth).
https://tcconfig.rtfd.io/
MIT License
792 stars 85 forks source link

Crash in tcshow Filter Parsing #160

Closed personalcomputer closed 2 years ago

personalcomputer commented 2 years ago

Describe the bug

Hi Hombashi-san, thanks for publishing tcconfig. It makes tc a lot more approachable. I have found a crash when parsing filters in tcshow:

$ sudo tcshow wlp3s0
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/core.py", line 663, in insert_many
    self.connection.executemany(query, records)
sqlite3.IntegrityError: NOT NULL constraint failed: filter.device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tcshow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/tcshow.py", line 212, in main
    tc_params = extract_tc_params(options)
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/tcshow.py", line 188, in extract_tc_params
    rule_parser.parse()
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/shaping_rule.py", line 105, in parse
    self.__parse_device(self.ifb_device)
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/shaping_rule.py", line 115, in __parse_device
    self.__parse_tc_filter(device)
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/shaping_rule.py", line 293, in __parse_tc_filter
    self.__filter_parser.parse(
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/_filter.py", line 139, in parse
    Filter.insert(self.__get_filter())
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/model.py", line 226, in insert
    cls.__connection.insert(cls.get_table_name(), record)
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/core.py", line 594, in insert
    self.insert_many(table_name, records=[record], attr_names=attr_names)
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/core.py", line 667, in insert_many
    raise OperationalError(
simplesqlite.error.OperationalError: /usr/local/lib/python3.8/dist-packages/simplesqlite/model.py(226) insert: failed to execute query:
  query=INSERT INTO filter(device,"filter_id",flowid,protocol,priority,[src-network],[dst-network],[src-port],[dst-port],classid,handle) VALUES (?,?,?,?,?,?,?,?,?,?,?)
  msg='NOT NULL constraint failed: filter.device'
  db=:memory:
  records=[[None, '800::801', '1bc4:3', 'ipv6', 6, '0.0.0.0/0', '0.0.0.0/0', None, None, None, None]]

To Reproduce

The simplest way to reproduce that I have found is by running the below two tcset commands in sequence, and then tcshow as third command. These two tcset commands are trying to set up a 100ms delay on both ipv4 and ipv6, so quite a normal usecase I think.

$ sudo tcset --direction incoming --delay 100  wlp3s0 
$ sudo tcset --direction incoming --delay 100  wlp3s0 --ipv6 --add
[ERROR] command execution failed
  command=/usr/sbin/tc qdisc add dev wlp3s0 ingress
  stderr=Error: Exclusivity flag on, cannot modify.

[ERROR] command execution failed
  command=/usr/sbin/tc qdisc add dev ifb7108 root handle 1bc4: htb default 1
  stderr=Error: Exclusivity flag on, cannot modify.
$ sudo tcshow wlp3s0
[... crash ...]

(note: 2nd command gives an error message, but does not crash. Crash only happens when running tcshow.)

Environments

Docker version: I was not using tcconfig in combination with docker. Installation method: pip Module Version
uname Linux 5.4.0-90-generic
Python CPython 3.8.10
platform Ubuntu 20.04
tcconfig 0.26.0
docker 4.3.1
SimpleSQLite 1.1.3

Additional context

Full --debug output:

$ sudo tcshow --debug wlp3s0 

DEBUG    | simplesqlite.core:connect:266 - connect to a SQLite database: path=':memory:', mode=w
DEBUG    | simplesqlite.core:create_table:1277 - CREATE TABLE IF NOT EXISTS 'filter' (device TEXT NOT NULL, "filter_id" TEXT, flowid TEXT, protocol TEXT, priority INTEGER, [src-network] TEXT, [dst-network] TEXT, [src-port] INTEGER, [dst-port] INTEGER, classid TEXT, handle INTEGER)
DEBUG    | simplesqlite.core:create_table:1277 - CREATE TABLE IF NOT EXISTS 'qdisc' (device TEXT NOT NULL, "direct_qlen" INTEGER, parent TEXT, handle TEXT, delay TEXT, [delay-distro] TEXT, loss TEXT, duplicate TEXT, corrupt TEXT, reorder TEXT, rate TEXT)
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc filter show dev wlp3s0 root
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc class show dev wlp3s0
DEBUG    | tcconfig.parser._class:parse:57 - tc class parse result: []
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc filter show dev wlp3s0
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc qdisc show dev wlp3s0
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc class show dev ifb7108
DEBUG    | tcconfig.parser._class:parse:51 - parse a class entry: {'device': 'ifb7108', 'classid': '1bc4:1', 'rate': '32Gbps'}
DEBUG    | tcconfig.parser._class:parse:51 - parse a class entry: {'device': 'ifb7108', 'classid': '1bc4:2', 'rate': '32Gbps'}
DEBUG    | tcconfig.parser._class:parse:51 - parse a class entry: {'device': 'ifb7108', 'classid': '1bc4:3', 'rate': '32Gbps'}
DEBUG    | simplesqlite.core:__create_table_from_tabledata:1726 - __create_table_from_tabledata:
    tbldata=table_name=class, headers=[device, classid, rate], cols=3, rows=3
DEBUG    | tabledata.normalizer:normalize:53 - normalize: SQLiteTableDataSanitizer
DEBUG    | dataproperty._extractor:to_dp_matrix:453 - max_workers=1, preprocessor=strip_str=None, replace_tabs_with_spaces=True, tab_length=2, line_break_handling=LineBreakHandling.NOP, line_break_repl= , escape_html_tag=False, escape_formula_injection=False
DEBUG    | dataproperty._extractor:to_column_dp_list:387 - converting to column dataproperty:
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -   params:
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -     headers=3
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -     prev_col_count=None
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -     matrix_formatting=MatrixFormatting.TRIM
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -     column_type_hints=()
DEBUG    | dataproperty._extractor:to_column_dp_list:417 -   results:
DEBUG    | dataproperty._extractor:to_column_dp_list:447 -     column=0, type=STRING, align=left, ascii_width=7
DEBUG    | dataproperty._extractor:to_column_dp_list:447 -     column=1, type=STRING, align=left, ascii_width=7
DEBUG    | dataproperty._extractor:to_column_dp_list:447 -     column=2, type=STRING, align=left, ascii_width=6
DEBUG    | simplesqlite.core:create_table:1277 - CREATE TABLE IF NOT EXISTS 'class' (device TEXT, classid TEXT, rate TEXT)
DEBUG    | simplesqlite.core:insert_many:631 - insert 3 records into class
DEBUG    | simplesqlite.core:commit:1579 - commit: path=':memory:'
DEBUG    | tcconfig.parser._class:parse:57 - tc class parse result: [
    {
        "device": "ifb7108",
        "classid": "1bc4:1",
        "rate": "32Gbps"
    },
    {
        "device": "ifb7108",
        "classid": "1bc4:2",
        "rate": "32Gbps"
    },
    {
        "device": "ifb7108",
        "classid": "1bc4:3",
        "rate": "32Gbps"
    }
]
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc filter show dev ifb7108
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ip pref 5 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: filter parent 1bc4: protocol ip pref 5 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:136 - failed to parse filter: filter parent 1bc4: protocol ip pref 5 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:136 - failed to parse filter: filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_flow_id:183 - succeed to parse flow id: flow-id=1bc4:2, line=filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_protocol:188 - succeed to parse protocol: protocol=ip, line=filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_priority:195 - succeed to parse priority: priority=5, line=filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_filter_id:202 - succeed to parse filter id: filter-id=800::800, line=filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: match 00000000/00000000 at 16
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: match 00000000/00000000 at 16
DEBUG    | tcconfig.parser._filter:__parse_filter_ipv4:354 - succeed to parse ipv4 filter: src_network=None, dst_network=0.0.0.0/0, src_port=None, dst_port=None, line=match 00000000/00000000 at 16
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: match 00000000/00000000 at 12
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: match 00000000/00000000 at 12
DEBUG    | tcconfig.parser._filter:__parse_filter_ipv4:354 - succeed to parse ipv4 filter: src_network=0.0.0.0/0, dst_network=0.0.0.0/0, src_port=None, dst_port=None, line=match 00000000/00000000 at 12
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:136 - failed to parse filter: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:136 - failed to parse filter: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_flow_id:183 - succeed to parse flow id: flow-id=1bc4:3, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_protocol:188 - succeed to parse protocol: protocol=ipv6, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_priority:195 - succeed to parse priority: priority=6, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_filter_id:202 - succeed to parse filter id: filter-id=801::800, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:parse:115 - store filter: Filter (device=ifb7108, filter_id=800::800, flowid=1bc4:2, protocol=ip, priority=5, src-network=0.0.0.0/0, dst-network=0.0.0.0/0)
DEBUG    | simplesqlite.core:insert_many:631 - insert 1 records into filter
DEBUG    | tcconfig.parser._filter:__parse_flow_id:183 - succeed to parse flow id: flow-id=1bc4:3, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_protocol:188 - succeed to parse protocol: protocol=ipv6, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_priority:195 - succeed to parse priority: priority=6, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_filter_id:202 - succeed to parse filter id: filter-id=801::800, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | simplesqlite.core:insert_many:631 - insert 1 records into filter
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/core.py", line 663, in insert_many
    self.connection.executemany(query, records)
sqlite3.IntegrityError: NOT NULL constraint failed: filter.device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tcshow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/tcshow.py", line 212, in main
    tc_params = extract_tc_params(options)
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/tcshow.py", line 188, in extract_tc_params
    rule_parser.parse()
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/shaping_rule.py", line 105, in parse
    self.__parse_device(self.ifb_device)
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/shaping_rule.py", line 115, in __parse_device
    self.__parse_tc_filter(device)
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/shaping_rule.py", line 293, in __parse_tc_filter
    self.__filter_parser.parse(
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/_filter.py", line 139, in parse
    Filter.insert(self.__get_filter())
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/model.py", line 226, in insert
    cls.__connection.insert(cls.get_table_name(), record)
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/core.py", line 594, in insert
    self.insert_many(table_name, records=[record], attr_names=attr_names)
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/core.py", line 667, in insert_many
    raise OperationalError(
simplesqlite.error.OperationalError: /usr/local/lib/python3.8/dist-packages/simplesqlite/model.py(226) insert: failed to execute query:
  query=INSERT INTO filter(device,"filter_id",flowid,protocol,priority,[src-network],[dst-network],[src-port],[dst-port],classid,handle) VALUES (?,?,?,?,?,?,?,?,?,?,?)
  msg='NOT NULL constraint failed: filter.device'
  db=:memory:
  records=[[None, '801::800', '1bc4:3', 'ipv6', 6, '0.0.0.0/0', '0.0.0.0/0', None, None, None, None]]

john@x1:~$ sudo tcshow wlp3s0 --debug | copy
DEBUG    | simplesqlite.core:connect:266 - connect to a SQLite database: path=':memory:', mode=w
DEBUG    | simplesqlite.core:create_table:1277 - CREATE TABLE IF NOT EXISTS 'filter' (device TEXT NOT NULL, "filter_id" TEXT, flowid TEXT, protocol TEXT, priority INTEGER, [src-network] TEXT, [dst-network] TEXT, [src-port] INTEGER, [dst-port] INTEGER, classid TEXT, handle INTEGER)
DEBUG    | simplesqlite.core:create_table:1277 - CREATE TABLE IF NOT EXISTS 'qdisc' (device TEXT NOT NULL, "direct_qlen" INTEGER, parent TEXT, handle TEXT, delay TEXT, [delay-distro] TEXT, loss TEXT, duplicate TEXT, corrupt TEXT, reorder TEXT, rate TEXT)
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc filter show dev wlp3s0 root
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc class show dev wlp3s0
DEBUG    | tcconfig.parser._class:parse:57 - tc class parse result: []
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc filter show dev wlp3s0
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc qdisc show dev wlp3s0
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc class show dev ifb7108
DEBUG    | tcconfig.parser._class:parse:51 - parse a class entry: {'device': 'ifb7108', 'classid': '1bc4:1', 'rate': '32Gbps'}
DEBUG    | tcconfig.parser._class:parse:51 - parse a class entry: {'device': 'ifb7108', 'classid': '1bc4:2', 'rate': '32Gbps'}
DEBUG    | tcconfig.parser._class:parse:51 - parse a class entry: {'device': 'ifb7108', 'classid': '1bc4:3', 'rate': '32Gbps'}
DEBUG    | simplesqlite.core:__create_table_from_tabledata:1726 - __create_table_from_tabledata:
    tbldata=table_name=class, headers=[device, classid, rate], cols=3, rows=3
DEBUG    | tabledata.normalizer:normalize:53 - normalize: SQLiteTableDataSanitizer
DEBUG    | dataproperty._extractor:to_dp_matrix:453 - max_workers=1, preprocessor=strip_str=None, replace_tabs_with_spaces=True, tab_length=2, line_break_handling=LineBreakHandling.NOP, line_break_repl= , escape_html_tag=False, escape_formula_injection=False
DEBUG    | dataproperty._extractor:to_column_dp_list:387 - converting to column dataproperty:
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -   params:
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -     headers=3
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -     prev_col_count=None
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -     matrix_formatting=MatrixFormatting.TRIM
DEBUG    | dataproperty._extractor:to_column_dp_list:415 -     column_type_hints=()
DEBUG    | dataproperty._extractor:to_column_dp_list:417 -   results:
DEBUG    | dataproperty._extractor:to_column_dp_list:447 -     column=0, type=STRING, align=left, ascii_width=7
DEBUG    | dataproperty._extractor:to_column_dp_list:447 -     column=1, type=STRING, align=left, ascii_width=7
DEBUG    | dataproperty._extractor:to_column_dp_list:447 -     column=2, type=STRING, align=left, ascii_width=6
DEBUG    | simplesqlite.core:create_table:1277 - CREATE TABLE IF NOT EXISTS 'class' (device TEXT, classid TEXT, rate TEXT)
DEBUG    | simplesqlite.core:insert_many:631 - insert 3 records into class
DEBUG    | simplesqlite.core:commit:1579 - commit: path=':memory:'
DEBUG    | tcconfig.parser._class:parse:57 - tc class parse result: [
    {
        "device": "ifb7108",
        "classid": "1bc4:1",
        "rate": "32Gbps"
    },
    {
        "device": "ifb7108",
        "classid": "1bc4:2",
        "rate": "32Gbps"
    },
    {
        "device": "ifb7108",
        "classid": "1bc4:3",
        "rate": "32Gbps"
    }
]
DEBUG    | subprocrunner._subprocess_runner:__debug_print_command:264 - /usr/sbin/tc filter show dev ifb7108
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ip pref 5 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: filter parent 1bc4: protocol ip pref 5 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:136 - failed to parse filter: filter parent 1bc4: protocol ip pref 5 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:136 - failed to parse filter: filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_flow_id:183 - succeed to parse flow id: flow-id=1bc4:2, line=filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_protocol:188 - succeed to parse protocol: protocol=ip, line=filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_priority:195 - succeed to parse priority: priority=5, line=filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_filter_id:202 - succeed to parse filter id: filter-id=800::800, line=filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: match 00000000/00000000 at 16
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: match 00000000/00000000 at 16
DEBUG    | tcconfig.parser._filter:__parse_filter_ipv4:354 - succeed to parse ipv4 filter: src_network=None, dst_network=0.0.0.0/0, src_port=None, dst_port=None, line=match 00000000/00000000 at 16
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: match 00000000/00000000 at 12
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: match 00000000/00000000 at 12
DEBUG    | tcconfig.parser._filter:__parse_filter_ipv4:354 - succeed to parse ipv4 filter: src_network=0.0.0.0/0, dst_network=0.0.0.0/0, src_port=None, dst_port=None, line=match 00000000/00000000 at 12
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:136 - failed to parse filter: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:126 - failed to parse flow id: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:136 - failed to parse filter: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801: ht divisor 1
DEBUG    | tcconfig.parser._filter:parse:92 - failed to parse mangle: filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_flow_id:183 - succeed to parse flow id: flow-id=1bc4:3, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_protocol:188 - succeed to parse protocol: protocol=ipv6, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_priority:195 - succeed to parse priority: priority=6, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_filter_id:202 - succeed to parse filter id: filter-id=801::800, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:parse:115 - store filter: Filter (device=ifb7108, filter_id=800::800, flowid=1bc4:2, protocol=ip, priority=5, src-network=0.0.0.0/0, dst-network=0.0.0.0/0)
DEBUG    | simplesqlite.core:insert_many:631 - insert 1 records into filter
DEBUG    | tcconfig.parser._filter:__parse_flow_id:183 - succeed to parse flow id: flow-id=1bc4:3, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_protocol:188 - succeed to parse protocol: protocol=ipv6, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_priority:195 - succeed to parse priority: priority=6, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | tcconfig.parser._filter:__parse_filter_id:202 - succeed to parse filter id: filter-id=801::800, line=filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw
DEBUG    | simplesqlite.core:insert_many:631 - insert 1 records into filter
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/core.py", line 663, in insert_many
    self.connection.executemany(query, records)
sqlite3.IntegrityError: NOT NULL constraint failed: filter.device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tcshow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/tcshow.py", line 212, in main
    tc_params = extract_tc_params(options)
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/tcshow.py", line 188, in extract_tc_params
    rule_parser.parse()
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/shaping_rule.py", line 105, in parse
    self.__parse_device(self.ifb_device)
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/shaping_rule.py", line 115, in __parse_device
    self.__parse_tc_filter(device)
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/shaping_rule.py", line 293, in __parse_tc_filter
    self.__filter_parser.parse(
  File "/usr/local/lib/python3.8/dist-packages/tcconfig/parser/_filter.py", line 139, in parse
    Filter.insert(self.__get_filter())
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/model.py", line 226, in insert
    cls.__connection.insert(cls.get_table_name(), record)
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/core.py", line 594, in insert
    self.insert_many(table_name, records=[record], attr_names=attr_names)
  File "/usr/local/lib/python3.8/dist-packages/simplesqlite/core.py", line 667, in insert_many
    raise OperationalError(
simplesqlite.error.OperationalError: /usr/local/lib/python3.8/dist-packages/simplesqlite/model.py(226) insert: failed to execute query:
  query=INSERT INTO filter(device,"filter_id",flowid,protocol,priority,[src-network],[dst-network],[src-port],[dst-port],classid,handle) VALUES (?,?,?,?,?,?,?,?,?,?,?)
  msg='NOT NULL constraint failed: filter.device'
  db=:memory:
  records=[[None, '801::800', '1bc4:3', 'ipv6', 6, '0.0.0.0/0', '0.0.0.0/0', None, None, None, None]]

Output of what I think it is failing to parse:

$ tc filter show dev ifb7108
filter parent 1bc4: protocol ip pref 5 u32 chain 0 
filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800: ht divisor 1 
filter parent 1bc4: protocol ip pref 5 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1bc4:2 not_in_hw 
  match 00000000/00000000 at 16
  match 00000000/00000000 at 12
filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 
filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801: ht divisor 1 
filter parent 1bc4: protocol ipv6 pref 6 u32 chain 0 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1bc4:3 not_in_hw

Output of what the sqlite3 database looks like at the time of the crash:

$ sqlite3 /tmp/tcshow.sqlite3 'select * from filter' -column -header
device      filter_id   flowid      protocol    priority    src-network  dst-network  src-port    dst-port    classid     handle    
----------  ----------  ----------  ----------  ----------  -----------  -----------  ----------  ----------  ----------  ----------
ifb7108     800::800    1bc4:2      ip          5           0.0.0.0/0    0.0.0.0/0
thombashi commented 2 years ago

@personalcomputer Hi,

Thank you for your detailed report. It would be a great help.

I had fixed the problem with c79d4f2de05a0cdfa94b20a6cb8b7a78699d6126. The commit will be included in the next release (probably soon).

thombashi commented 2 years ago

Released tcconfig 0.27.1. Please try to upgrade this version.

Feel free to reopen if you still have any problems about the issue.

personalcomputer commented 2 years ago

Thanks @thombashi! I just tested, and I can confirm you've fixed this bug now in 0.27.1. :pray: