flowd_aggregate died with message Traceback (...) KeyError: 'dst_port'

[ericschdt]

Important notices [x] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md [x] I have searched the existing issues and I'm convinced that mine is new. [x] The title contains the plugin to which this issue belongs

Describe the bug Basically the same as https://github.com/opnsense/core/issues/4735 I changed Netflow v9 to v5 and then pressed on the "refresh button" in the "service" widget on the lobby page, which resulted in

2021-02-25T11:30:58     flowd aggregate died with message Traceback (most recent call last): File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 160, in run aggregate_flowd(self.config, do_vacuum) File "/usr/local/opnsense/scripts/netflow/flowd_aggregate.py", line 80, in aggregate_flowd stream_agg_object.add(copy.copy(flow_record)) File "/usr/local/opnsense/scripts/netflow/lib/aggregates/source.py", line 107, in add flow['service_port'] = min(flow['dst_port'], flow['src_port']) KeyError: 'dst_port'
2021-02-25T11:30:58     flowparser failed to unpack flow_engine_info (unpack requires a buffer of 12 bytes)
2021-02-25T11:30:58     flowparser failed to unpack as_info (unpack requires a buffer of 12 bytes)
2021-02-25T11:30:58     flowparser failed to unpack flow_times (unpack requires a buffer of 8 bytes)
2021-02-25T11:30:58     flowparser failed to unpack agent_info (unpack requires a buffer of 16 bytes)
2021-02-25T11:30:58     flowparser failed to unpack if_indices (unpack requires a buffer of 8 bytes)
2021-02-25T11:30:58     flowparser failed to unpack srcdst_port (unpack requires a buffer of 4 bytes)
2021-02-25T11:30:53     event @ 1614249053.01 exec: system event config_changed
2021-02-25T11:30:53     event @ 1614249053.01 msg: Feb 25 11:30:53 OPNsense.localdomain config[90487]: config-event: new_config /conf/backup/config-1614249053.0081.xml
2021-02-25T11:29:32     start watching flowd
2021-02-25T11:29:25     startup, check database.

Location of the error https://github.com/opnsense/core/blob/de291e62660996b04d589381cad3396db5e9eb80/src/opnsense/scripts/netflow/lib/aggregates/source.py#L107

Environment OPNsense 21.1.2-amd64 FreeBSD 12.1-RELEASE-p13-HBSD OpenSSL 1.1.1j 16 Feb 2021

[ericschdt]

I guess it would sense if all variables do have meaningful initial/default values. Maybe unit tests would also help to discover those issues? :)

[AdSchellevis]

you're right, added them in https://github.com/opnsense/core/commit/43b50ed086ad34050a42f1afd0a1437f35693e9b