search

pmacct / pmacct

pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].
http://www.pmacct.net
Other
1.06k stars 264 forks source link

Recent changes causing Avro schema issues #807

Closed sgaragan closed 1 month ago

sgaragan commented 2 months ago

We tried running with the most recent changes on main to test out the ingress/egress VRF name functionality. When pmacct came up, we saw errors like this in the logs and pmacct stopped consuming/producing messages for BMP/BGP as well as IPFIX. YANG push telemetry continued to be processed though

The error we saw was:

2024-08-13T19:20:35Z WARN ( default/core ): serdes_schema_add() failed: REST request failed (code 409): {"error_code":409,"message":"Schema being registered is incompatible with an earlier schema for subject \"XXXX_msglog_rm\", details: [{errorType:'READER_FIELD_MISSING_DEFAULT_VALUE', description:'The field 'tag' at path '/fields/5' in the new schema has no default value and is missing in the old schema', additionalInfo:'tag'}, {errorType:'READER

When we switched back to a build from main on August 1, things went back to normal and pmacct started collecting BMP/BGP and IPFIX again

sgaragan commented 2 months ago

One thing I realized I forgot to add when I redacted this is that the issue was with schemas for '-bmp_msglog_rm'

paololucente commented 2 months ago

Hi Sean ( @sgaragan ),

To help narrowing down the issue can you please test: a681b10802753183c602457a7be8781119103bfc , 7b3ad465770ef750a6e8476223bc1fe51a23c2f0 and 3450d74b7398c2c0ec8863dbc0b65c076104953c ?

Paolo

sgaragan commented 2 months ago

Hey Paolo,

Looks like the error happens when I sync to commit 49796d2 which includes a681b10.

Thanks, Sean

2024-08-16T14:44:13Z WARN ( default/core ): serdes_schema_add() failed: REST request failed (code 409): {"error_code":409,"message":"Schema being registered is incompatible with an earlier schema for subject \"XXXX_msglog_rm\", details: [{errorType:'READER_FIELD_MISSING_DEFAULT_VALUE', description:'The field 'tag' at path '/fields/5' in the new schema has no default value and is missing in the old schema', additionalInfo:'tag'}, {errorType:'READER

rodonile commented 2 months ago

Hi Sean, I'm guessing you need to enable forward schema compatibiliy for the relevant subjects in the schema registry. If I remember correctly by default it is only backwards compatible, which means no new fields can be added. I would just configure NONE compatibility for dev environments. Reference: https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html#compatibility-types