Unable to load data from Redis

[dansmith-it]

Describe the bug

The icingadb process crashes upon importing existing data, the output is a bit chaotic, but the main problem seems be the ID of incident containing the "minus" character = "d03e4381-4376-4e60-80a5-0e1c3e57cb34"

encoding/hex: invalid byte: U+002D '-'#012can't decode hex "d03e4381-4376-4e60-80a5-0e1c3e57cb34"#012github.com/icinga/icingadb/internal.CantDecodeHex#012#011/builds/packaging/deb-icingadb/build/icingadb/internal/internal.go:10#012github.com/icinga/icingadb/pkg/types.(Binary).UnmarshalText#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/types/binary.go:56#012github.com/icinga/icingadb/pkg/structify.parseString#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:113#012github.com/icinga/icingadb/pkg/structify.structifyMapByTree#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:85#012github.com/icinga/icingadb/pkg/structify.structifyMapByTree#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:101#012github.com/icinga/icingadb/pkg/structify.structifyMapByTree#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:101#012github.com/icinga/icingadb/pkg/structify.structifyMapByTree#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:101#012github.com/icinga/icingadb/pkg/structify.MakeMapStructifier.func1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:41#012github.com/icinga/icingadb/pkg/icingadb/history.writeOneEntityStage.func1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/history/sync.go:182#012github.com/icinga/icingadb/pkg/icingadb/history.writeMultiEntityStage.func1.1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/history/sync.go:219#012golang.org/x/sync/errgroup.(Group).Go.func1#012#011/home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57#012runtime.goexit#012#011/usr/local/go/src/runtime/asm_amd64.s:1371#012can't parse id into the Binary NotificationHistory#HistoryTableEntity.EntityWithoutChecksum.IdMeta.Id: d03e4381-4376-4e60-80a5-0e1c3e57cb34#012github.com/icinga/icingadb/pkg/structify.structifyMapByTree#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:96#012github.com/icinga/icingadb/pkg/structify.structifyMapByTree#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:101#012github.com/icinga/icingadb/pkg/structify.structifyMapByTree#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:101#012github.com/icinga/icingadb/pkg/structify.structifyMapByTree#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:101#012github.com/icinga/icingadb/pkg/structify.MakeMapStructifier.func1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:41#012github.com/icinga/icingadb/pkg/icingadb/history.writeOneEntityStage.func1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/history/sync.go:182#012github.com/icinga/icingadb/pkg/icingadb/history.writeMultiEntityStage.func1.1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/history/sync.go:219#012golang.org/x/sync/errgroup.(Group).Go.func1#012#011/home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57#012runtime.goexit#012#011/usr/local/go/src/runtime/asm_amd64.s:1371#012can't structify map map[string]interface {}{"author":"", "endpoint_id":"635dac26cec9c93b615de46212d3894cb0153148", "environment_id":"da39a3ee5e6b4b0d3255bfef95601890afd80709", "event_id":"ba045b20-b166-4b08-999c-813ee9d13203", "event_type":"notification", "host_id":"e01ef43de4eda2c392218d6dcf2050a904360c39", "id":"d03e4381-4376-4e60-80a5-0e1c3e57cb34", "notification_id":"e3d78ac961b8a040654095977be4656263fa2174", "object_type":"service", "previous_hard_state":"2", "send_time":"1636829811643", "service_id":"ab891626aa821cc1e032e960ec48464bfe7665e0", "state":"1", "text":"WARNING", "type":"32", "users_notified":"1"} by tree []structify.structBranch{structify.structBranch{field:0, leaf:"", subTree:[]structify.structBranch{structify.structBranch{field:0, leaf:"", subTree:[]structify.structBranch{structify.structBranch{field:0, leaf:"", subTree:[]structify.structBranch{structify.structBranch{field:0, leaf:"id", subTree:[]structify.structBranch(nil)}}}}}}}, structify.structBranch{field:1, leaf:"", subTree:[]structify.structBranch{structify.structBranch{field:0, leaf:"environment_id", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:1, leaf:"endpoint_id", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:2, leaf:"object_type", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:3, leaf:"host_id", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:4, leaf:"service_id", subTree:[]structify.structBranch(nil)}}}, structify.structBranch{field:2, leaf:"notification_id", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:3, leaf:"type", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:4, leaf:"send_time", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:5, leaf:"state", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:6, leaf:"previous_hard_state", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:7, leaf:"author", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:8, leaf:"text", subTree:[]structify.structBranch(nil)}, structify.structBranch{field:9, leaf:"users_notified", subTree:[]structify.structBranch(nil)}}#012github.com/icinga/icingadb/pkg/structify.MakeMapStructifier.func1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/structify/structify.go:41#012github.com/icinga/icingadb/pkg/icingadb/history.writeOneEntityStage.func1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/history/sync.go:182#012github.com/icinga/icingadb/pkg/icingadb/history.writeMultiEntityStage.func1.1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/history/sync.go:219#012golang.org/x/sync/errgroup.(Group).Go.func1#012#011/home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57#012runtime.goexit#012#011/usr/local/go/src/runtime/asm_amd64.s:1371#012can't structify values map[string]interface {}{"author":"", "endpoint_id":"635dac26cec9c93b615de46212d3894cb0153148", "environment_id":"da39a3ee5e6b4b0d3255bfef95601890afd80709", "event_id":"ba045b20-b166-4b08-999c-813ee9d13203", "event_type":"notification", "host_id":"e01ef43de4eda2c392218d6dcf2050a904360c39", "id":"d03e4381-4376-4e60-80a5-0e1c3e57cb34", "notification_id":"e3d78ac961b8a040654095977be4656263fa2174", "object_type":"service", "previous_hard_state":"2", "send_time":"1636829811643", "service_id":"ab891626aa821cc1e032e960ec48464bfe7665e0", "state":"1", "text":"WARNING", "type":"32", "users_notified":"1"}#012github.com/icinga/icingadb/pkg/icingadb/history.writeOneEntityStage.func1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/history/sync.go:184#012github.com/icinga/icingadb/pkg/icingadb/history.writeMultiEntityStage.func1.1#012#011/builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/history/sync.go:219#012golang.org/x/sync/errgroup.(*Group).Go.func1#012#011/home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57#012runtime.goexit#012#011/usr/local/go/src/runtime/asm_amd64.s:1371

Expected behavior

Not crashing ;-)

Your Environment

• Icinga DB version: RC2
• Icinga 2 version: 2.13.2-1.buster
• Operating System and version: debian buster

[lippserd]

Hi,

What do you mean with existing data? The RC2 requires Redis and the database to be clean.

All the best, Eric

[dansmith-it]

Hello, yes the redis is clean, the mysql icingadb database is created as new, but the icinga2 itself runs for about 5 years. And upon start it feeds to the redis current state, where the ids of object have dashes in ids, like for Downtime

object Downtime "5803501f-e9c1-40fe-9ed3-5aa7b4ec5e81" ignore_on_error {

or Notification object in the crash dump, where is

"notification_id":"e3d78ac961b8a040654095977be4656263fa2174" parsed fine but "id":"d03e4381-4376-4e60-80a5-0e1c3e57cb34" and "event_id":"ba045b20-b166-4b08-999c-813ee9d13203" not parsable. As far as I understand it, the icinga2 itself writes this objects (and objects ids) into the redis and icingadb reads the redis and converts the entries to records in mysql icingadb database, which is not happening, the mysql icingadb database is still empty and the icingadb process crashes. I tried to remove some objects from icinga2 (like all comments which aren't needed and was the first object to crash the icinadb) but its never ending story.

[dansmith-it]

Another example which crashes the icingadb:

can't structify values map[string]interface {}{"attempt":"1", "check_source":"some_icinga_server", "endpoint_id":"635dac26cec9c93b615de46212d3894cb0153148", "environment_id":"da39a3ee5e6b4b0d3255bfef95601890afd80709", "event_id":"e6209fff-8630-4a2e-8256-f6ea8e9fd866", "event_time":"1636824979065", "event_type":"state_change", "hard_state":"0", "host_id":"96c2100a2bfc89b1f8c1810b20781a50c872e56f", "id":"4c708683-f3e9-4886-bacf-246396aa3719", "max_check_attempts":"5", "object_type":"service", "output":"CRITICAL - *****: rta 68.198ms, lost 20%", "previous_hard_state":"0", "previous_soft_state":"0", "service_id":"0db3a8030889033470fe7411b257fc4286958713", "soft_state":"2", "state_type":"0"}

[lippserd]

Is there a chance that this is a cluster setup and not all of the Icinga nodes writing to Redis have been updated to version 2.13.2? Because the events that cause Icinga DB to crash are from an older Icinga version. How was Icinga updated?

[RincewindsHat]

@dansmith-it: Same error for me, might be an artefact of an older icingadb installation. Vanished after: redis-cli FLUSHALL

[dansmith-it]

You are right not all cluster nodes were upgraded, I expected rollback and would't do it everywhere. But now it seems be working fine (after upgrade of all nodes). So the ticket can be closed, sorry about the this.

[lippserd]

You are right not all cluster nodes were upgraded, I expected rollback and would't do it everywhere. But now it seems be working fine (after upgrade of all nodes). So the ticket can be closed, sorry about the this.

Don't worry, I'm glad it works now.