bergen876
commented
1 month ago Please find the attached logs preceding the crash of the A1T
2024-09-11T22:59:32.224333596Z stdout F 2024-09-11T22:59:32.223Z INFO github.com/onosproject/onos-a1t/pkg/stream stream/broker.go:129 Delete watcherID: 7eb03fb0-ecf1-48c2-a014-187e5393d67f, watchers%!(EXTRA map[stream.ID]map[uuid.UUID]chan *stream.SBStreamMessage=map[{a1:sd-ran-rimedo-ts-5bc8758c9d-s9lk6-EnrichmentInformation a1ei-controller}:map[] {a1:sd-ran-rimedo-ts-5bc8758c9d-s9lk6-PolicyManagement a1p-controller}:map[0d6d3f80-ed48-4fcd-81d0-b80a71bb0f69:0xc000578360 19e5d58b-49d8-40bd-adcf-bfac2e8895dc:0xc000202120 25c26e04-d08c-4c65-9b4a-60427d5416b6:0xc0001b4600 31cd2cb5-6495-48a3-8931-b5baf7307123:0xc00011c8a0 3995827a-7aad-4f12-8ac4-5ca9c346a663:0xc00009e360 3ecf5d9a-d9ce-4c57-a629-ebfe86bea5e7:0xc000684000 42e2bf16-d8c1-41ff-95f7-53ef1e8aba2f:0xc0001b44e0 5905cda0-a7e6-4080-9b07-9d728417d82c:0xc00011ca20 7eb03fb0-ecf1-48c2-a014-187e5393d67f:0xc00011c000 983ac871-3e8a-4136-8292-b3262aef8fe5:0xc0001b4420 a5521708-b609-415d-a919-0586954d756d:0xc0005a6060 affd5d65-0f40-4a11-82b5-a175ea7c88c0:0xc000202fc0 b4b4fcc3-6192-433a-b58e-8b3ce9cae8a7:0xc0001b4060 cb2bdf03-a11b-4df5-8327-651fd98adbd8:0xc00009e6c0 d793e62b-9f8e-415b-ab46-1387c6eb8500:0xc000202cc0 f333de37-ca40-49d7-b5cc-9928dc1e4299:0xc0005a6900] {a1ei-controller a1:sd-ran-rimedo-ts-5bc8758c9d-s9lk6-EnrichmentInformation}:map[53770b41-aed6-4fb3-82f1-b5480311ee19:0xc000578ba0] {a1p-controller a1:sd-ran-rimedo-ts-5bc8758c9d-s9lk6-PolicyManagement}:map[2b7aa5cf-bb04-48a9-b02a-adada9718cec:0xc00011c300]])
2024-09-11T22:59:32.224363397Z stdout F 2024-09-11T22:59:32.223Z INFO github.com/onosproject/onos-a1t/pkg/stream stream/broker.go:132 Deleted watcherID: 7eb03fb0-ecf1-48c2-a014-187e5393d67f,watchers%!(EXTRA map[stream.ID]map[uuid.UUID]chan *stream.SBStreamMessage=map[{a1:sd-ran-rimedo-ts-5bc8758c9d-s9lk6-EnrichmentInformation a1ei-controller}:map[] {a1:sd-ran-rimedo-ts-5bc8758c9d-s9lk6-PolicyManagement a1p-controller}:map[0d6d3f80-ed48-4fcd-81d0-b80a71bb0f69:0xc000578360 19e5d58b-49d8-40bd-adcf-bfac2e8895dc:0xc000202120 25c26e04-d08c-4c65-9b4a-60427d5416b6:0xc0001b4600 31cd2cb5-6495-48a3-8931-b5baf7307123:0xc00011c8a0 3995827a-7aad-4f12-8ac4-5ca9c346a663:0xc00009e360 3ecf5d9a-d9ce-4c57-a629-ebfe86bea5e7:0xc000684000 42e2bf16-d8c1-41ff-95f7-53ef1e8aba2f:0xc0001b44e0 5905cda0-a7e6-4080-9b07-9d728417d82c:0xc00011ca20 983ac871-3e8a-4136-8292-b3262aef8fe5:0xc0001b4420 a5521708-b609-415d-a919-0586954d756d:0xc0005a6060 affd5d65-0f40-4a11-82b5-a175ea7c88c0:0xc000202fc0 b4b4fcc3-6192-433a-b58e-8b3ce9cae8a7:0xc0001b4060cb2bdf03-a11b-4df5-8327-651fd98adbd8:0xc00009e6c0 d793e62b-9f8e-415b-ab46-1387c6eb8500:0xc000202cc0 f333de37-ca40-49d7-b5cc-9928dc1e4299:0xc0005a6900] {a1ei-controller a1:sd-ran-rimedo-ts-5bc8758c9d-s9lk6-EnrichmentInformation}:map[53770b41-aed6-4fb3-82f1-b5480311ee19:0xc000578ba0] {a1p-controller a1:sd-ran-rimedo-ts-5bc8758c9d-s9lk6-PolicyManagement}:map[2b7aa5cf-bb04-48a9-b02a-adada9718cec:0xc00011c300]])
2024-09-11T22:59:32.226533974Z stderr F panic: close of nil channel
2024-09-11T22:59:32.226691678Z stderr F
2024-09-11T22:59:32.227248567Z stderr F goroutine 2793 [running]:
2024-09-11T22:59:32.232332063Z stderr F github.com/onosproject/onos-a1t/pkg/stream.(*broker).DeleteWatcher(0xc000454ed0, {{0xc0006fc040?, 0xc0003a1e01?}, {0xf8afe4?, 0xc000212240?}}, {0x6c, 0x5f, 0x4a, 0x60, 0xf9, ...})
2024-09-11T22:59:32.233338181Z stderr F /go/src/github.com/onosproject/onos-a1t/pkg/stream/broker.go:130 +0x2f4
2024-09-11T22:59:32.240805936Z stderr F github.com/onosproject/onos-a1t/pkg/controller.waitRespMsgWithTimer({{0xc0006fc040?, 0xc000517980?}, {0xf8afe4?, 0xc0001a17d0?}}, {0x6c, 0x5f, 0x4a, 0x60, 0xf9, 0xfe, ...}, ...)
2024-09-11T22:59:32.245219909Z stderr F /go/src/github.com/onosproject/onos-a1t/pkg/controller/utils.go:34 +0x35a
2024-09-11T22:59:32.25040561Z stderr F created by github.com/onosproject/onos-a1t/pkg/controller.(*a1pController).HandleGetPolicytypesPolicyTypeIdPolicies
2024-09-11T22:59:32.251427928Z stderr F /go/src/github.com/onosproject/onos-a1t/pkg/controller/a1p.go:447 +0x825
Issue Summary –
Issue: Our tests revealed that the ONOS A1 Termination component of SD RAN ORAN deployment was susceptible to a Denial of Service attack through the continuous replay of concurrent HTTP policies. This attack resulted in the termination of the service, impacting the management of A1 policies.
Logs Overview: The strace logs show a panic was triggered during a sequence of epoll_pwait and futex operations, which can hint about thread synchronization issues. Multiple write system calls were noted, corresponding to log outputs of watcher deletions. This aligns with the A1 Termination service logs, which indicate a panic when attempting to close a nil channel in the DeleteWatcher method.
Attack Reproduction : We tested the SDRAN-in-a-Box (RiaB) v1.4.3, specifically the RAN Simulator and Rimedo Traffic Steering xApp edition ( https://docs.sd-ran.org/master/sdran-in-a-box/docs/Installation_RANSim_RIMDEO_TS.html ). However, we believe this attack is applicable to all other editions that utilize the AI Termination module. The attack consists of replaying the JSON-based Traffic Steering Policies (ORAN_TrafficSteeringPreference_2.0.0) by scripting a multithreaded curl command.
Technical Details:
Strace Logs:
A1 Termination Logs:
The error originated in the DeleteWatcher function (stream/broker.go:129-132) with a panic message: "close of nil channel" at broker.go:130. onos-a1t/pkg/stream/broker.go at master · onosproject/onos-a1t · GitHub
The goroutine in question (goroutine 2793) was active during the panic, attempting to delete a watcher from a map with UUIDs tied to components like EnrichmentInformation and PolicyManagement.
Potential Causes:
The nil channel panic suggests a potential logic flaw where the code attempts to close a watcher channel that is either not properly initialized or has already been closed.
Thread synchronization issues as indicated by futex behavior could be causing a race condition, leading to the panic during the watcher deletion.
Next Steps: