Please explain error: build-id: no sub meta 'id_format'

[kittinan]

I load test my SSE Application and i got the error error: build-id: no sub meta. it happens when there is high load traffic.

• 60+ messages per second and 200 users consume SSE
• Tested on Pushpin 1.37 and 1.40 / Ubuntu 22.04
• django-eventstream 4.5.1 with custom storage model (Redis)

# grep -ir "sub meta" *

connmgr.log:[TRACE] 2024-09-13 20:58:10.826 [pushpin::connmgr::zhttpsocket] IN stream connmgr { "seq": 3587, "from": "handler_3132005", "id": "0-14-9b" } 50 "\n\nfilter error: build-id: no sub meta \'id_format\'\n"
connmgr.log:[TRACE] 2024-09-13 21:02:49.213 [pushpin::connmgr::zhttpsocket] IN stream connmgr { "seq": 4851, "from": "handler_3132005", "id": "0-14-9f" } 50 "\n\nfilter error: build-id: no sub meta \'id_format\'\n"
connmgr.log:[TRACE] 2024-09-13 21:06:07.086 [pushpin::connmgr::zhttpsocket] IN stream connmgr { "seq": 3437, "from": "handler_3132005", "id": "0-14-107" } 50 "\n\nfilter error: build-id: no sub meta \'id_format\'\n"
pushpin-handler.log:[DEBUG] 2024-09-13 20:58:10.811 zhttp server: OUT connmgr { "seq": 3587, "from": "handler_3132005", "id": "0-14-9b" } 50 "\n\nfilter error: build-id: no sub meta 'id_format'\n"
pushpin-handler.log:[DEBUG] 2024-09-13 21:02:49.212 zhttp server: OUT connmgr { "seq": 4851, "from": "handler_3132005", "id": "0-14-9f" } 50 "\n\nfilter error: build-id: no sub meta 'id_format'\n"
pushpin-handler.log:[DEBUG] 2024-09-13 21:06:07.075 zhttp server: OUT connmgr { "seq": 3437, "from": "handler_3132005", "id": "0-14-107" } 50 "\n\nfilter error: build-id: no sub meta 'id_format'\n

Could you please explain to me what could be causing this?

[jkarneges]

Very strange. The specific problem is a channel was set up to use the build-id filter, but no id_format value was found in the metadata at a time when the filter was applied. django-eventstream sets up that filter here, and it looks like it's impossible for the build-id filter to be specified without id_format also being provided. So it's not obvious to me how this happens.

One idea is the initial responses set up the filter, but later responses for the same connections don't, and Pushpin doesn't like this. I could imagine this happening if storage configuration was removed while clients were connected, but I suspect that's not what's going on here especially if it's repeatable. Or maybe you have multiple Django app instances and not all of them have storage enabled. It would be interesting to see the Grip-Channel headers of responses from the Django app, to see if they inconsistently include param filter=build-id.

[kittinan]

Thanks for the advice, it gives me more clues to investigate the bug.

[kittinan]

My team found the bug. The problem is django-grip does not set the Grip-Set-Meta header when instruct is not hold https://github.com/fanout/django-grip/blob/master/django_grip.py#L388-L399.

• Pushpin call django-eventstream to get events and more = True https://github.com/fanout/django-eventstream/blob/master/django_eventstream/eventresponse.py#L96-L97 it is not set hold stream.
• django-grip does not parse the Grip-Set-Meta header when instruct not hold https://github.com/fanout/django-grip/blob/f176c9b7c625acfa7f0a0cf51981f9a61cf6a0f1/django_grip.py#L388-L399

I created PR in django-grip to fix this. https://github.com/fanout/django-grip/pull/20

[jkarneges]

Thanks for finding this. I've posted a new version of django-grip.