n3tsurge
commented
7 months ago Is that 225,000 events per day? Thats not overly large but what I think is happening is you aren't signaturing your events correctly resulting in a large number of signatures over your search window. Can you run the following via dev tools? If the total number of unique signatures is above 10000 you're not signaturing correctly and deduplicating your events. Non signaturing correctly will also allow duplicates of the same event to come in which can result in Event explosion. I would also recommend getting to the latest release.
GET reflex-events/_search
{
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "status.name: New"
}
},
{
"range": {
"created_at": {
"gte": "now-7d"
}
}
}
]
}
},
"aggs": {
"unique_sigs": {
"cardinality": {
"field": "signature"
}
}
},
"size": 0
}
greycel
JustinHendersonSMAPPER
Hi Team,
1) Event Queue Issue retrieving a large dataset We've been receiving about 2,25,000 alerts per day roughly from different log sources and last checked there were 7,45,000 alerts for the last 7 days. For the last 10-15day the event queue page has been taking time to load and sometimes not loading eventually throwing
opensearchpy.exceptions.TransportError: TransportError(503, 'search_phase_execution_exception')below is the error log from the API service.2) After restarting all the reflex docker services, the Memcached docker container state is displaying as unhealthy and I've been receiving Memcached problems in the "reflex-api" service logs. service is reachable and connected when checked with "nc localhost 11211". I'm not sure which reflex components will be affected by this, need help.