search

elastic / synthetics

Synthetic Monitoring with Real Browsers
MIT License
66 stars 41 forks source link

feat: increase chunk_size for lightweight monitors !! #978

Open shahzad31 opened 3 days ago

shahzad31 commented 3 days ago

We have 100 default chunk_size for pushing monitors. This makes sense for browser monitors, since there bundle size can be big or small. So we set a very small number of 100, but it doesn't make sense for lightweight monitors, which are very small in size.

Why

small chunk_size add extra load on kibana in terms of processing with subsequent requests, we make bulk operations in kibana and having size of 100 or 1000 won't make a difference but having back to back 10 requests makes batching useless in some aspect.

Solution

We should increase size of chunk for lightweight monitors to 1000 and for browser monitors we can keep it to 100.

Kibana changes

We will also need to update kibana to accept more monitors, since it's also set to 250 at the moment.

vigneshshanmugam commented 3 days ago

I like the idea of having different limits for the Lightweight and Browser monitors. But we need to do some stress testing on the Kibana side as it might cause some memory spikes when processing 1000s of monitors s well as the how it affects the Integration policies on the Private Locations.

@shahzad31 Is there a plan to also increase the limits between Kibana and our globally managed locations? I believe that is already at 250 per location?