search

peter-mount / nre-feeds

go library & Application for handling the NRE DarwinD3 feeds
Apache License 2.0
8 stars 0 forks source link

D3 Performance issues when there's disruption #14

Closed peter-mount closed 5 years ago

peter-mount commented 5 years ago

Whenever there's any serious disruption the D3 feed falls behind, sometimes by up to 5-10 minutes.

It seems that the recent refactor away from using single files per RID (as the V12 service uses) to BBolt limits us in the amount of updates that can be done.

Right now the V16 D3 seed has 1200 messages queued and is processing at around 50-60 per second but the inbound rate is hitting 80/s (I've seen that hit 150/s in the past).

The solution post V16 go-live is to revert the storage of the V16 feed back to the old way as that could handle >300/s on lower hardware than the test environment.

peter-mount commented 5 years ago

The solution of using a Disk based memory cache rather than bbolt has been running for several releases now & although there can still be a backlog it handles it pretty well, only now limited by disk IO on the SSD's Live & UAT use.