buildbarn / bb-portal

Build Event Stream capturing
Apache License 2.0
11 stars 5 forks source link

Start storing data and serving views in response to streaming events instead of cumulatively at the end #20

Open trey-ivy opened 1 month ago

trey-ivy commented 1 month ago

larger builds can take a long time to serialize to the database. Can we alleviate this by batching writes as they come in instead of waiting until the very end? some data isn't supplied until the build is complete, but we could show targets as their accumulated and build log output, etc.

trey-ivy commented 1 month ago

a little additional discussion suggests we want to look at a message queue to help us achieve this. maybe roll in something like kafka to be deployed alongside bbportal