This patch ensures that the indexer does not overwhelm any listening consensus groups with index file updates in large ledgers by applying back pressure that will slow down the running index process when the listening consensus groups can't process the file updates quickly enough to match the volume.
The changes to apply back pressure to the indexer when necessary are in commits 4b39855d4bfd45629d8a57388f00c403af831e2c, c68496416822dd78a81f41acc7c4d8ae9b9be15a, d28088a73c569f086f133d8e303759685f13456b, and bcaf973bb06ce78144e004787193b0cc6d4f2f54 (mostly in 4b39855d4bfd45629d8a57388f00c403af831e2c and bcaf973bb06ce78144e004787193b0cc6d4f2f54).
The rest of these changes are the beginnings of a reorganization and streamlining of our protocols to better separate concerns between components while making those components more flexible and interchangeable.
I had planned on addressing another bug where the indexer can't keep up with new commits for high volume ledgers as well in this patch. Those changes depended on reorganizing the protocols to separate concerns between the indexer indexing ledgers and the connection writing commits, which I began here.
To fully separate those concerns however, I would have had to make changes to the nameservice protocol and implementations. There are a lot of changes to the nameservices in flight, so I decided to hold off on finishing that reorganization and fixing the other indexing bug until those changes land to minimize merge conflicts.
This patch ensures that the indexer does not overwhelm any listening consensus groups with index file updates in large ledgers by applying back pressure that will slow down the running index process when the listening consensus groups can't process the file updates quickly enough to match the volume.
The changes to apply back pressure to the indexer when necessary are in commits 4b39855d4bfd45629d8a57388f00c403af831e2c, c68496416822dd78a81f41acc7c4d8ae9b9be15a, d28088a73c569f086f133d8e303759685f13456b, and bcaf973bb06ce78144e004787193b0cc6d4f2f54 (mostly in 4b39855d4bfd45629d8a57388f00c403af831e2c and bcaf973bb06ce78144e004787193b0cc6d4f2f54).
The rest of these changes are the beginnings of a reorganization and streamlining of our protocols to better separate concerns between components while making those components more flexible and interchangeable.
I had planned on addressing another bug where the indexer can't keep up with new commits for high volume ledgers as well in this patch. Those changes depended on reorganizing the protocols to separate concerns between the indexer indexing ledgers and the connection writing commits, which I began here.
To fully separate those concerns however, I would have had to make changes to the nameservice protocol and implementations. There are a lot of changes to the nameservices in flight, so I decided to hold off on finishing that reorganization and fixing the other indexing bug until those changes land to minimize merge conflicts.