test feat: BDD record stream validators need usage documentation (at least) to prevent flaky tests

[david-bakin-sl]

It's hard to use BDD record stream validators (*) correctly in the absence of documentation, and/or possibly framework changes. There are delays that are necessary to prevent flaky tests (especially since the CI environment uses testSubprocess where the normal dev usage on a local machine is to use testEmbedded, and they behave differently).

• (*) I.e., to create your own record stream validator for a specific BDD test

Suggest either a cookbook entry describing how to do this correctly/reliably and/or framework changes so that if you use record stream validation "is just works".

Some questions that could be answered:

• Before validating when is a freeze necessary vs when is a cryptotransfer followed by a 2s sleep sufficient
  • And if you have more than one test in a suite that does record stream validation is there a conflict between these delays (esp. if you need to use a freeze!)
• When is a call to TxnUtils::triggerAndCloseAtLeastOneFileIfNotInterrupted needed?
• What is the role of StreamValidationOp class and when/how do we use it?

[david-bakin-sl]

(w.r.t. StreamValidationOp class is when should you add a validator to RECORD_STREAM_VALIDATORS? Also, to make that class easier to use, it it turns out it is supposed to be used generally, can RECORD_STREAM_VALiDATORS be protected so you can subclass it in a test? Or maybe just a setter so you can add to it? Or something like that?)