TN: Make submit.py able to handle 'stale' txgen output files

[relativityboy]

As a developer bootstrapping a testnet. I want maximum ease of scripting, without having to consider when the actions file (currently created by txgen.py) for the boostrap process (currently done by submit.py) was created.

Supporting Materials

• Followup to #80

AC

• [x] submit.py should not be dependent on time-derived information within the actions file for any of it's own time-dependent behaviors (ex, submitting wait_blocks operations to cover gaps in the blockchain)

[inertia186]

Currently txgen directs block formation. First, by calculating the missed blocks since genesis, then every n transactions (default 40), and finally, by filling the remaining bootstrap phase with wait_blocks. This means that in addition to creating transactions, it is involved in block formation.

Also currently, the job of submit is to take actions from its input (usually a pipe) and broadcast them. It also has an additional requirement to avoid broadcasting transactions into the future (if --realtime is set). It is otherwise oblivious to how many actions there are and when they will end.

Option 1: Rename txgen

One way to address this is to rename txgen to actiongen. Problem solved.

Option 2: Smart Submit

Another way is to strip all action logic out of txgen and add it to submit. So when submit first runs, it creates the missed blocks, processes txgen input, and when the txgen pipe ends, fill wait_blocks. For this to work, submit would need to take a peek at the same snapshot.json that txgen sees so it can predict the number of transactions. It would also need to have a configuration or argument to know where snapshot.json is and how many transactions per block.

This supports the idea of a unified tinman.conf that all modules look at for configuration.

This also creates a possible issue: If txgen and submit both read tinman.conf to look for snapshot.json (for different reasons), this requires snapshot.json to exist in both phases. The deployments scripts currently generates actions on a different schedule than submit. In that scenario, submit does not have access to the original snapshot.json.

[relativityboy]

Option 1 does nothing to address the general description of the story's title nor AC. It isn't a valid option.

Option 2 sounds partially correct. But snapshot.json need not be present for both stages. txgen can output the number of transactions it will be writing at the beginning of it's output. This problem has already been conceptually solved in branch 80-make-start-time-better though that branch has a high-memory consumption txgen based on an older version. A problem which can easily be solved with the current version of txgen.

[inertia186]

Sorry, yes Option 1 was a bit too myopic and partially based on a previous discussion that txgen is should only generate transactions and never block formation.

Therefore renaming it was one solution while also implying that stale actions should be discarded because the very first line of the actions output is based on a real-time-clock that has long since passed.

---

Switching to Option 2:

txgen can output the number of transactions

I guess I should just competely forget about the notion that txgen should only generate transactions.

Actually, this is not a big deal if we widen the term a little: the very first thing txgen is sending is still about transactions, even though it's not a transaction. It's transaction metadata.

If that's the case, txgen sends this single metadata "action" about the predicted number of transactions so submit can generate its own miss schedule.

Also, submit needs to know the number of transactions per block, so that could be part of the metadata from txgen (which is also about transactions). Or the number of transactions per block should be a config setting for, or argument to, submit.

The only thing that's weird about this is that submit would suddenly be worried about things outside of its original mandate ("here's a stateless action, do it"). But as long as the other modules don't mention anything about transaction metadata to submit, it won't be a problem.

[inertia186]

Idea: initial wait-block offsets

[relativityboy]

Expand plz.

[relativityboy]

124 should be done 1st. Marking as such.