This eliminates both the deserialization and serialization of chunks at the Batcher.
This matters because the cost on just the serialization of chunks can be significant: e.g. a large operator took 61s to serialize the dispersal request
Such perf/cost is on the critical path of E2E flow, so by eliminating them (as well as eliminating the cost to deserialize chunks from Encoder) will contribute directly the E2E perf.
Testing
In addition to the unit test, also tested it in Preprod, with following setup
Original chunks are in Gob
The 3 Nodes are running with: 1) v0.7.4; 2) v0.8.0+ with Gnark disabled (i.e. using Gob); 3) v0.8.0+ with Gnark enabled
And the Batcher was tested in these two cases:
Batcher shipping Gob chunks to Node: chunks will be shipped as it is
Batcher shipping Gnark chunks to Node: chunks will be converted to Gnark (since they were in Gob) and then shipped to Nodes
This shows the compatibility/safety of:
1) interaction with the ongoing Gob->Gnark migration is correct
2) the migration of frames (deserialized) -> ChunksData (zero ser/deser) itself is correct
Checks
[x] I've made sure the lint is passing in this PR.
[x] I've made sure the tests are passing. Note that there might be a few flaky tests, in that case, please comment that they are not relevant.
Why are these changes needed?
This eliminates both the deserialization and serialization of chunks at the Batcher.
This matters because the cost on just the serialization of chunks can be significant: e.g. a large operator took 61s to serialize the dispersal request
Such perf/cost is on the critical path of E2E flow, so by eliminating them (as well as eliminating the cost to deserialize chunks from Encoder) will contribute directly the E2E perf.
Testing
In addition to the unit test, also tested it in Preprod, with following setup
And the Batcher was tested in these two cases:
This shows the compatibility/safety of: 1) interaction with the ongoing Gob->Gnark migration is correct 2) the migration of frames (deserialized) -> ChunksData (zero ser/deser) itself is correct
Checks