Spec says about persistent user data: For a step that does not do checkpoints, it is saved after the step ends. Chunk step too should be guaranteed to persist user data after checkpointing.
the problem here is that we persist the PUD at end of step (to catch updates in the step-end listener) whether this is a chunk or batchlet. For a batchlet it makes sense, but for a chunk we might have rolled back and the in-memory version of the PUD has data from the last chunk...we didn't commit that tran, so we didn't commit the PUD updates...but the step-end commit of the step execution stuff will sort of accidentally persist the PUD from that partial chunk. Which could mess you up on a restart.
scottkurz
commented
3 years ago
Spec says about persistent user data: For a step that does not do checkpoints, it is saved after the step ends. Chunk step too should be guaranteed to persist user data after checkpointing.
the problem here is that we persist the PUD at end of step (to catch updates in the step-end listener) whether this is a chunk or batchlet. For a batchlet it makes sense, but for a chunk we might have rolled back and the in-memory version of the PUD has data from the last chunk...we didn't commit that tran, so we didn't commit the PUD updates...but the step-end commit of the step execution stuff will sort of accidentally persist the PUD from that partial chunk. Which could mess you up on a restart.