Worker: do not send input_dataclip_id if the previous output was too large

josephjclark commented 2 months ago

Tracking an issue right now where:

Step 1 does not send a dataclip id back to lightning
Step 2 starts and says "I'm using this dataclip id" in step:start
Lightning says "woah nelly, I've never heard of such a wondrous dataclip, I'm not having that"
Lightning aborts the run on its side
Meanwhile the engine is still running and trying to send messages, which are now getting blocked because the run doesn't actually exist any more

This PR fixes it by simply not sending an input_dataclip_id if that dataclip was withheld from Lightning. I believe this is safe in lightning.

I will create an integration test against the mock and run a manual test against lightning locally

josephjclark commented 2 months ago

Nice - this ran locally with a payload of 0 (all log lines are redacted too)

josephjclark commented 2 months ago

Skipping the integration test - it won't tell us much against the mock, so it's not very valuable.

I'm happy with the code, unit tests and manual test

OpenFn / kit