Frame dependencies are incorrect with an interleaved range and chunking

[mmichaelc]

Describe the bug I've discovered a case where the frame dependencies aren't set up correctly and it leads to frames running before they should, normally resulting in file not found errors. It can be reproduced using basic shell commands. In our real world scenario, we have a render layer which produces images and a denoise layer which denoises the rendered image and produces a second image. The denoise layer has a chunk size of 3.

To Reproduce

cmd = 'echo "Frame: %{FRAME}, zframe: %{ZFRAME}, range: %{RANGE}"'
layer1 = Shell("mock_render_layer", command=cmd, env={"CUE_ENV": "show"}, range="1001-1200:10", chunk=1, shell=True)
layer2 = Shell("mock_denoise_layer", command=cmd, env={"CUE_ENV": "show"}, range="1001-1200:10", chunk=3, shell=True)
layer2.depend_on(layer1)  # frame on frame
# dispatch an outline with those 2 layers added.

Expected behavior After the first round of frames on 10s, the frame dependencies on the mock_denoise_layer are incorrect, as viewed in cuetopia. The output for denoise frame 1046: The dependencies as shown in cuetopia:

Second example It can be shown better by changing the commands to write and read files: You'll need to substitute a file path that is accessible by the render hosts.

render_cmd = `touch /jobs/pipeline/interleave_test/mock_render.%{FRAME}'
layer1 = Shell("mock_render_layer", command=render_cmd, env={"CUE_ENV": "show"}, range="1001-1200:10", chunk=1, shell=True)

denoise_cmd = 'ls /jobs/pipeline/interleave_test/mock_render.%{FRAME}'
layer2 = Shell("mock_denoise_layer", command=denoise_cmd, env={"CUE_ENV": "show"}, range="1001-1200:10", chunk=3, shell=True)
layer2.depend_on(layer1)  # frame on frame
# dispatch an outline with those 2 layers added.

This will lead to a lot of denoise frames failing as they attempt to ls the file before the render layer has touched it.

Version Number 0.21

Please let me know if any more information is needed.

Many thanks, Michael

[DiegoTavares]

This does seem like a bug to me. I'm adding to my list to investigate.

Sorry for taking months to reply to this. Now that we've got the PRs under control, we're working towards cleaning up the issues.

Did you manage to find a work around or even investigate where the bug is coming from?

[mmichaelc]

Hi Diego,

I haven't managed to get my dev environment set up to debug this. I tried reading through the source but couldn't see where the issue might occur.

Our workaround is to not use frame interleaving for now which is ok but of course would be nice to have it.

Thanks, Michael