Wishlist: Resumption of hung cooks, using RAWcooked output files as source info

[stephenmcconnachie]

Related to / derived from Issue https://github.com/MediaArea/RAWcooked/issues/287

Would be great to pass rawcooked a set of RAWcooked output files - eg piped from find, or in a watched folder, not sure of best implementation - and have RAWcooked resume cooking on stalled / killed cases, based on logic such as modification time of the output txt file is greater than 6 hours, or similar.

Alternative implementation: pass RAWcooked a folder path to DPX folders, and a path to output txt files, and it resumes cooking as above.

[JeromeMartinez]

I have difficulties to understand the workflow. Is it for encoding or for decoding?

For encoding: I don't think it is doable to append content to an already started file, so it would be later (when I have my own encoder & muser)

For decoding: I don't see what I could do with content of find or a watched folder. Resuming a decoding would be checking that a file already exist, and skip the read & decoding of the corresponding frame in the MKV. So using the same command would be fine (RAWcooked would list the files already present in the directory, possibly check hash for being sure it is fine, and decode only frames corresponding to files not yet present in the output dire or without right hash).

[stephenmcconnachie]

It's encoding of a MKV, including the --check stage. We see hung cooks regularly, maybe caused by network timeout or other factors. (We don't yet do much decoding of MKV to DPX other than in the --check function while cooking).

My suggestion was to find a programatic way of resuming any hung cooks, appending to the partial MKV by parsing an output txt from the previous cook attempt, and resuming at correct frame.

But I guess resuming at correct frame could be achieved by examining the partially encoded Matroska and the source DPX, without need for the txt file.

[JeromeMartinez]

So still 2 parts:

For encoding: we need to have our own encoder/muxer first, not a small thing, then we "just" check the end of the MKV file, no need to have the log.

For checking: here we have no clue about where check stopped so we need the log. for simplicity here I suggest to have an option e.g. --check -ss x (same of for the related tickets, --ss is for a start frame, but we could have a percentage too), and user would provide the value (could be automated by parsing the log). Second step would be to parse the log directly if there is a value added.

Generally speaking, resuming a job is not an easy task if we take all cases in mind. I'll implement each case step by step.

[JeromeMartinez]

Additional thought: I could also add a "watchdog" internal thread, and report when there is an hung (and say if it is a network time out and so on, I hope), also I could kill the worker thread and resume (I have all in RAM so easy). For encoding, it would be still same issue (I first need to implement my encoder/muxer).

[stephenmcconnachie]

This!