Wishlist: Segmented processing

[eddycolloton]

Allow for segmenting a sequence or file, to be able to process a percentage of a dpx or ffv1/mkv at a time. Something like the ffmpeg -ss or -t flags, that could allow the process to start from a certain frame in a rawcooked created ffv1/mkv.

This could be ideal for aborted rawcooked processes, which could then be resumed from their point of failure. I'm about to abort a process that is hanging, but is near 60%. The ability to segment could allow a user to pick up the process where it left off, rather than start over.

This could also facilitate overnight processing in an environment where the computer running rawcooked was needed for other work during the day. The user could run a segment that would occupy the machine while it was not being used, then run another segment at the next opportunity, so that a sequence didn't have to be completed all at once.

Breaking the dpx sequence into segments from the beginning could achieve this manually, but once the ffv1/mkv has been made from the sequence, this is no long viable - I think (right? making excerpts form the ffv1/mkv would lose reversibility metadata, right?).

[stephenmcconnachie]

I'd support this wish, we have encountered a lot of hung cases, and have built bash scripting to mitigate / manage the outcomes - basically by testing for the relevant status in the output txt file (modification time greater than 12 hours, no success / fail status, etc) and deleting to allow retry from the start.

Would be A m A z I n G to have RAWcooked capable of restarting from hung point. Would be even better if RAWcooked could use the output txt file as input for the second attempt, somehow.

[stephenmcconnachie]

Related wish: Pass RAWcooked a set of RAWcooked output files as input (eg piped from find), and have RAWcooked resume cooking for hung cases, based on rules.

[stephenmcconnachie]

Jerome I may be able to use this as my 2020 rawcooked investment. Would be great to have a list, easy to parse, of all already-gathered wishes, dev requests, issues, to help community of potential investors discuss priority etc I know Github here has those, but they're distributed - is a single list a feasible thing, for consensus building by users?

[JeromeMartinez]

@stephenmcconnachie it is different implementation, please open a dedicated ticket so we can track better requests.

[JeromeMartinez]

Jerome I may be able to use this as my 2020 rawcooked investment.

😃

I know Github here has those, but they're distributed - is a single list a feasible thing, for consensus building by users?

I don't get what you mean about the "criticism" of GitHub issue tickets, and which kind of better display you would like? a dedicated page with a list of features, their cost, and editable by sponsors who would vote for prioritizing features and/or would say that they would like to find a co-sponsor? Would MediaArea voting system be acceptable?

[stephenmcconnachie]

Perfect! Thanks

[JeromeMartinez]

Allow for segmenting a sequence or file, to be able to process a percentage of a dpx or ffv1/mkv at a time.

Wondering if you want both to MKV and from MKV, which are 2 different implementations: Here I understand that it is about reverting to DPX. If you want being able to select a start DPX frame (and duration), please open a new issue ticket (easier for all to track status of features).

A start frame (and duration, i.e. similar to FFmpeg -ss and -t) MKV to DPX decoding is easily doable. It would be also doable to have an option for automatically detecting the last decoded DPX file, and start automatically from this point.

making excerpts form the ffv1/mkv would lose reversibility metadata, right?

Reversibility would be kept for the image files (exactly same hash of DPX files compared to source). If audio support is implemented (more complex, I would not do it at the same time, prioritizing DPX, the thing you are looking for), it would not be the source file (no reversibility at 100%) as the WAV file is unique for the whole stream so decoding only a part taints the file (I will have to modify the size of the data chunk) except in automatic mode (I check where was the crash/break, and keep writing data after it, so I would have the whole content at the end).

[eddycolloton]

Thanks, yes this is in response to going from mkv -> dpx, although the other option seems worthwhile as well, I'll start a separate issue ticket, as you suggest.

A start frame (and duration, i.e. similar to FFmpeg -ss and -t) MKV to DPX decoding is easily doable.It would be also doable to have an option for automatically detecting the last decoded DPX file, and start automatically from this point.

This is great news! Automatically detecting the last decoded DPX file would be great, I have had a few processes hang that I have had to restart in the last few weeks. This change would make resuming such processes very straightforward.

Am I understanding you correctly that, for my current issue - 55% complete dpx sequence made from a rawcooked created ffv1/mkv - I can create an excerpt of my ffv1/mkv (something like ffmpeg -i input.mkv -ss last:decoded:frame -c copy output.mkv), and then run rawcooked on that, to complete my 55% dpx sequence? Or are you saying this would be a feature in the future?

[JeromeMartinez]

I can create an excerpt of my ffv1/mkv (something like ffmpeg -i input.mkv -ss last:decoded:frame -c copy output.mkv), and then run rawcooked on that, to complete my 55% dpx sequence?

Sorry, but no, the FFmpeg processing would discard the RAWcooked file, and even if you use the possibility to copy attachments it would not work (in that case RAWcooked file would be unsynced with the MKV frames).

Or are you saying this would be a feature in the future?

Right. it would be a feature to be developed, reason this ticket will stay open ;-). Something like directly (no FFmpeg preprocess) rawcooked input.mkv -ss last:decoded:frame -o output/ (and for automatic detection of last decoded frame, maybe just rawcooked input.mkv --resume -o output/).

[eddycolloton]

Sorry, but no, the FFmpeg processing would discard the RAWcooked file, and even if you use the possibility to copy attachments it would not work (in that case RAWcooked file would be unsynced with the MKV frames).

Makes sense, that's what i assumed, I just got confused. Thanks for clarifying.

[abouvardngt]

How much time/money do you think you'll need to develop this feature ? I have to convince some of my colleagues that your software is what we need. The fact that rawcooked cannot segment parts of .mkv to create little dpx sequences is an obstacle for us. Thank you.

[JeromeMartinez]

It is not the most expensive feature, it will also depend if you are willing to sponsor in full (complete cost, available for everyone) or partial (with a license key until we have the full cost covered). Please contact us at info@mediaarea.net for talking about that.