Combine frame and state recordings in a single file

[jbarnoud]

Recordings are save in separate files for the trajectory and the shared state. This was easier to implement but is inconvenient in practice:

• it makes it possible to mismatch recordings (i.e. provide a state and a trajectory recordings from two separate sessions)
• it makes it more difficult to work reason about, especially in terms of UI (see #202)

In terms of drawbacks, having both data streams in a single file makes it more difficult to read. For instance, it becomes more difficult to get to the next record of a given stream when records of the other stream are interleaved. Note that merging the two streams in a single file after the recording is done is not really an option: one requirement we have is that the data should not be lost if the program doing the recording crashes; if a crash happens, the program won't be able to do the combining step.

[Ragzouken]

How about something simple like: the combined format is literally a .zip archive of the two files, which the recorder will create if it doesn't crash. In the event of the crash you have the two source files and have a clear path to finishing the recording process manually (just zip the two files together yourself). Or alternatively the same workflow except it's not zip but some particular process and small utility to run the final combination step on two intermediate files in the event of a crash?

[jbarnoud]

It is possible to zip them together, indeed. It was my initial idea, but I have been reluctant to implement it. I am not sure about the user experience when the recorder crashes. The priority is not to lose any data so a user should find the not-yet-zipped individual recording if the server crashes, so the user must know where they are. Yet, in the normal scenario, the server won't crash, so the user should not have to care about the crashing scenario. The way I see it, the user specifies the path for the final file in the interface (GUI or CLI), but the recorder won't be able to say anything to the user if it crashes, so it won't be able to tell "Careful I could not write the ZIP, here are your intermediate files".

I may be overthinking the problem, though.

[Ragzouken]

I guess I'm thinking of how file transfer tools do it: Firefox download and WinSCP upload write, in the target directory, an intermediate file that is deleted as the final step. In the event of a crash or disconnect, the file is left there and can be used to resume. yt-dlp (a youtube downloader) sometimes has to download audio and video separately and will put them in intermediate files in the target directory that are deleted after the final combination step. So there's precedent for this kind of thing, though admittedly there's some difference between downloading/uploading a specific target twice vs recording from the same server twice (you expect different results when repeating it).

Probably the benefits of a single ZIP file are worth the necessity of a manual cleanup in the (hopefully rare) event of a crash. But I think there's also the possibility to add an additional intermediate file that's just a README telling you how to recover from a crash if this README file wasn't deleted?

[jbarnoud]

That's a good idea!