curran
commented
1 year ago Hey thanks!
Yes, definitely interested in taking on additional ideas and directions for this project. I want to build a thing that's actually useful, and I think being open to new ideas and feedback is critical for that to happen.
an editor that could "play" through the state of the code along side a video
Very cool idea. It's definitely possible, in theory. Scrimba does this, check it out. I've seen in Scrimba they have just audio that goes along with the code editor, which updates in sync with the audio, and you can scrub back and forth in the history. Scrimba also lets you pause the video at any point and make your own edits, which is amazing. They have actually totally executed this idea of yours, so we should look to what they've done as inspiration for anything we do along those same lines.
Regarding how it could be implemented, yes it is a state management problem, in which there is a "meta-state" which is the revision history of the document. Each version in the revision history would have a timestamp and an OT "op" (operation) that's a small diff against the files, like a few keystrokes. Luckily for us, ShareDB already is doing this internally! To implement a system that plays code changes in sync with a video, we'd need to solve many sub-problems:
- Play a video in the page
- Be able to listen for events from the video player (e.g. play, pause, scrub). Not sure if the YouTube embedded player gives us this. We'd probably need to look at alternatives, which gets complicated real fast because it introduces the need for hosting video files somewhere, and the whole workflow around getting videos into that system.
- Alternatively, maybe audio is enough, because the video mostly would be showing the code edits... Maybe audio-only would be a good "Phase I" to prototype, with video coming later.
- Get access to the revision history of the document. ShareDB may give us this out of the box (using its "oplog" concept), but it may be simpler to introduce ShareDB middleware to store our own version of this history the way we want to.
- Somehow, we'd need to be able to synchronize the video/audio player with the timestamps of the edit operations. If the video were un-edited, this would be relatively easy, because you could just match up the timestamps of the first edit in the op log and the first edit in the video, then you'd have the mapping (just a timestamp offset). HOWEVER, if the video is edited after the fact, this introduces a huge challenge of cutting out the same periods of time in the op history as were cut out in the video editing. How could this be solved?
- One way to solve that problem would be to create a system that actually records the video/audio inside the editor and allows the author to play/pause the recording. In this way, video editing could be bypassed entirely, and it would be done "live" by the author. If we did this, we could totally synchronize the timestamps for each recorded segment, then string them together.
- To actually play back the ops, we'd need to build a very time-oriented system that would handle play/pause and scrub interactions. For this part, when it's playing, we could do a fast poll of like every 1/10 of a second, check if there were any ops in that tiny part of the history and play them back (could use D3 bisect for this).
Overall it's a super cool idea, and I've thought about it. It would take quite some work to implement, but would be sweet!
I hesitate to include this as a "core feature" of this project, but maybe we could start to think about a plugin architecture where something like this could be a plugin. That phase of the project is still quite far in the future, but thanks for your input!
I'm not sure if this is doable, or if you have any interest in taking on additional ideas with completely different structures, but after three and a half years of watching youtube videos of coding it would be so dang cool if there were an editor that could "play" through the state of the code along side a video. I realize that this is mostly a state management problem, and what you have here is not that at all. But! The educational use of your real time editor is real, which is why I'm leaving my comment/(undeveloped)idea here as I think it would also have real educational value. Also thank you for all the videos!