Video support

[jimjam-slam]

I've added the first part of this work in 998617c and 485f235: videos that autoplay and loop. Really it's just a class to ensure that they run full-bleed; the looping and autoplay is done by the browser.

But as #132 discusses, we might want video to hold off starting until it's visible. So the next part is to trigger that manually wth JS.

I think it probably makes sense to add a shortcut, analogous to Quarto's video shortcode, to do this while inserting the necessary attributes (eg. preload) on the video tag. (In fact, we may end up borrowing their chortcode and tweaking it!)

The last part of this is the most complicated: image sequences that progress as you scroll through the container (although I'd love to know if you can do this with a traditional video!).

Most of the implementations of this I've seen (eg. this one on dev.to — it uses React, but the principle should apply with vanilla JS) use a canvas element: you preload the images by calling a function ASAP to download them, then call requestAnimationFrame() on scroll to update which image is showing.

A user might either want to use regenerated images or ones generated by a code block in the doc — but it isn't clear to me whether there's some special treatment we can give, for example, an R code block to say "use the images emitted from this code block for a scroll video".

That said, perhaps you could have:

a. a sticky block where the image glob or image path is specified as an attribute, and then b. a trigger attribute that specifies how far through the sequence. Or perhaps a progress block... or perhaps both are viable options?

::::{.cr-section}

<!-- or maybe a shortcode makes sense for this sticky? -->
:::{#cr-images video-scroll-first="images/pic001.png" video-scroll-last="images/pic135.png"}
:::

Check out this awesome video [@cr-images]{video-scroll=1}

It's pretty rad [@cr-images]{video-scroll=20}

Now we're going quite fast [@cr-images]{video-scroll=80}

We've paused for a bit [@cr-images]{video-scroll=80}

Now we're finished [@cr-images]{video-scroll=120}

::::

Or maybe with a progress block (although no option to vary the speed here):

<!-- can we reuse focus-on and detect that it's scrolling? -->
:::{.progress-block focus-on="cr-images"}

Step 1

Step 2

Step 3

Step 4
:::

What do you think, @andrewpbray?

[andrewpbray]

Thanks for getting this started! Lots of fun stuff here.

Feature 1: Play videos with trigger

In terms of API, what do you think of:

Checkout this lovely video. [@cr-vid]{play-video="true"}

I'm inclined to avoid any mention of autoplay since that's less well-defined in this setting since the user is in control of when the video would become un-transparent. Not sure what the best attribute name is: play, play-vid, play-video? I can take a crack at the necessary JS for this later on today.

Feature 2: Control video progression with scroll

Seems simpler to use a progress block! Controlling the play speed could always be done on the video file itself in pre-processing, right? (i.e. removing frames).

I'm not quite sure what the best API for this is. I like what you have. There's also this:

:::{.progress-block scroll-video="cr-vid"}

Step 1

Step 2

Step 3

Step 4
:::

A few things:

• While we could infer what to do if a progress block is focusing on a video, might be better to make that explicit in the attribute name.
• Seems like we'll be able to do the png sequence and less clear about setting the frame of the video file. That'd be good to know for deciding if we should use separate attribute names for those two or combine them.
• Seeing scroll-video here makes me wonder if start-video is clearer for feature 1 (since scrolling is also a form a playing a video).

[jimjam-slam]

@andrewpbray Agree with you on Feature 1! autoplay is too confusing (and it's a browser attribute, so it ought to be avoided).

I agree that a progress block is a nice interface! While I do also agree that you can change the number of frames to tweak the output speed, I think at minimum it would be good for users to be able to pause the progression between a number of steps where they want to stop and discuss something in more detail. You could technically run duplicate frames off, but it'd be a big waste of bandwidth. Maybe something like:

:::{.progress-block scroll-video="cr-vid"}

Step 1

Step 2 [@cr-vid]{pause-video}

Step 3 [@cr-vid]{pause-video}

<!-- video resumes as you pass this step -->
Step 4

Step 5
:::

These sorts of pauses would be helpful for videos that sweep over an area (eg. as events unfold over time).

[andrewpbray]

Good point - that'd be some very useful functionality.

One thing I'm realizing: we have crProgressTrigger and crProgressBlock. Do we want an interface that would, in concept, work for both?

Scroll a video through a single narrative block

:::{#cr-myvid}
{{< video myvid.mov >}}
:::

Step 1. [@cr-myvid]{.scroll-video}

Scroll a video through a progress block

:::{#cr-myvid}
{{< video myvid.mov >}}
:::

:::{.scroll-video focus-on="cr-myvid"}

Step 1.

Step 2.

Step 3.
:::

This raises a few interface questions:

1. Should focus effects always be attributes or can they be classes in situations where this is no arguments/values that we need to pass?
2. How should we get the trigger and progress block version aligned? For the former, .scroll-video or scroll-video="true" seem to make sense since the ref to the sticky is right there. But then how should we pass the ref to the sticky in the progress block version? focus-on is one option but would actually mess things up: I think it'd put all three steps as separate Paras in the same narrative block without the padding. .progress-block scroll-video="cr-vid" would work but then we have a different interface than the single trigger scroll.
3. If we use .progress-block scroll-video="cr-vid", do we even need .progress-block? Seems like no. Seems to make sense to keep that one narrowly scoped to exposing that ojs variable.

[andrewpbray]

Maybe we wade into getting a working example of an image sequence and see how it turns out. That'll probably help clarify which interface makes the most sense. Looking back, my stuff above is all based on the general notion of a "video" but image sequences are a slightly different beast; there's not just one file to point to.

I love the {{< image-sequence start="file1.png" end="file15.png" >}} idea by the way! (or something in that neighborhood.

[andrewpbray]

@jimjam-slam Here's a first pass at implementing feature 1.

It's implemented generically so that play-video="true" and pause-video="true" both work, but really any available video method will work when passed as ___-video="true".

One thing to decide is question 1 above. Should these be classes instead of attributes? This should be determined I think just based on what seems like the best interface. The implementation is straightforward - change this JS function to reference classes instead of attributes and then modify the lua filter so that a trigger will pass both classes and attributes to its enclosing block (wrap_block currently only passes attributes).

While testing this out, I did bump into this: https://developer.chrome.com/blog/autoplay/. Essentially, if the user does nothing but scroll straight down, it might not execute play().

[andrewpbray]

I've been tinkering this morning with tying progressBlock.progress to video.currentTime; effectively skipping the video to an appropriate frame as the progressBlock increments. I haven't quite gotten it working yet but just discovered this:

https://scrollyvideo.js.org/

In order words, I've been trying the "traditional" method 3. What are your thoughts on using a dedicated library for this sort of thing? It looks like the author has been working is within the last 3 mo, which is a good sign. If these video standards change over time, sure would be nice if we could take advantage of his keeping things up to date.

[jimjam-slam]

I've gotta come back to your comments properly, sorry @andrewpbray! I had half an hour and banged out some rayshader code to generate a sample video.