search

OneZoom / OZtree

OneZoom Tree of Life Explorer
Other
90 stars 20 forks source link

Audio / video embedding for TourStops #535

Closed lentinj closed 1 year ago

lentinj commented 2 years ago

Support any embedded audio / video in a TourStop, which may take the form of:

For each, TourStop template will need to include the relevant HTML, and the A/V assets hosted by Nginx / whatever. For the most part this will just work now.

For each embed type, the Tour API will need notice each in the HTML and know how to:

We should also have documentation on, for each:

lentinj commented 2 years ago

We can use MutationObservers to trigger autoplay events via. CSS state classes.

lentinj commented 2 years ago

Image & A/V asset ownership / rights information:

Do we want to offer upload of images, at risk of moderation woes? Probably worth the risk, but a separate tool to upload and include a URL.

lentinj commented 2 years ago

@jrosindell The naked mole rat video isn't rated, which means it isn't publicly accessible.

Sounds like you need to set the movie rating according to this: https://stackoverflow.com/questions/49485537/vimeo-not-accessible-without-log-in

lentinj commented 2 years ago

The above adds in embed support as discussed in #601, and the following modifies the existing superpowers tour to work with it:

https://github.com/OneZoom/tours/commit/21636d148c20f7d4f6388ad3666f66470044e91d

Note that instead of a video entry, there's a media array that can have any number and type of in-line media. The type of media isn't really relevant, as we have to decide what to do with it based on URL anyway (embedding vimeo & youtube are entirely different operations, even if the end result looks similar). Having more than one bit of in-line media seems like an obvious thing to want at some point (image of bird & birdsong sound clip), thus the array.

There's no integration to the tours engine yet, but you do get to play the videos within the tourstops (interestingly the embedded version of the naked mole rat video plays fine).

lentinj commented 2 years ago

Video autoplay: Do we start when the tourstop appears (which may be at the start of the zoom), when we arrive at the tourstop, or should it be configurable?

lentinj commented 2 years ago

The trickiest part here is blocking the progression of the tour until the A/V asset has finished playing, without having tendrils of every single autoplay plugin reaching into TourStop.js. What I'm thinking of doing is continuing the CSS-class driven approach to add a series of block- classes:

Then wait_and_goto_next equivalent can monitor the CSS classes of a stop, and if there's no block-*, off we go.

This means each of the autoplay mechanisms can be entirely decoupled from each other, and just off/on their respective classes. This is nice for testing too, we only need to make sure an autoplay plugin sets/clears the block- class.

Whether I'm getting carried away with MutationObservers is a good question. But the events will be fired a similar amount to any other pub/sub mechanism we use here anyway. We also get to remove the polling for hidden browser windows, which is definitely another good thing(tm).

lentinj commented 2 years ago

Comments I mistakenly made in #601:

A tourstop should wait for a video to end before advancing. But what if I leave a video paused? I think that should be equivalent to pausing the tourstop, and we should wait until the video actually reaches it's end.

Also, to move off after a video has finished playing, you have to specify a wait period. Otherwise we wait for the user to press next as well as waiting for the video to finish. A wait period is also useful to paper over the time between a tourstop appearing and the video autoplaying (or never, if we're not allowed to autoplay yet). It feels like an annoying thing that will get forgotten.

The youtube tour plugin could force a wait period for any stop that doesn't have one, but the TourStops instances are already initialised by this point, so we'd have to do some jiggling about to change the value. And it's not entirely impossible that a tour author wants the user to watch the video, do something else, click next.

Finally, the name "wait" is problematic since it's a reserved word, so we can't have a column with that name. Instead I called it "stop_wait" in the model, and made an attempt to use Web2Py's rname to paper around the difference. However, this isn't working. Any objections to instead of working this out, just rename it to stop_wait everywhere and be done with it?

lentinj commented 2 years ago

Anyway, there's now a youtube plugin now that largely works. Leaving the tourstop after video end is stuck behind the "wait" / "stop_wait" confusion. Once the timer is set it'll work.

Also there's no way to disable autoplay yet. It just happens whenever you arrive at the tourstop.

jrosindell commented 2 years ago

@jrosindell The naked mole rat video isn't rated, which means it isn't publicly accessible.

Sounds like you need to set the movie rating according to this: https://stackoverflow.com/questions/49485537/vimeo-not-accessible-without-log-in

Done!

jrosindell commented 2 years ago

Video autoplay: Do we start when the tourstop appears (which may be at the start of the zoom), when we arrive at the tourstop, or should it be configurable?

I think configurable from the JSON e.g. narration to begin on zoom in or video to play on arrival. Best keep it general

jrosindell commented 2 years ago

Any objections to instead of working this out, just rename it to stop_wait everywhere and be done with it?

no objections to this.

jrosindell commented 2 years ago

I think the wait period concept to enable auto move on from video end (if needed) is fine. In case it's forgotten then no problem worst case is user has to press next anyway and stay awake. That's still massively better than deciding themselves where to go in the tree next.

lentinj commented 2 years ago

Right, we can happily autoplay the jellyfish, and move on when it finishes. I've set stop_wait to 6 seconds, so if it doesn't autoplay then you have 6 seconds to decide to press play yourself or it will move on anyway.

lentinj commented 2 years ago

I've written a Vimeo and HTML audio plugin that both work like the youtube plugin. You get to the tourstop, media starts playing (modulo autoplay), and stops when you go away.

I've added wikipedia commons audio support, but not necessarily correctly, the main aim was to act as a convenient source of HTML audio for testing, if this is something we really wanted to support we'd (possibly) want to use their web player plugin.

I also added a song thrush to the superpowers tour, just as an example. But I'm guessing it's a fairly realistic use case for audio in a tourstop.

lentinj commented 2 years ago

@hyanwong I'm thinking of removing the separate TRANSITION_IN_WAIT step from TRANSITION_IN. Not removing the delay logic within, just calling that TRANSITION_IN. Now would be a good time to bleat if you think I'm doing something silly :)

I need to loosen the seams on the "block-*" above so autoplay can also block arrival and/or departure, and having the extra in_wait/out_wait states in the way will make this much more annoying to configure. It already gets in the way of the CSS rules for defining tourstop visibility (which assume transition_in_wait is part of transition_in). TourStop.skip() also ignores the difference. I can't find anywhere where being in _WAIT does something that a regular TRANSITION_IN wouldn't.

The only thing I can think if is you might want a brief period where both tourstops are visible before the flight, but we could manage that with CSS if desperate, e.g. .tsstate-transition-in.block-transitionwait.

lentinj commented 2 years ago

And in a similar vein, do you know when TourStop.block_arrival comes into play, from https://github.com/OneZoom/OZtree/commit/68882ceae693066aa3ed2952e2370b9c7749c5b7? The basic idea seems to be to cancel the flight promise chain, so we don't inadvertently arrive at the next tourstop whilst paused.

However, setting tree_state.flying = false will (AIUI) cause flights to throw an error, breaking the promise chain and block_arrival never gets a chance to do it's thing. EDIT: Whoops, this is completely wrong, and in the process probably answered my question. These errors are caught before we continue to arrive_at_tourstop. That's the way to solve this.

We could express similar in arrive_at_tourstop by saying if the tour is paused don't bother arriving, but that clashes with the block behaviour, where removing block-userpaused should mean that the arrival can happen, so I'd rather not have it there if it's not required.

hyanwong commented 2 years ago

@hyanwong I'm thinking of removing the separate TRANSITION_IN_WAIT step from TRANSITION_IN. Not removing the delay logic within, just calling that TRANSITION_IN. Now would be a good time to bleat if you think I'm doing something silly :)

I don't think it's silly, as long as we retain the ability to pause with something shown on screen before entering the transition. It can be a little disconcerting to start a transition immediately on entering the next stop. There are examples of its use in the minlife_tour.html example here: https://github.com/OneZoom/OZtree/blob/e349aada1eb313c02080e55b8a1d7ab4de79e6f1/views/treeviewer/minlife_tour.html#L119

So for instance, we pop up a box saying "you can pan sideways", then we wait 2 seconds, and start panning. The box is part of the tour stop that does the panning, rather than the previous stop.

If that possibility is kept, I trust your take on the best logic to accomplish the effect!

lentinj commented 2 years ago

I don't think it's silly, as long as we retain the ability to pause with something shown on screen before entering the transition.

Yup. the actual logic behind "transition_in_wait" won't change at all, just we stop calling the state TRANSITION_IN_WAIT.

Actually, one case it does make a difference is if we're transition-in-waiting, get paused and resumed then currently we'd re-transition-in-wait then fly. Without the separate state we'd instantly start flying. But I think that's an improvement meself.

lentinj commented 2 years ago

This is dragging itself into being useful. The above reworks transitions so they also work with the same concept of blocks; just as with active_wait we stay in transition_in until all the blocks have cleared---the most important one being the block for the flight to the new tourstop. This means videos can also set blocks during a transition, so we can finally(!) think about narration A/V that talk through active_wait & transition_out.

In the process I've converted the blocks from being manually added/removed classes to methods on the tourstop. This is a much more obvious way of doing things, and saves the observer since we can advance() whenever we run out of blocks. Currently this means we can rush through states too quickly though---when pressing previous during a flight we try to cancel the flight and navigate to the previous stop before the treeviewer has had a chance to cancel the flight properly. This means rethinking play() a bit to hold fire to allow the treeviewer to catch up.

We still set the block-* CSS classes, they're quite useful to see into the mind of a tour, but not required now.

lentinj commented 1 year ago

Right, the above fixes the comment about rushing through states. cancel_flight now returns a promise that resolves when the flight is actually cancelled, which will normally happen on the next framerate tick.

Instead of having per-tourstop locking for the flight mechanism, we now use this, trying to cancel any potentially-existant flight before starting a new one. This removes the need for setTimeout bodges, and stops 2 separate tourstops from trying to fly at the same time (since the previous lock was per-tourstop).

lentinj commented 1 year ago
lentinj commented 1 year ago

I've got something locally that works, adding data-ts_autoplay="tsstate-active_wait tsstate-transition_in" to a media embed will:

However the tourstop_observer mechanism is running out of steam. I've bodged it for now to work, but it needs tidyup before I can finally declare this done.

jrosindell commented 1 year ago

All the above is looking great...

Just pulling out a note from #534 which is now closed - that assets in tours may need an alt text for accessibility - this is especially true for audio. I guess that's actually an HTML tag that we just need to manage properly for those assets

lentinj commented 1 year ago

that assets in tours may need an alt text for accessibility - this is especially true for audio

The point there was that there'll be no such thing as an invisible tourstop, as alluded to in the description of the ticket. Either there'll need to be a tourstop on-screen with media controls of some form, or text that describes what the audio / video is on about. Neither of which requires any special support here.

Alt texts and/or transcripts are relevant to #601 though, and another example of metadata to cache from wikimedia commons.

lentinj commented 1 year ago

The above adds the work I was describing yesterday for Vimeo.

Note I've moved the current tourstop state out of being a CSS class to being it's on data-state attribute. The clashes between this working, the block mechanism working, and anything else that may modify CSS classes was becoming ridiculous.

lentinj commented 1 year ago

FWIW my test example was to modify the first tourstop of the tutorial to include a video. This should autoplay on open, not block the flight to the next stop, but once there it will wait for the video to finish before we advance:

diff --git a/views/tour/tutorial_MD.html b/views/tour/tutorial_MD.html
index 489ef67d..fd0fc7e8 100644
--- a/views/tour/tutorial_MD.html
+++ b/views/tour/tutorial_MD.html
@@ -12,6 +12,7 @@ from applications.OZtree.modules.embed import media_embed
     <h2 class="title">{{=T('What is the tree of life?')}}</h2>
     <div class='window_text'>
       {{=T('The tree of life shows how all life on earth is related.  Each leaf represents a different species. The branches show how all these species evolved from common ancestors over billions of years.')}}
+      {{=XML(media_embed('https://player.vimeo.com/video/313477509', ts_autoplay='tsstate-active_wait tsstate-transition_out'))}}
     </div>
     {{ include "tour/tourstop/footer.html" }}
   </div>
lentinj commented 1 year ago

Insert the tour into your local database with:-

curl -X PUT -H "Content-Type: application/json" --user admin \
    http://.../tour/data.json/edge_species \
    -d @edge_species.json

Then go to:-

/life_MDmouse/@Turritopsis=566398?tour=/tour/data.html/superpowers

Or:-

onezoom.controller.tour_start('/tour/data.html/superpowers')
lentinj commented 1 year ago

re: Autoplay, we can consider it an Editorial issue, and say that tours should have non-wait front-covers so users have to press start.

lentinj commented 1 year ago

@hyanwong if deploying this to beta, then the example tours also need to be copied into the DB, see https://github.com/OneZoom/tours.

For the superpowers tour, all media should start playing when you get to the tourstop (i.e. active_wait) and leave once the video / audio is finished. You may need to bully autoplay by clicking on the YouTube video and skipping back.

lentinj commented 1 year ago

This work is done bar the final comment, which I've broken out into it's own issue instead: https://github.com/OneZoom/OZtree/issues/651

hyanwong commented 1 year ago

Neat, I'd love to see a demo.