Support Video display

[light-matters]

Manipulating/creating a Wolfram Video returns a strange format that no visual tool knows how to interpret.

(w/VideoTrim [12 24])

Returns

(Video /home/ctw/Documents/Wolfram/Video/VideoTrim-5ef8ad95-051d-4e6f-a1b2-1f1e24b62217.mkv (-> Appearance Automatic) (-> AudioOutputDevice Automatic) (-> SoundVolume Automatic))

We should some implement some sort of 'kindly' support (e.g. metadata) that allows visual tools such as Clay and Clerk to display these outputs correctly.

[holyjak]

@daslu Does Clay/Kindly support metadata on return values to know how to display them? So that when we return (Video "/file/path/..") we could add some metadata to tell Clay to get the video path (i.e. second on the data structure) and display as a video? 🙏

I obviously cannot simply (with-meta '(Video "path..." #_...) {:kind/video true}) because that does not communicate that the value of interest is a local file path, and how to get it out of the data structure (here, a List). I don't know if :kindly/options could be leveraged? Currently, it seems from the Kindly Book that only embedded video from youtube are supported... Perhaps I should rather use hiccup of html and the video tag - but still need to instruct the viewer tool how to get from the actual value to that... Perhaps something like :kind/custom (fn [wolfram-video] ^:kind/hiccup [:video [:source {:src (second wolfram-video)}]]) ?

[daslu]

Hi.

I'm copying a few details from a private chat with @light-matters:

• kind/video supports only youtube, and following your proposal, we will add support for local files.
• kind/hiccup is supposed to support anything that HTML can do.
• If you render through Quarto (incliding Reveal slideshows), you include in your markdown comments anything that Quarto supports: Videos.

If you are pointing to files, you need to put them under a directory which is copied by Clay under the target directory, as explained in Referring to files. Then, you can just refer to them with the path relative to the target directory.

There is also some support for custom functions, as you suggested: Functions

Does it seem to make sense?

[holyjak]

@daslu thank you! How would we make Clay output Quarto's {{< video local-video.mp4 >}} ?

@light-matters perhaps we can just do

^:kind/hiccup
(let [url (->> (wl/eval (w/VideoTrim ...)) second)]
  [:video [:source {:src url :type "video/mp4" }]])

WDYT?

Or, using Clay fns:

(defn wolf-video->hiccup [wl-result]
  {:pre [(list? wl-result) (= 'Video (first wl-result))]}
  ^:kind/hiccup
  [:video [:source {:src (second wl-result) :type "video/mp4"}]])

(k/fn [wolf-video->hiccup (wl/eval ....)])

The fn could also make the video to another place, if desired...

[daslu]

Hi!

Quarto Markdown can simply be included in Clojure comments or kind/md.

I hope to add some better support to the other ways during the coming weekend.

[light-matters]

My feeling is that we should avoid explicit quarto notation where possible. Whether the output uses quarto or not to garner the particular effect seems like an implementation detail and not something for Wolframite to worry about. An argument can be made for the case of the website etc. where the exact output goal is known in advance.

[light-matters]

@daslu thank you! How would we make Clay output Quarto's {{< video local-video.mp4 >}} ?

@light-matters perhaps we can just do

^:kind/hiccup
(let [url (->> (wl/eval (w/VideoTrim ...)) second)]
  [:video [:source {:src url :type "video/mp4" }]])

WDYT?

Or, using Clay fns:

(defn wolf-video->hiccup [wl-result]
  {:pre [(list? wl-result) (= 'Video (first wl-result))]}
  ^:kind/hiccup
  [:video [:source {:src (second wl-result) :type "video/mp4"}]])

(k/fn [wolf-video->hiccup (wl/eval ....)])

The fn could also make the video to another place, if desired...

Can we highjack the return from Wolfram and add the kind information automatically (after detecting the return)?

[holyjak]

Can we highjack the return from Wolfram and add the kind information automatically (after detecting the return)?

That would be best, but not possible with the way kind/fn works now, as far as I understand it. It either wants a vector with the fn and value (where value in our case is the returned (Video ...) or a map with.. @daslu we would want to be able to do

^{:kind/f (fn [the-value] ^:kind/hiccup [:video [:source {:src (second the-value)}]])}
the-value

because then we could leave the value as-is and just attach metadata to it. However that is not how kind/fn works, as far as I see. It requires either a vector or a map. WDYT?

[light-matters]

Maybe we should avoid the kind function? Let's discuss this later.

[daslu]

@holyjak having :kind/f as metadata is an interesting idea worth thinking about.

Anyway, kind/video should support a file path, and that will happen in the coming Clay release tomorrow.

[daslu]

Oh, sorry, I see what you mean now.

You need the custom function to avoid changing the value (Video "path/to/file.mkv" ...) and just define the way it is rendered.

I'll sleep on the :kind/f metadata idea.

[daslu]

Clay version 2-beta17 adds kind/video support for urls, and also additional support to passing the function as metadata as @holyjak proposed.

See: https://scicloj.github.io/clay/#functions

So, we can do something like:

(k/fn (w/eval ...)
  {:kindly/f wolf-video->hiccup})

Thanks for these helpful ideas.

[holyjak]

@light-matters could you be so kind and verify that https://github.com/scicloj/wolframite/pull/131 works for you?

@daslu This is amazing, thanks a lot! Does the meta we attach here https://github.com/scicloj/wolframite/pull/131/files#diff-62bf9ddf1ee0d76b66dec5a61c7d8766df142e9471c228ce2e7925b67a5403d3R18-R20 look good to you?

[daslu]

Looks good!

[holyjak]

@daslu I am not sure how the video display is supposed to work. Is this correct?!:

^{:kindly/kind :kind/video} ["file:///Users/me/Movies/example.mp4"]

(I'd prefer direct dependency on the kindly lib, hence the DIY metadata)

[light-matters]

From the other thread (https://github.com/scicloj/wolframite/pull/131),

The format

(k/video {:src "./resources/video/Baby.mp4"}) 

works for me rather than file://... The DIYness can be extracted from whatever k/video produces.

[light-matters]

We also need to make those folders visible to clay.

[holyjak]

Thank you, the code works now.

However, as you point out, the video must be in a folder which Clay can read. Not sure how to ensure that, when we work with a video from Wolfram. When using a notebook, we could start with having the video in an appropriate folder - just need to document this well. But what about the output of w/Video and friends? Do they return relative, or absolute paths? If we start with a relative path and they keep it relative, and in the same folder, then we are good. If they absolutize it, or add files to a different location, then we have a problem.

We could/should perhaps check that the url is relative, and return something else such as hiccup with a warning message, when it is not...?

[light-matters]

For reference, absolute paths can be given to the clay options, see this thread

This is particularly necessary for operations like VideoTrim which store the results centrally. Probably we need to pass some of these default paths to whatever viewer the user uses at startup.