Create high-quality video for Datamations site

[jhofman]

@sharlagelfand: can you create a high-quality video of datamations that we can use as an animated gif for the soon-to-be datamations website?

instead of the web app, could you make one that's in Rstudio where you type a line of code, then "datamate" it, and then see the animation show up in the viewer pane?

[sharlagelfand]

Something like this @jhofman?

https://user-images.githubusercontent.com/15895337/120372470-3789a900-c2e5-11eb-8923-8a343253e50e.mov

[msftbozo]

This is good @sharlagelfand and @jhofman .We will need to to "contextualize" this a bit more for the website. Meaning, show the graph without any datamations and ask what a typical user would ask " hmm this plot makes not sense, how did they get to this graph?" then we show what @sharlagelfand is showing below. Does this make sense?

[jhofman]

good point.

@sharlagelfand can you try this:

1. show a comment like "# say you analyze and plot some data"
2. show a line of code that renders a plot (skip the library loading and stuff to preserve reader attention) and run it
3. show a comment like "# if you'd like to better understand the steps, just "datamate" your code"
4. then run the datamation command

[sharlagelfand]

Here's another cut! I've "stepped through" each of the steps at the end too, but that could easily be trimmed out

https://user-images.githubusercontent.com/15895337/120377508-378ca780-c2eb-11eb-8806-24bd8e683c86.mov

[msftbozo]

This perfect thanks @sharlagelfand . @jhofman I'll send this to Brenda if you also like it?

[jhofman]

agreed, this is awesome @sharlagelfand!

@msftbozo, sounds great on getting this posted.

cc @dggoldst in case there's any other thoughts / improvements.

[jhofman]

@sharlagelfand looks like we need to trim the video a bit, both for conciseness and file size (under 0.5MB).

can you regenerate it without actually typing letter-by-letter? instead just copy/paste so that the entire first comment appears, leave enough time to read, then the code appears for the plot, leave some time to digest, then the next comment, etc.?

two other requests: 1) can you use theme_minimal or something like that? 😁 and 2) could you add SE error bars to the plot version?

btw, i included some code below to convert the mov to a compressed gif.

ffmpeg -i datamations.mov -vf scale=640:-1 -r 10 datamations/ffout%3d.png
convert -delay 8 -loop 0 datamations/ffout*.png datamations.gif
mogrify -layers 'optimize' -fuzz 10% datamations.gif

[sharlagelfand]

Yep @jhofman I can have another go!

Re: error bars on the plot version, that would mean the code is different between the plot and between the datamation, because we have to actually calculate the sd and plot the errorbars, whereas it's all done internally in datamations

small_salary %>%
  group_by(Work) %>%
  summarize(
    mean = mean(Salary),
    se = sd(Salary) / sqrt(n())
  ) %>%
  ggplot(aes(x = Work, y = mean)) +
  geom_point() +
  geom_errorbar(aes(ymin = mean - se, ymax = mean + se))

or probably using pointrange to avoid the horizontal bit of errorbars:

small_salary %>%
  group_by(Work) %>%
  summarize(
    mean = mean(Salary),
    se = sd(Salary) / sqrt(n())
  ) %>%
  ggplot() +
  geom_pointrange(aes(x = Work, y = mean, ymin = mean - se, ymax = mean + se))

vs what we'd show for datamations

small_salary %>%
  group_by(Work) %>%
  summarize(mean = mean(Salary)) %>%
  ggplot(aes(x = Work, y = mean)) + 
  geom_point()

Is that fine?

[jhofman]

ah no worries on the error bar addition if you think it causes weirdness / confusion.

if you do want to go with it, i'd say geom_pointrange to match instead of geom_errorbar, but happy to leave it off if you think that's better.

[sharlagelfand]

Here are a couple versions @jhofman! It's pretty tough to get it at the 0.5MB mark, so the first one here shows clicking through the frames, but it's 1.2mb:

And the second excludes that and still almost 1mb:

But you can see the GIFs are fairly choppy, especially with grid weirdness and overlapping, and I have to rush through the commands a bit. But let me know what you think!

[sharlagelfand]

Just one more cut of the "full" video, but everything is scaled down a bit - I realized the actual height / width was probably too large!

[msftbozo]

this looks good @sharlagelfand what's the total size?

[msftbozo]

this is still 1MB :( @jhofman what do u think?

[jhofman]

this is with all of the compression and whatnot, right?

i agree that it could look a bit better.

do you think below .5mb is possible @sharlagelfand?

i can check back later today or over the weekend.

On Fri, Jun 4, 2021 at 1:44 PM msftbozo @.***> wrote:

this is still 1MB :( @jhofman https://github.com/jhofman what do u think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/datamations/issues/67#issuecomment-854899068, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAATNS3RJ3LNE3OCYXXES6LTREGHZANCNFSM455FAHHQ .

[sharlagelfand]

Yeah this is with all of the compression - let me play around with settings a bit and see if I can get it smaller and without some of the bugginess!

[msftbozo]

that would be awesome thank you @sharlagelfand .

[sharlagelfand]

Coming in at 479.5kb! 🏆

I clipped the original video a bit to remove some of the parts where you're just "waiting" at the console, also made the actual gif smaller in size (width 375 instead of 640, so reduced a fair bit), removed every 10th image (just to reduce the number of frames a bit), and increased the fuzzy color matching from 10% to 12% (to get those extra few dozen kb off).

Here is the code to reproduce:

ffmpeg -i datamations.mov -vf scale=375:-1 -r 10 datamations/ffout%3d.png
rm -f datamations/*[0].png
convert -delay 8 -loop 0 datamations/ffout*.png datamations.gif
mogrify -layers 'optimize' -fuzz 12% datamations.gif

[msftbozo]

This looks good to me @sharlagelfand thank you so much. It didn't look too bad quality wise. @jhofman the animated gif is good but it's a bit hard to follow if someone wanted to really "step through things" so I wonder if we should also link to the original video sharla made which would allow u user to pause, and it's slower to you can actually follow better. Thoughts? For now, we can use this animated gif.

[jhofman]

awesome thanks @sharlagelfand!

we’ll push back for a larger file size limit but glad we have this.

i’d be happy to use this along with an explainer image or banner that clarifies things. that could avoid the need for an extra video, and we’ll need that at some point anyway.

sound good @msftbozo?

On Fri, Jun 4, 2021 at 5:30 PM msftbozo @.***> wrote:

This looks good to me @sharlagelfand https://github.com/sharlagelfand thank you so much. It didn't look too bad quality wise. @jhofman https://github.com/jhofman the animated gif is good but it's a bit hard to follow if someone wanted to really "step through things" so I wonder if we should also link to the original video sharla made which would allow u user to pause, and it's slower to you can actually follow better. Thoughts? For now, we can use this animated gif.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/datamations/issues/67#issuecomment-855007831, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAATNS6BNAOLY4IYHA3VY5LTRFAWRANCNFSM455FAHHQ .

[msftbozo]

yes agreed @jhofman . I'll send this animated gif now and let's think of an explainer image (maybe even a schematic?).

[jhofman]

@sharlagelfand: we got approval to double the file size :)

can you regenerate with best quality you can get at or around 1MB?

[sharlagelfand]

Awesome! This one is 977kb, so just under 1MB:

It's quite a bit larger (600px wide), doesn't exclude frames, and has fuzzy colour matching back down at 10% - seems to have way less bleeding of axes and fuzziness around the points in general.

Code to reproduce:

ffmpeg -i datamations.mov -vf scale=600:-1 -r 10 datamations/ffout%3d.png
convert -delay 8 -loop 0 datamations/ffout*.png datamations.gif
mogrify -layers 'optimize' -fuzz 10% datamations.gif

[jhofman]

awesome, thanks. this looks a lot better. i'll forward it to folks.

is there somewhere you can make the full res version available in case, in my infinite free time, i get a chance to experiment w/ other compression approaches?

thanks!

[sharlagelfand]

Yep! Full video is here and full gif is here - don't want to clutter the repo with them since it's nearly 40mb between them

[sharlagelfand]

The GIF is live! https://www.microsoft.com/en-us/research/project/datamations/ Site looks awesome, I like the background and description.

Closing this!