Feature Request: Draw on Slides

[LauraRK]

I would really love the ability to write on my HTML slides as needed while presenting. I saw that Akkana Peck https://shallowsky.com/blog/speaking/drawing-on-slides.html was able to do that with her slide making code and I looked at her js files and tried to copy the appropriate parts and get it to work on my slides but I am not knowledable enough to make it work. I would love the shift D bring up a small ability to draw and then keep scrolling feature. I have tried a variety of solutions to this and none of them work the way I want them to, which is basically I want to draw and move on with out getting asked if I want to save anything or having my drawings stay present on the screen when I move to the next slide. I also like how her solution just has a little object with the colors and not a big obtrusive feature. I will continue trying to make it work but I think this would be a great thing to have in xaringanExtra. Thank you.

[mattwarkentin]

Cool idea, @LauraRK. I actually saw your post on the RStudio Community forum and it got me thinking about this feature. I have some thoughts, but I want to run it by Garrick, since he's both a JS and xaringan expert!

@gadenbuie, do you think it would be possible to define a CSS class that can be added to a slide in the normal fashion, and then when that slide is the visible slide (.remark-visble), use some JS magic to wrap the slide (.remark-slide-container) in a transparent <canvas> and record the mouse movement and mouse down events to draw? Actually, it could probably be more general - the class could be added anywhere (.canvas[] ?) and wrap the div/parent div so you could arbitrarily define the drawing surface. Thoughts?

If you think this is reasonable, I would be interested in making an attempt to implement it as I have been actively working on my JS skills recently.

[LauraRK]

Thank you for maybe taking this on. It might be helpful to see the code @akkana wrote at the bottom of navigate.js in her presto package.

[rpruim]

I'm also very interested in this idea. I'd certainly be willing to test, and perhaps to help. I'm familiar with javascript/css/etc and a user of xaringan, but I've not done any sort of deep dive into how xaringan is written.

It would be especially great if what was written persisted on slides, so that if you leave and return, what was written is still visible. More complicated would be the ability to save all the writing (say for students to return to later) while also being able to clear the writing (say for another presentation of the same slides).

[rpruim]

I wonder if fabric.js might be useful for this: http://fabricjs.com/

fabric.js is used in this R package: https://cran.r-project.org/web/packages/fabricerin/index.html

[mattwarkentin]

Thanks for sharing, @rpruim.

Looks like the {fabricerin} package already works with {xaringan} to create a standalone canvas for drawing etc.. I have already hacked together enough JS to add a canvas to the entire visible slide with the press of a button. Perhaps connecting my JS with this package would get some of the way to a working feature.

[rpruim]

That's just the sort of thing I was thinking -- a transparent canvas over the entire slide on which one can draw. The ability to toggle on and off is important especially if the slide has other interactive things on it (links, play buttons, etc.)

Let me now if you get to a point where you need some to test drive something.

[rpruim]

Looks like (at least in the default use) {fabricen} has an erase feature, but not a "clear all" option. The eraser seems to be very small -- same size as the pen? -- so erasing isn't very easy, especially if you want to erase a lot.

[LauraRK]

Thank you both. It is fine with me if I can't save it at this point, although I agree it is a nice long term feature. I would think that perhaps when you print it to pdf they would save if they are written via canvas? That is my next thing to figure out.

[rpruim]

@mattwarkentin , I'm wondering if it makes sense to think about a more general solution -- basically a way to add a transparent canvas element to fill a div (and resize as needed). That div could be a slide, but could also be any other element -- including from HTML that is generated in other ways. {xaringan} could then use this more general solution applied to its slide divs. This is a little out of scope for {xaringan}, but I could see this being generally useful for other types of documents as well.

[rpruim]

Quick note on erasing: {fabricerin} has a gumSize argument that controls the size of the eraser.

[mattwarkentin]

Off to a decent start...

https://user-images.githubusercontent.com/27825069/106315834-d1f3b600-6239-11eb-903a-a407814249b0.mov

[mattwarkentin]

I would think that perhaps when you print it to pdf they would save if they are written via canvas? That is my next thing to figure out.

@LauraRK, the drawings will stay even when you flip between slides, by default, or can be removed if you toggle off the canvas (I will add a "Clear" button also). The drawings will appear if you decide to "print" the slides to PDF, for example.

[LauraRK]

That is wonderful. Thank you so so much for working on this. This is what I want. 👍

[LauraRK]

When you are ready to have it tested out let me know and hopefully I can figure out how to get it working. I am a bit new to this process flow, using Github, knitting slides, etc.

[mattwarkentin]

I think the basic functionality is ready to be tested. The canvas can be toggled by pressing "d" on any slide. The controls for stroke colour and thickness show up in the top right corner.

For various reason I didn't end up using the {fabricerin} package. Instead I just used the fabric.js library directly, and wrote my own JS/CSS to get things working. This has the side-benefit of also not depending on another R package, should Garrick consider this for inclusion in {xaringanExtra}.

It turns out erasing is tricky. In {fabricerin}, what looks like erasing is actually just painting over your current drawing with the same colour as the canvas background colour. This isn't really viable for xaringan since painting over your drawing will obfuscate the slide content underneath. As of now, you can only clear the entire slides drawing by toggling the canvas off and back on again.

You can try out this development version by running this code in R:

remotes::install_github("mattwarkentin/xaringanExtra")

And all you need to do is add the following code to your xaringan Rmd file:

xaringanExtra::use_fabric()

Please let me know if anything doesn't work as expected, or whether you think any features are missing.

[rpruim]

I haven't tried it yet, but I have a question: if I'm using my ipad, how do I toggle into drawing mode?

I'll be giving it a try shorty.

[mattwarkentin]

Hmm, thats a good question that I don't know the answer to. I suppose that issue applies to all of the behaviours triggered by keystrokes. Perhaps I could write some JS to detect the device and if its mobile, add a button to toggle draw mode. Thoughts?

[rpruim]

Gave it a quick try. Was able to draw. I like the pen and color adjustment widgets. Things I'm noticing:

• hitting D a second time clears the writing
• for a slide built gradually, each portion of the build is a "slide", so when you advance you lose any writing from the previous portion of the slide.
• You are able to write on top of R code. I made a fork of {fabricerin} to experiment a bit, and one quirk I'm seeing is that I can't write on top of R code for some reason.

Haven't tried it on an ipad yet.

[rpruim]

Since drawing is easiest on a tablet, adding some support for tablets would be really good.

[rpruim]

In case my comment about drawing on top of R code was unclear, here is a screen shot:

[mattwarkentin]

hitting D a second time clears the writing

This is the expected behaviour, for now. Open to suggestions.

for a slide built gradually, each portion of the build is a "slide", so when you advance you lose any writing from the previous portion of the slide.

Good observation. Hmm, I will have to think about whether this can be handled gracefully.

Since drawing is easiest on a tablet, adding some support for tablets would be really good.

Thinking about this some more, I think it would be relatively straightforward to figure out how to toggle draw-mode on a mobile device, but the bigger issue is that screen gestures on a tablet changes slides, so I don't think drawing would play nicely. I need to think about this some more.

In case my comment about drawing on top of R code was unclear

This looks like you're using {fabricerin}, yes? I don't think this is an issue with my implementation in xaringan. At least not in my tests.

https://user-images.githubusercontent.com/27825069/106335572-180d4180-625b-11eb-8a2e-c1ceb0bd2ae6.mov

[rpruim]

I didn't have any trouble writing on R code with your stuff either.

I was using my slightly hacked version of {fabricerin} -- basically adding a small div hold the controls and then inserting into it the larger canvas rectangle so that you can draw over what comes below. The R code behavior surprised me, but I haven't done more digging to figure out why it behaves that way.

[rpruim]

I'll have to think a bit more about ipad gestures, but perhaps swipe left and swipe right could be used to advance slides rather than tap -- at least when the fabric stuff is turned on.

[LauraRK]

It works for me :) I have one small request, and that is if perhaps a few colors could be there as a square, or maybe if I reset the color it would stay that color. Right now it is black and almost all my slides are dark blue, so every time I click d I would have to change the color. It is basically exactly what I want though besides that, I love how it is pretty basic and not fancy. I am very grateful. Makes me believe in the good of humanity :) simple, easy, love it 💯 Thank you!!!

[rpruim]

First, I agree that the basic functionality is there. So that's great.

In terms of colors, two thoughts come to mind:

• Perhaps there could be global setting for the default color as an argument to use_fabric()? That way the default color could be chosen to reflect the colors used in the slides. Many people will just want one color, but will want to choose it to be visible on their slides. Perhaps this could be overridden on a slide basis (in the RMarkdown).
• For multicolor stuff (which is probably less important, but some people might really want), it would be nice to have a small number of quickly selectable colors. But I also think we don't want too much clutter. Like @LauraRK , I think simple, basic, and functional is nice.

[LauraRK]

OK, so I did test it on my touch screen and one thing that is causing issues is that when I am writing it thinks I am advancing slides, because I believe a touch also advances slides, so it works when I write with my mouse but not my stylus or finger. Maybe there is a work around if I put in some other option when I use the drawing?

[LauraRK]

I see above reading @rpruim 's comments that it is the tap that is the problem. This is a problem on my touch screen laptop as well and since I want to use the touch screen to write that causes a challenge.

[gadenbuie]

This looks great and is really exciting! Thanks everyone for working on this.

A few thoughts about implementation:

• Instead of a keyboard shortcut, I'd rather add a button with a pencil icon to the corner of the slide. Clicking the icon enables the writing feature. This makes the interface easier to access in touch devices.
• We'd give two options for exiting drawing mode:
  • Turn of drawing mode but leave the drawing on the slide
  • Clean the canvas (and exit drawing mode)
• While drawing is enabled we need to block the remarkjs touch event listeners. I've done this in another extension, I think panelset blocks both touch and drag events.
• I'd rather pick a reasonable default for the line width and simplify the ui a bit to take out the line width slider and make some room for a color picker (maybe three colors?) We'd pick three good colors (e.g. red, green, blue), but we can let the user set the palette in the use_fabric() function.
• Regarding continuation slides I think our options are to either attach the canvas to an individual slide -- i.e. a new canvas for each continuation -- or to have one canvas for the entire slide deck. Maybe this could be a user option, but I'd vote for the first. remarkjs treats each partial slide as if it were a completely new slide and it's probably overly complicated to try to sync all the drawings in partial slides.

Thanks again @mattwarkentin for the work on this!

[gadenbuie]

Oh and we're going to need a way to erase lines that isn't painting over existing lines. This might be complicated. I'd aim for the type of eraser where the user draws an "erase" line (which is probably white) and then any existing drawn paths that intersect that line, and the erase line itself, are removed from the canvas.

[mattwarkentin]

Thanks everyone for the great feedback! I will keep working on these changes and see what we can come up with.

[rpruim]

Seems like we are getting really close. I like the suggestions of @gadenbuie. A few more things to think about:

• I'd suggest that line width should also be settable in use_fabric().
• Perhaps the number of colors in the palette could be flexible, and if only one color is specified, then no color controls need to be displayed (for people who want to take a minimalist approach).
• I agree that per-slide is better than per deck -- but I'm sure people will ask about this.
• Is it workable to have a way to advance slides while in drawing mode?
• Would it be better to have "clear" be a separate option from exit drawing mode? That way if you make a mistake, you can just clear and keep drawing rather than exit drawing mode and re-enter it. Also, I fear inadvertent clearing if the interface makes the operation of clearing too closely related to the operation of leaving drawing mode.

[gadenbuie]

Thanks for the comments @rpruim!

• I'd suggest that line width should also be settable in use_fabric()

Agreed.

• Is it workable to have a way to advance slides while in drawing mode?

Maybe with the arrow keys? But I think for consistency we'd want to block those too and require that you exit drawing mode to move between slides. In fact, we'd want to block all key event listeners since we don't want to accidentally trigger slide advancing and other remarkjs features while in drawing mode.

• Would it be better to have "clear" be a separate option from exit drawing mode?

I agree, I think it would be more intuitive to have these as separate actions. The flow would be:

1. Click the pencil to enter drawing mode
2. In drawing mode:
   1. Click the exit button to "freeze" drawing and go back to slides mode (or press Escape)
   2. Or click the clear button to clear the current drawing, but staying in drawing mode

Exiting to a clean slide would require "clear" and then "exit" but the buttons would be next to each other, so that's not a big ask.

[rpruim]

Regarding navigation, perhaps small arrows to the left and right of the pencil when in drawing mode?

Among other use cases, I can imagine wanting to quickly flip back to a previous slide and then return and keep drawing. But I consider this a minor issue and not something that should block progress or be a deal breaker.

If erasing proves challenging, would an undo button (to remove the last drawn line) be easier -- at least as a first go? The most common things you want to remove are very recent additions, so this would get us most of the way.

[rpruim]

Another note regarding controls (especially on a touch device). There are potentially three states: drawing, manipulating, and neither. fabric.js has a mode where you can grab the things you have drawn, move them, resize them, etc. Perhaps we have no need of that, but it exists. See isDrawingMode at http://fabricjs.com/fabric-intro-part-4#free_drawing.

Also, I found a relatively easy way to disable the fabric stuff using CSS: set pointer-events: none so that the events pass through to the things below. That's what I'm currently doing here when you hit the "toggle" button.

https://rsconnect.calvin.edu:3939/connect/#/apps/139/

[LauraRK]

Thank you, let me know when you are ready for me to do more testing. I like the pencil icon idea for easy usability. I also like the eraser idea. I agree with @gadenbuie that basic reasonable defaults are best at this point in regards to line width.

[mattwarkentin]

Will do. It's going to take me a little bit to back track and redo some things properly in order to achieve what we've discussed. But I really like what I have so far and I'm excited to use this feature implementation as a bit of a learning opportunity.

@gadenbuie I'm running into a few xaringan-specific challenges. Any advice on best practices for ensuring the slideshow is really ready? An event listener for "DOMContentLoaded" doesn't seem to be good enough, probably because it takes a moment for remark.js to do its thing. I've got a hacky solution for now using setTimeout().

[mattwarkentin]

Actually, I think I found a good solution using slideshow.on("afterShowSlide", ...)that actually simplifies things quite a bit.

[rpruim]

Small idea on colors: If the palette of colors is empty, then include a generic color picker. If a non-empty palette is provided, then display color swatches for choosing a color.

I guess if exactly one color is provided, that color could be used to seed the generic color picker.

[gadenbuie]

Any advice on best practices for ensuring the slideshow is really ready?

@mattwarkentin This is the setup I use with the broadcast feature (and most of the extensions): https://github.com/gadenbuie/xaringanExtra/blob/93429efefb268a4df2f43935304a8f781f73ece8/inst/broadcast/broadcast.js#L231-L251

I'm surprised that DOMContentLoaded isn't working. remarkjs is initialized with a script tag in the body (unless something has changed recently), which means that DOMContentLoaded shouldn't fire until after the slideshow is initialized.

Another idea would be to check for the slideshow object and recursively call an init function with a small delay until slideshow exists or you try too many times.

Btw, feel free to open a draft PR here if you want or whenever you want. That would make it easier to test/give feedback.

[mattwarkentin]

Okay, so I'm pretty happy with how it's coming along. Probably about 90% feature complete. Just making sure touch-device support works as expected.

@gadenbuie Would you be willing to have a quick chat? I'd love to get your feedback on a few things, but I'm mindful that everyone is overburdened with responsibilities these days.

[rpruim]

@mattwarkentin , is this ready for some testing? If so, where is it? I looked at your repo, and it looks like the last update was 4 days ago, so it must be somewhere else.

[mattwarkentin]

@rpruim It can be tested now by re-installing the dev version from my repo. Thanks for your patience. This version should provide full support for touch devices, @LauraRK.

Currently, there is a known issue with how the canvas/mouse coordinates are affected by the remark slide scaler, so the line may not be drawn precisely where your cursor is. I am working on fixing this.

[rpruim]

I just installed it, and I see use_fabric() and the help for that function. But the 'd' shortcut isn't doing anything as far as I can tell. Is there another step I'm omitting?

The short-cut is also not listed in the help screen.

I've restarted the R session and rebuilt the slide deck -- no changes.

I can see that canvas stuff is being loaded:

packageVersion("xaringanExtra")
## [1] ‘0.2.4’

[mattwarkentin]

Sorry, I should have explained things better. The API has changed a lot since the last version:

• use_canvas() replaces use_fabric(). This is because fabric.js is no longer used as the backend for a variety of reasons that I will describe when I submit the PR.

• The toolbox of buttons that control the drawings do not show up when the presentation is first loaded, but they will/should show up (in the top right corner) when you start navigating slides. The "d" keypress only toggles the visibility of the toolbox. To initiate drawing, click the edit/drawing icon.

[rpruim]

I'm not seeing the toolbox, whether or not I hit the D key. I typed use_fabric(), but I meant use_canvas() -- I did get that correct.

[mattwarkentin]

Does the toolbox not show up when you change slides?

[rpruim]

That's correct:

[mattwarkentin]

Hmm. Can you let me know which browser you are using? Also, could you check the console for any errors?

[rpruim]

Looks like an undeclared variable:

canvas.js:130 Uncaught ReferenceError: i is not defined at canvas.js:130 at r (remark-latest.min.js:1) at HTMLCollection.t.forEach (remark-latest.min.js:1) at HTMLDocument. (canvas.js:129)

[rpruim]

This was in chrome. I also tried in firefox.