sagemathinc / cocalc

CoCalc: Collaborative Calculation in the Cloud
https://CoCalc.com
Other
1.17k stars 216 forks source link

Whiteboard UI/UX #5874

Open mforbes opened 2 years ago

mforbes commented 2 years ago

Here are some comments about the whiteboard from a quasi-first-time user experience. It is not quite at the point that I can use reliably, but getting there. For background, my previous experience is drawing with a Surface Pro 3 using OneNote.

For this exercise, I am not using my Surface, but am trying out my new Wacom One tablet with a pen. This is connected to my computer which allows me to use my keyboard too (a new experience for me). I am writing comments in the order which they marred my initial experience. After using things for a bit, I discovered features that made some of these annoyances irrelevant - I have commented about this with Edits so you can see what bothered me at first.

Erase

My biggest issue is how hard it is to erase: I expected the eraser to erase elements. Instead it seems like it is actually white paint, and it paints over the grid. While this may be a useful tool for crafting careful drawings, I would make the default erase tool erase elements (and would like this element-erase tool the be what I get when I click the button on my tablet.)

Move and Select

It would be nice to be able to click-select objects rather than having to use the rectangle tool. (Maybe with Shift). Edit 1 Now I see that you can just drag objects. That is nice (but was not obvious at first). Edit 2 Aha, now I see that you can double-tap to select. That works nicely - an intro video showing this is needed, or maybe a live animation on the Whiteboard showing how to use features? I think the UI is actually pretty good, just a bit hard to figure out without hints.

Keyboard Shortcuts

There was no clear list of keyboard shortcuts, and had to randomly guess to figure them out. I would like "p"en and sele"v"t? close to my delete key (until pen-erase works) for example. Edit I now see them in the tool tips: I did not notice that for some time and was looking for a help or menu item with a list.

Toolbar

I find the pen and tool position intrusive: It will probably just take some getting used to, but the left side of my tablet is my working space, so for right-handed individuals, it is probably better to have the toolbar on the right side. I also expected the "Pen" selector to get out of my way once I select a new pen, but perhaps keeping it active is better (until a custom toolbar exists). Edit: Now I see that if you click pen again, it hides the pen-selector. Perhaps it would be better if the selector did not appear by default (i.e. when switching from select to pen).

Wacom Specific comments

Performance

There can be latency issues which significantly impact drawing. Sometimes this is so bad as to be unusable for writing, but other times it is reasonable. I noticed it was specifically bad when mirroring my Wacom display, but when using the Wacom as a secondary display without mirroring, it was reasonable. Thus, there can be issues at the OS level. Is there some way to have symbols drawn fully on the client before sending them so that the interaction is smooth?

Update: I could not reproduce the mirroring latency issue today, signalling that it might depend on things like the network connection, or machine load (was running Zoom at the time too.) Will try again to see if I can find a reproducible way to cause this problem.

williamstein commented 2 years ago

I expected the eraser to erase elements

Can you define very precisely what you mean by "erase elements"? Do you mean that you want to delete the element? If so, you could select the element, then click the trash can button (or hit the delete key). Given that this is literally your biggest issue out of all of the other issues, and it would be very helpful if you can define it, since I don't understand what you mean at all.

... intro video or docs ...

Unfortunately, as you have found, we have not created even a shred of documentation yet for the whiteboard.

Perhaps keyboard shortcuts could be only active while holding down. Thus, if you are holding "v", then you can select, but when you let go, the UI will go back to the previous tool (i.e. the pen).

That's an interesting idea, but isn't it different that literally all other 1000 drawing programs out there, so likely to be confusing? Maybe something like that, but with a modifier key, e.g., shift+v or control+v, would work? Note that the reason for the shortcuts is that they are consistent with dozens of other similar whiteboard/drawing programs, which have probably mostly copied things from pioneering Adobe products, I would guess. For example, "v" for select makes no logical sense to me, but it is standard.

Perhaps it would be better if the selector did not appear by default (i.e. when switching from select to pen).

Maybe the state could just be what it was when you last used it? E.g., the very first time you click pen it shows all the option, but if you click pen again to close them, then switch to another tool, then click on pen again, they are still closed? That would be easy to implement, but would have the advantage the you see the options the very first time you click a tool.

How to get my Wacom pen "Erase" button working (or customizable)?

Can you send me a link on amazon.com to your model (or a close model) of pen? I currently don't have access to anything like that hardware, so probably need to buy it. I fortunately do have Windows tablets with touch screens...

... It looks like this is an issue with all apps including OneNote, so a solution probably requires something at the OS/Driver level.

It's likely that it could be solved at the webapp level, but doing so breaks some basic accessibility assumptions. Thus it could be something that is an option that is off by default.

There can be latency issues which significantly impact drawing. Sometimes this is so bad as to be unusable for writing, but other times it is reasonable. I noticed it was specifically bad when mirroring my Wacom display, but when using the Wacom as a secondary display without mirroring, it was reasonable.

I don't even really know what a "Wacom display" is, so it's hard for me to think about how to debug this. Can you provide a link?

williamstein commented 2 years ago

It might be very useful if you could use your phone to shoot a video showing how the hardware is setup and working together, and also illustrate the latency issue with the display.

mforbes commented 2 years ago

Can you define very precisely what you mean by "erase elements"? Do you mean that you want to delete the element? If so, you could select the element, then click the trash can button (or hit the delete key).

Exactly: I would like to be able to do this very easily. One option would be by pressing a button on my pen (or using the back of the Surface pen). Another might be rapid motion (scrubbing) over the elements (useful for devices that don't have buttons). I really dislike having to move the pen/mouse away from where I am writing to erase something.

Perhaps keyboard shortcuts could be only active while holding down. Thus, if you are holding "v", then you can select, but when you let go, the UI will go back to the previous tool (i.e. the pen).

That's an interesting idea, but isn't it different that literally all other 1000 drawing programs out there, so likely to be confusing? Maybe something like that, but with a modifier key, e.g., shift+v or control+v, would work? Note that the reason for the shortcuts is that they are consistent with dozens of other similar whiteboard/drawing programs, which have probably mostly copied things from pioneering Adobe products, I would guess. For example, "v" for select makes no logical sense to me, but it is standard.

Ah: I have really only used a couple of drawing programs, so must defer to your experience here. I have never before used a keyboard together with a tablet, so am getting used to it. I don't want to have to look at my keyboard when selecting different options, and "v" and "p" are rather far away from the trackpad (which I am using to scroll). I will try to make a video.

Maybe the state could just be what it was when you last used it? E.g., the very first time you click pen it shows all the option, but if you click pen again to close them, then switch to another tool, then click on pen again, they are still closed? That would be easy to implement, but would have the advantage the you see the options the very first time you click a tool.

Sounds like a good option.

How to get my Wacom pen "Erase" button working (or customizable)?

Can you send me a link on amazon.com to your model (or a close model) of pen? I currently don't have access to anything like that hardware, so probably need to buy it. I fortunately do have Windows tablets with touch screens...

https://www.wacom.com/en-us/products/pen-displays/wacom-one

As far as I can tell, it is the best inexpensive (<$500) input device that has a screen builtin. It is nice and large, and functions as a good portable extra monitor. Wacom also has inexpensive (<$100) tablets without a display, but they take more work getting use to as you need to draw in one place, but look in another. (There are also much more expensive big devices). Wacom seems to be the standard for graphic artists.

There can be latency issues which significantly impact drawing. Sometimes this is so bad as to be unusable for writing, but other times it is reasonable. I noticed it was specifically bad when mirroring my Wacom display, but when using the Wacom as a secondary display without mirroring, it was reasonable.

I don't even really know what a "Wacom display" is, so it's hard for me to think about how to debug this. Can you provide a link?

The Wacom One is an input tablet with a pen, but also functions as an external display (second monitor). On my Mac, when I "mirror" displays, the performance is bad. I will try to make a video later tonight.

williamstein commented 2 years ago

Thanks. I've ordered a Wacom One which I'll get Thursday. Many of the negative reviews on Amazon are about compatibility issues with various software (e.g., One Note), so I'll get ahead of that for CoCalc! I didn't even know this thing existed until you told me about it.

mforbes commented 2 years ago

I uploaded a couple of videos showing my setup in my test project. I demonstrate how I would like Erase to work. Unfortunately I could not trigger the latency/lag issue... it must be more transient, depending on network and/or computer load. I will try to record if it happens again. (I am editing comments in my main post with Update:... let me know if I should just reply instead.)

williamstein commented 2 years ago

Thanks. The second longer video is truncated at 1 minute. However, that seemed like enough for me to get a good sense of what you want. In particular, I see how you want to delete. My guess/hope is that erase just sends a touch event to the browser that has some modifier, and I can just log it and see when I get my Wacom one tomorrow.

There's also a library https://pressurejs.com/ which I can use to add pressure support that should support all types of pens.

mforbes commented 2 years ago

(I meant to pause the video, but accidentally stopped it.)

Pressure support looks cool, and is good for sketching, but I think might not be ideal for teaching, so maybe should be optional somewhere. (I can probably turn that off in the pen settings though, so there might not be any reason to keep this customization in CoCalc except to support less-functional input devices.)

williamstein commented 2 years ago

I could make it so when you configure a pen there's optionally a range of pen widths (rather than a single width), and pressure determines which width in that range is used.

mforbes commented 2 years ago

Maybe... would have to try. I am not sure how accurate my pressure is right now:-)

williamstein commented 2 years ago

Update:

Erase: "My biggest issue is how hard it is to erase"

I did some more investigation and there are two kinds of erase implemented in a lot of these systems: object and pen.

I bought a Wacom tablet and reverse engineered how their button-to-erase works, and implemented it as object erase. This is now implemented and live.

Also, you can now drag to select (not edit) a single object (that didn't work before), and type the "delete" key on your computer to delete something, or click the trash can icon below the object.

I think the erase tool in the pen dialog should be this white pen version of erase. There could definitely be another tool elsewhere that does object erase.

When not in focus, the browser takes two clicks to get drawing

I confirmed that this is also the case with OneNote, as you said. I think this may be a deliberate security or design decision in Google Chrome, because using this website shows that the mouse events never get to the website at all; however, in Firefox the events do get to the site. So probably what you want is completely impossible to implement in Chrome, but maybe possible in Firefox.

I didn't notice any mirroring performance issues in my testing so far.

mforbes commented 2 years ago

@williamstein Thanks! Erase works now in Chrome, but does not work in Firefox or Safari for me (on Mac OS X). I tried the link you posted above but don't see any difference between Chrome and Firefox. Safari does not seem to transmit the pressure data (but Chrome and Firefox do).

Edit: There is some issue with the object erase, where it erases large objects, like the radical below instead of the character inside I am pointing at:

CoCalcEraseBug .