Handwritten notes

[pcause]

I'd like to be able to use a pen to create a note that is my handwritten notes. of course, mouse could also be used but is awkward. I found this demo code (https://github.com/nicknytko/canvas-drawing) can capture the stylus/mouse input. for me it would be fine to capture and save an a png and a separate note type as opposed to an element of a note.

[zadam]

Some kind of drawing support is planned, but no promises yet.

[pcause]

thanks.

[Sebiann]

Would it work opening a small window with this in it (https://github.com/nicknytko/canvas-drawing) and make it make a screenshot of the window and then closing it.

[thfrei]

I am working on a new note type that allows to draw using fabric.js. I have a working MVP, however I have trouble storing the data.

I figured, that each Widget needs to have methods getContent() that gives the content, and somehow sends the content via this.spacedUpdate.scheduleUpdate(); (https://github.com/thfrei/trilium/blob/2f9930f7b882c19818a8364d6a1e2276e4bb62f9/src/public/app/widgets/type_widgets/canvas_note.js#L98)

My expectation was, once it is sent with this, in the next refresh (F5) I would see it as NoteComplement.content. However, it remains undefined

Can you give me a hint, how to best store the data? Maybe you also have some doc for how to store and retrieve data?

This is my branch: https://github.com/thfrei/trilium/tree/canvas-note

I would love to do a PR in the future, once it is polished, etc. How do you prefer to do this? New Issue or create a PR and continue discussion there? Also, maybe some discussion is needed in how to best add support for handwritten notes?

FYI:

• I am using this for drawing. This allows to later drag and drop painted lines: http://fabricjs.com/freedrawing
• And I plan to use this for pen/touch/mouse distinguish. This will allow to use pen pressure (at least for a Surface Pen) and pen tilt.: https://patrickhlauke.github.io/touch/tracker/multi-touch-tracker-pointer-hud.html
• The goal is to give more or less OneNote-like draw features (clean up UI, etc.)

[thfrei]

Sample Animation of PointerEvents (for people without Pen): https://patrickhlauke.github.io/touch/tracker/multi-touch-tracker-pointer-hud.html

[thfrei]

I just looked through the documentation and saw the paragraph about custom widgets. I'll try with this: https://github.com/zadam/trilium/wiki/Custom-Widget

When I started I just used relation-map and calendar and hacked some code together.

[zadam]

Hi, this looks interesting. Some points I noticed:

• don't use IDs to reference elements since this will break when you have multiple canvas notes open in tabs - instead search for elements inside the widget element using this.$widget.find()
• I'm not sure why the content is not getting saved - code looks correct in this regard. Are you sure that saveData() is getting triggered? Is there any exception on the backend process/log or in browser console?
• How do you want to use custom widgets? Your initial approach seems correct to me - canvas note would be a new note type so subclass of TypeWidget
• the code will need to use jQuery in the end but if it's easier to prototype in pure JS then of course it's no problem for now ...

I would love to do a PR in the future, once it is polished, etc. How do you prefer to do this? New Issue or create a PR and continue discussion there?

Yes, PR would be probably the best place to talk about this. You can create it in WIP state ...

[thfrei]

Hi @zadam Thank you for your feedback.

don't use IDs to reference elements since this will break when you have multiple canvas notes open in tabs - instead search for elements inside the widget element using this.$widget.find()

I will add this

How do you want to use custom widgets? Your initial approach seems correct to me - canvas note would be a new note type so subclass of TypeWidget

I just found this documentation about Widgets and confused it with the Note Type

the code will need to use jQuery in the end but if it's easier to prototype in pure JS then of course it's no problem for now ...

If I replace my $$$ selector with the $ it doesnt work. Somehow jquery does not seem to be the normal selector? I have to admit, I am also not sure how this TPL thing actually works under the hood (I am just a jquery script kiddy :-) ):

const TPL = `
<div class="note-detail-canvas-note note-detail-printable">

<style> ...`;

....

        this.$widget = $(TPL); // <--- I figure it somehow adds content to DOM, but i dont know. I alwas thought jquery $([selector])  selects elements...?

I'm not sure why the content is not getting saved - code looks correct in this regard. Are you sure that saveData() is getting triggered? Is there any exception on the backend process/log or in browser console?

No exceptions. Here are some sample requests.

So the save gets definitely called.

PUT

curl 'http://localhost:8080/api/notes/l0GepzYRQK7R' \
  -X 'PUT' \
  -H 'Connection: keep-alive' \
  -H 'Accept: */*' \
  -H 'x-csrf-token: bnj2mvzn-p5jQ3VEoCvPIFZHmZITeebC0sJ0' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'trilium-source-id: comp-kgaNYZfd' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36' \
  -H 'trilium-local-now-datetime: 2020-06-27 00:11:01.615+0200' \
  -H 'Content-Type: application/json' \
  -H 'Origin: http://localhost:8080' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: http://localhost:8080/?' \
  -H 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7' \
  -H 'Cookie: hblid=kG7EkQQSqD199Yq23m39N0NI6abTj3kB; olfsk=olfsk5819825421241713; __wt1vic=881439f1a1f3566; __wt1vpc=dipInstanceId%3Dghcmgpor-a9fbbbaae25b6b34d6c1e06a067e06c8-138f004b0bf5a11411136e4bedf68f34%26dssLicenseKind%3DCOMMUNITY%26bkdDistrib%3Dubuntu%26bkdDistribVersion%3D18.04%26isAutomation%3Dfalse%26dssVersion%3D5.1.5%26vdssUser%3D92668751; Webstorm-7adc446c=2f7a27c9-3510-4748-9a1c-2d108a914b24; Webstorm-b84c32c3=057c302c-2840-40da-b369-03ea8ee3d158; trilium-device=desktop; connect.sid=s%3AJ8sG4cXLSUdUesIiRjc1O9u4fd0ca79k.v8LPCkyvks%2BbPv%2BDZmsXMNTe%2FvjqJkD4vSI0nIS7hAY; _csrf=vBr0HEwgYiWL1gdWJh2E2-tV; protectedSessionId=' \
  --data-binary '{"attributes":[],"targetRelations":[],"parents":["pr3dytIQz7af"],"children":[],"parentToBranch":{"pr3dytIQz7af":"dj39i6aw2bsm"},"childToBranch":{},"noteId":"l0GepzYRQK7R","title":"canvasnote","contentLength":4901,"isProtected":false,"type":"canvas-note","mime":"application/json","isDeleted":0,"content":"{\"version\":\"4.0.0-beta.12\",\"objects\":[{\"type\":\"path\",\"version\":\"4.0.0-beta.12\",\"originX\":\"left\",\"originY\":\"top\",\"left\":140.18,\"top\":131.57,\"width\":0.06,\"height\":0,\"fill\":null,\"stroke\":\"#005e7a\",\"strokeWidth\":30,\"strokeDashArray\":null,\"strokeLineCap\":\"round\",\"strokeDashOffset\":0,\"strokeLineJoin\":\"round\",\"strokeMiterLimit\":10,\"scaleX\":1,\"scaleY\":1,\"angle\":0,\"flipX\":false,\"flipY\":false,\"opacity\":1,\"shadow\":{\"color\":\"#005e7a\",\"blur\":0,\"offsetX\":0,\"offsetY\":0,\"affectStroke\":true,\"nonScaling\":false},\"visible\":true,\"backgroundColor\":\"\",\"fillRule\":\"nonzero\",\"paintFirst\":\"fill\",\"globalCompositeOperation\":\"source-over\",\"skewX\":0,\"skewY\":0,\"path\":[[\"M\",155.18425444248933,146.57141744559217],[\"L\",155.24425444248934,146.57141744559217]]}]}"}' \
  --compressed

GET

{"noteId":"l0GepzYRQK7R","title":"canvasnote","contentLength":736,"isProtected":false,"type":"canvas-note","mime":"application/json","hash":"nYE/CVCufz","isDeleted":false,"deleteId":null,"isErased":0,"dateCreated":"2020-06-26 23:59:59.420+0200","dateModified":"2020-06-27 00:11:01.635+0200","utcDateCreated":"2020-06-26 21:59:59.420Z","utcDateModified":"2020-06-26 22:11:01.635Z","isContentAvailable":true}

If I add more "points" to my drawing, the content increases:

PUT

curl 'http://localhost:8080/api/notes/l0GepzYRQK7R' \
  -X 'PUT' \
  -H 'Connection: keep-alive' \
  -H 'Accept: */*' \
  -H 'x-csrf-token: bnj2mvzn-p5jQ3VEoCvPIFZHmZITeebC0sJ0' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'trilium-source-id: comp-kgaNYZfd' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36' \
  -H 'trilium-local-now-datetime: 2020-06-27 00:14:45.558+0200' \
  -H 'Content-Type: application/json' \
  -H 'Origin: http://localhost:8080' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: http://localhost:8080/?' \
  -H 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7' \
  -H 'Cookie: hblid=kG7EkQQSqD199Yq23m39N0NI6abTj3kB; olfsk=olfsk5819825421241713; __wt1vic=881439f1a1f3566; __wt1vpc=dipInstanceId%3Dghcmgpor-a9fbbbaae25b6b34d6c1e06a067e06c8-138f004b0bf5a11411136e4bedf68f34%26dssLicenseKind%3DCOMMUNITY%26bkdDistrib%3Dubuntu%26bkdDistribVersion%3D18.04%26isAutomation%3Dfalse%26dssVersion%3D5.1.5%26vdssUser%3D92668751; Webstorm-7adc446c=2f7a27c9-3510-4748-9a1c-2d108a914b24; Webstorm-b84c32c3=057c302c-2840-40da-b369-03ea8ee3d158; trilium-device=desktop; connect.sid=s%3AJ8sG4cXLSUdUesIiRjc1O9u4fd0ca79k.v8LPCkyvks%2BbPv%2BDZmsXMNTe%2FvjqJkD4vSI0nIS7hAY; _csrf=vBr0HEwgYiWL1gdWJh2E2-tV; protectedSessionId=' \
  --data-binary '{"attributes":[],"targetRelations":[],"parents":["pr3dytIQz7af"],"children":[],"parentToBranch":{"pr3dytIQz7af":"dj39i6aw2bsm"},"childToBranch":{},"noteId":"l0GepzYRQK7R","title":"canvasnote","contentLength":2824,"isProtected":false,"type":"canvas-note","mime":"application/json","isDeleted":0,"content":"{\"version\":\"4.0.0-beta.12\",\"objects\":[{\"type\":\"path\",\"version\":\"4.0.0-beta.12\",\"originX\":\"left\",\"originY\":\"top\",\"left\":140.18,\"top\":131.57,\"width\":0.06,\"height\":0,\"fill\":null,\"stroke\":\"#005e7a\",\"strokeWidth\":30,\"strokeDashArray\":null,\"strokeLineCap\":\"round\",\"strokeDashOffset\":0,\"strokeLineJoin\":\"round\",\"strokeMiterLimit\":10,\"scaleX\":1,\"scaleY\":1,\"angle\":0,\"flipX\":false,\"flipY\":false,\"opacity\":1,\"shadow\":{\"color\":\"#005e7a\",\"blur\":0,\"offsetX\":0,\"offsetY\":0,\"affectStroke\":true,\"nonScaling\":false},\"visible\":true,\"backgroundColor\":\"\",\"fillRule\":\"nonzero\",\"paintFirst\":\"fill\",\"globalCompositeOperation\":\"source-over\",\"skewX\":0,\"skewY\":0,\"path\":[[\"M\",155.18425444248933,146.57141744559217],[\"L\",155.24425444248934,146.57141744559217]]},{\"type\":\"path\",\"version\":\"4.0.0-beta.12\",\"originX\":\"left\",\"originY\":\"top\",\"left\":113.18,\"top\":264.57,\"width\":0.06,\"height\":0,\"fill\":null,\"stroke\":\"#005e7a\",\"strokeWidth\":30,\"strokeDashArray\":null,\"strokeLineCap\":\"round\",\"strokeDashOffset\":0,\"strokeLineJoin\":\"round\",\"strokeMiterLimit\":10,\"scaleX\":1,\"scaleY\":1,\"angle\":0,\"flipX\":false,\"flipY\":false,\"opacity\":1,\"shadow\":{\"color\":\"#005e7a\",\"blur\":0,\"offsetX\":0,\"offsetY\":0,\"affectStroke\":true,\"nonScaling\":false},\"visible\":true,\"backgroundColor\":\"\",\"fillRule\":\"nonzero\",\"paintFirst\":\"fill\",\"globalCompositeOperation\":\"source-over\",\"skewX\":0,\"skewY\":0,\"path\":[[\"M\",128.18425609043845,279.5714093279169],[\"L\",128.24425609043845,279.5714093279169]]},{\"type\":\"path\",\"version\":\"4.0.0-beta.12\",\"originX\":\"left\",\"originY\":\"top\",\"left\":225.18,\"top\":321.57,\"width\":0.06,\"height\":0,\"fill\":null,\"stroke\":\"#005e7a\",\"strokeWidth\":30,\"strokeDashArray\":null,\"strokeLineCap\":\"round\",\"strokeDashOffset\":0,\"strokeLineJoin\":\"round\",\"strokeMiterLimit\":10,\"scaleX\":1,\"scaleY\":1,\"angle\":0,\"flipX\":false,\"flipY\":false,\"opacity\":1,\"shadow\":{\"color\":\"#005e7a\",\"blur\":0,\"offsetX\":0,\"offsetY\":0,\"affectStroke\":true,\"nonScaling\":false},\"visible\":true,\"backgroundColor\":\"\",\"fillRule\":\"nonzero\",\"paintFirst\":\"fill\",\"globalCompositeOperation\":\"source-over\",\"skewX\":0,\"skewY\":0,\"path\":[[\"M\",240.18424925450137,336.57140584891323],[\"L\",240.24424925450137,336.57140584891323]]},{\"type\":\"path\",\"version\":\"4.0.0-beta.12\",\"originX\":\"left\",\"originY\":\"top\",\"left\":325.18,\"top\":257.57,\"width\":0.06,\"height\":0,\"fill\":null,\"stroke\":\"#005e7a\",\"strokeWidth\":30,\"strokeDashArray\":null,\"strokeLineCap\":\"round\",\"strokeDashOffset\":0,\"strokeLineJoin\":\"round\",\"strokeMiterLimit\":10,\"scaleX\":1,\"scaleY\":1,\"angle\":0,\"flipX\":false,\"flipY\":false,\"opacity\":1,\"shadow\":{\"color\":\"#005e7a\",\"blur\":0,\"offsetX\":0,\"offsetY\":0,\"affectStroke\":true,\"nonScaling\":false},\"visible\":true,\"backgroundColor\":\"\",\"fillRule\":\"nonzero\",\"paintFirst\":\"fill\",\"globalCompositeOperation\":\"source-over\",\"skewX\":0,\"skewY\":0,\"path\":[[\"M\",340.18424315098616,272.57140975516296],[\"L\",340.2442431509861,272.57140975516296]]}]}"}' \
  --compressed

GET

{"noteId":"l0GepzYRQK7R","title":"canvasnote","contentLength":2824,"isProtected":false,"type":"canvas-note","mime":"application/json","hash":"nYE/CVCufz","isDeleted":false,"deleteId":null,"isErased":0,"dateCreated":"2020-06-26 23:59:59.420+0200","dateModified":"2020-06-27 00:12:49.550+0200","utcDateCreated":"2020-06-26 21:59:59.420Z","utcDateModified":"2020-06-26 22:12:49.550Z","isContentAvailable":true}

If I do the same for a relation map it looks pretty much the same (smaller json payload) but I get a result:

PUT:

curl 'http://localhost:8080/api/notes/sm3Bcq0B7JdZ' \
  -X 'PUT' \
  -H 'Connection: keep-alive' \
  -H 'Accept: */*' \
  -H 'x-csrf-token: bnj2mvzn-p5jQ3VEoCvPIFZHmZITeebC0sJ0' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'trilium-source-id: comp-kgaNYZfd' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36' \
  -H 'trilium-local-now-datetime: 2020-06-27 00:17:07.378+0200' \
  -H 'Content-Type: application/json' \
  -H 'Origin: http://localhost:8080' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: http://localhost:8080/?' \
  -H 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7' \
  -H 'Cookie: hblid=kG7EkQQSqD199Yq23m39N0NI6abTj3kB; olfsk=olfsk5819825421241713; __wt1vic=881439f1a1f3566; __wt1vpc=dipInstanceId%3Dghcmgpor-a9fbbbaae25b6b34d6c1e06a067e06c8-138f004b0bf5a11411136e4bedf68f34%26dssLicenseKind%3DCOMMUNITY%26bkdDistrib%3Dubuntu%26bkdDistribVersion%3D18.04%26isAutomation%3Dfalse%26dssVersion%3D5.1.5%26vdssUser%3D92668751; Webstorm-7adc446c=2f7a27c9-3510-4748-9a1c-2d108a914b24; Webstorm-b84c32c3=057c302c-2840-40da-b369-03ea8ee3d158; trilium-device=desktop; connect.sid=s%3AJ8sG4cXLSUdUesIiRjc1O9u4fd0ca79k.v8LPCkyvks%2BbPv%2BDZmsXMNTe%2FvjqJkD4vSI0nIS7hAY; _csrf=vBr0HEwgYiWL1gdWJh2E2-tV; protectedSessionId=' \
  --data-binary '{"attributes":["eq4p3bDrnbo0","KYCey4lDR8rN"],"targetRelations":[],"parents":["pr3dytIQz7af"],"children":["FL2ziOUfGxEy","M4mD6gvVNpgo","CkTvYdiNnmK1"],"parentToBranch":{"pr3dytIQz7af":"MGWoCIHSzaZK"},"childToBranch":{"FL2ziOUfGxEy":"U9aW9kXHG2pE","M4mD6gvVNpgo":"DTvfDW03CT1L","CkTvYdiNnmK1":"LRjtwHcsz76G"},"noteId":"sm3Bcq0B7JdZ","title":"new note","contentLength":162,"isProtected":false,"type":"relation-map","mime":"application/json","isDeleted":0,"content":"{\"notes\":[{\"noteId\":\"M4mD6gvVNpgo\",\"x\":29,\"y\":369},{\"noteId\":\"CkTvYdiNnmK1\",\"x\":327,\"y\":165}],\"transform\":{\"x\":-1,\"y\":1,\"scale\":1}}"}' \
  --compressed

GET:

{"noteId":"sm3Bcq0B7JdZ","title":"new note","contentLength":131,"isProtected":false,"type":"relation-map","mime":"application/json","hash":"5+WFYF3Di+","isDeleted":false,"deleteId":null,"isErased":0,"dateCreated":"2020-06-26 23:58:44.478+0200","dateModified":"2020-06-27 00:17:07.397+0200","utcDateCreated":"2020-06-26 21:58:44.478Z","utcDateModified":"2020-06-26 22:17:07.397Z","isContentAvailable":true,"content":"{\"notes\":[{\"noteId\":\"M4mD6gvVNpgo\",\"x\":29,\"y\":369},{\"noteId\":\"CkTvYdiNnmK1\",\"x\":327,\"y\":165}],\"transform\":{\"x\":-1,\"y\":1,\"scale\":1}}"}

Any ideas?

[zadam]

If I replace my $$$ selector with the $ it doesnt work. Somehow jquery does not seem to be the normal selector? I have to admit, I am also not sure how this TPL thing actually works under the hood (I am just a jquery script kiddy :-) ):

You use e.g. $$$('drawing-mode'), but the drawing-mode is not a selector (or technically it is, but a wrong one), #drawing-mode would be a selector which would work (for ID). But for prototype phase it's OK to use whatever works for you. It's not a big deal to fix this afterwards.

$(TPL) just takes the string and converts it into the jQuery/DOM element, but it doesn't add it into the DOM document. That's the responsibility of the parent component, respectively the layout code.

No exceptions. Here are some sample requests.

Ok, so I guess the problem lies in the backend code. You could check the services/notes.js/updateNote() which is responsible for the content update.

[thfrei]

So i figured it out, there were a couple of places in the backend code that handles the content. Whenever the mime type is application/json there is some different stuff going on. In my case it was converted into an UIntArray... It also got converted to type file, after restart.

It took me a while, but long story short, I added the canvas-note type in most places where I found the relation-map and now it works.

To further work on this feature, I created a PR that shows my current status. I would love to get your feedback there: https://github.com/zadam/trilium/pull/1135

[thfrei]

Suggestion: close this, since Feature branch was merged? #2798

[CobriMediaJulien]

@thfrei

Hey, i don´t want to bother you. Do you think it´s hard to add smaller stroke width button in the GUI for free drawing like in the obisidan plugin of excalidraw?

Would help to use the canva even more.

The perfect stroke width would be 0.75. The smallest one that is 1.25 just looks so weird if you combine it with other shapes. I spent my day searching how to implement this but all i figured out was to open the canva notes as a javaskript frontend note and then change the strokewidth attribute manually.