(Feature request) new note type: Draw

[Dmitrykrz]

Preflight Checklist

• [X] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Describe feature

I want to have a feature like in microsoft onenote. It would be really nice to add a new note type called "Draw" or "Paint" where I can draw things like in MS Paint with mouse and when I am done it saves it like a image.

Additional Information

No response

[zadam]

Hi, yes, this is on a (long term) TODO list.

[sigaloid]

1165 #858 #671 #188 for some other issues on this

[p0358]

I guess infinite possible height and vector graphics? But width might be the hard concern here...

[thfrei]

I did some early trials with fabric.js, roughly a year ago (see #671) As I see it today, I think our best bet would be to include the open source library: https://github.com/excalidraw/excalidraw

It gives us an drawing, infinite canvas, libraries for other shapes and stuff, well developed code base, active community.

One drawback: It is based on react. So we would need to load react and interface with it. It would bloat the JS payload size, but I think this should not be too much of an issue, since JS size codebase is not of our main concern. We could try with preact to minimize the impact here.

[thfrei]

@Dmitrykrz The note type I developed then was called Canvas Note. I don't know if canvas note speaks to people about what it can do? https://github.com/thfrei/trilium/commits/canvas-note

[zadam]

excalidraw looks very nice, even though it is pretty new. I like that it seems to be usable for both simple diagramming and simple drawing.

[CorneliousJD]

I just wanted to chime in to say that while I love Trilium and do not want to migrate away from it, the lack of a drawing/handwriting option is starting to be cumbersome for me. I am supplementing with OneNote for now but would prefer to stay self-hosted.

There's a HUGE gap in the self-hosted community where nothing really offers drawing/handwriting, except Wiznote, and I'd MUCH prefer to stay away from that haha

[JonaHesselmann]

@thfrei Is this still in development. I am working on an Idea similar to yours. otherwise I would work on the excalidraw implementation

[thfrei]

@yugimuo Well, I have to admit, I am not working on it right now, and haven't been for some time. I am blocked to start somehow. If you would work on the excalidraw implementation, that would be great. Hopefully you can find some code in my canvas-note-branch that helps you where to add trilium-specific functionality regarding the note-type.

I am curious, what is the idea you are/were working on? Do you see a benefit over excalidraw? What did you use?

[JonaHesselmann]

@thfrei i have done some research and found out that its more complicated .. as always. there are a few libraries we could use 1.litteraly canvas is very modular and has infinite canvas, is not further in development tho 2.Excalidraw is very feature rich and has support and a good codebase. however its written in typescript and based on react so it would bloat up the file size and we would have to make some adjustments to the code. there are other libraries but these two are my top right now. maybe zadam has something to say about. I dont want to implement something that would not be used @zadam

[thfrei]

@yugimuo Yes, it's more complicated as always :-) I have done extensive research myself. Literteraly canvas seems nice, but it has probably the same problem due to its react dependency. From all tools I looked at, I think Excalidraw has the best feature set. My trials with fabric.js where quite nice, however it would take a great effort to reach this level of code quality and diagramming support. One plus of excalidraw is, that it is actively maintained. In order to abstract the Excalidraw react dependency I tried to create a library that can simple be used as a jQuery "plugin". Maybe this helps:

• Repo with a working and hot reloading excalidraw: https://github.com/thfrei/excalidraw-test
  • With a little tweaking we could build a a single-file library
  • I used the version 0.10.0-beta. A month ago excalidraw released 0.11.0 with good image pasting support: https://www.npmjs.com/package/@excalidraw/excalidraw
• Article for some guidance about how to use a React component as a jQuery Plugin: https://www.pluralsight.com/guides/how-to-use-reactjs-components-as-jquery-plugins

My dream is, that we can at some point paste a "drawing-note" as an image (or maybe other types too?) onto a drawing note (like a picture in a text note now), so that we can then work with layers. This would be insane.

Sidenote: Zadam seemed to be quite open to excalidraw, if you read some earlier comments. If I may summarize his points from earlier discussions, the active community and the ability to do basic diagramming (if you think about users without stylus) are the biggest plus.

[JonaHesselmann]

excalidraw it is then I guess. thought about literaly canvas because it has a pure js core that does not uses react. However excalidraw is much better thats true. I dont have experience with complex react rewrites but i could try. No promises tho. Maybe we can join forces somehow. I will definitly browse through your code, it will be of great help. I just thought that excalidraw is a little bit too much for simple drawing.

[zadam]

my 2 cents - I still see excalidraw as the best candidate. Pure JS would be a plus, but new libraries having a react dependency is just how it is these days. excalidraw is huge as it is, react.js dependency is a drop in the bucket (it will be lazy loaded anyway). The biggest disadvantage of literallycanvas is that it's old and unmaintained.

[frapples]

Hi, I recommend draw.io, available online at https://app.diagrams.net/. This product is open source and should be easy to integrate into the electron project as I observed it has an integrated plugin for vscode. Compared with free drawing, draw.io is more like Microsoft viso. It can draw flow charts, data flow diagrams, uml diagrams that developers need, as well as general mind maps and concept maps. I think it is more suitable for knowledge management. @zadam

[JonaHesselmann]

@frapples I dont think that draw.io would be easy to implement as stated in the offical repository. also the question should be asked if we need another way to make diagrams. mermaid.js is already implemented and would become obsolete. In my opinion we should keep it KISS and only implement free hand drawing and simple shapes and text not more.

[zadam]

FWIW mermaid.js is great for some types of diagrams (e.g. sequence), but pretty bad for other types (flowchart). So I think something like draw.io or excalidraw would still be very useful.

[thfrei]

I resumed development in this branch: https://github.com/thfrei/trilium/tree/excalidraw

Currently I have a working prototype of using excalidraw inside trilium. There are still a lot of issues regarding load/save and refresh, and also sizing and css and stuff.

[thfrei]

I think we are getting somewhere:

• Pasteing of clipboard images work (not trilium images yet)
• multiple instances
• save/load switch notes, etc.

Known issue:

• a lot :-)

[sigaloid]

Wow, this is awesome, I cannot wait for this. I'd like to try to work on the implementation of the sharing of this canvas note type. I'll see what I can do this weekend maybe.

[sigaloid]

Never mind, grabbed a red bull and did this.

:D somewhat successful! It currently embeds about 30 lines of javascript required for rendering the graph all in the head (header variable in content_renderer.js). @zadam , would this be too much? In other cases like in math-tex, it's about 10 lines. I'm just worried about maintainability of big blocks of javascript within this file. I tried to abstract it out in a separate JS file but the actual content of the drawing needs to be embedded inside to be read. I thought about adding a route just for this note type but that also might be too much kludge.

Let me know how you want me to merge this; I can hold onto it until thfrei/trilium#excalidraw is merged then make a PR to add to zadam/trilium.

I can also make some docs for the new note type, if it's merged.

[thfrei]

Hi @sigaloid. That looks great. For sharing, we might also be able to use something like for note revision: I implemented note_revision and found, that we do not need react, but rather can use a utils library (https://www.npmjs.com/package/@excalidraw/utils). With this library we can convert it to svg.

https://github.com/thfrei/trilium/commit/13a9c22eb7d254b1e56124198b5e5b490a826fc6#diff-42a9a713c26565a44445ef5b04a3c34ebc5c4b19115e9838149de333664ce90c

I am also thinking about using this library to be able to include the automatically-generated SVG into text-notes, or even into another excalidraw canvas-note. Then we can work with note in note in note :-)

[thfrei]

Currently, for note_revisions, I am not quite sure if it is better to load react and render it, or just use svg. With react, we get the interactivity (and we might lock down editing functionality), however it is a little bit more complicated to call and embedd. What do you guys think?

[sigaloid]

I think the best option would be to render SVG in the note revisions menu. I don't think there'll be much need for interaction when looking at revisions. Maybe an option to load the full library and render it when clicked.

Same with shared notes, what if it renders the SVG and has a "render" button?

[zadam]

I agree with SVG option for note revisions and sharing. Having SVG is also a good safeguard against bit rot.

[zadam]

I agree with SVG option for note revisions and sharing. Having SVG is also a good safeguard against bit rot.

[zadam]

I agree with SVG option for note revisions and sharing. Having SVG is also a good safeguard against bit rot.

[JonaHesselmann]

Alright you were faster than me. gonna bug test then and fix things if I find errors @thfrei good job

[thfrei]

OK, I am making some progress, and I have some questions and remarks to the community and to @zadam:

• I definetly think we should use SVG rendering for note revisions and book preview (and other preview). However, I saw, that the libraries use some excalidraw fonts from https://unpkg.com. I guess we should include all assets, so that no external request is being done?
• For shareing just thecanvas note, I think it might make sense to have react-excalidraw with a minimal set of functions (basic view functions, just like a miro board). Since, if you want to share the Canvas, you get the canvas and it's interactivity. If you want it as an image, you can call the image endpoint.
• Should we stick internally with the type: canvas-note? Or rather excalidraw to make it analog to mermaid? I personally think canvas-note is nice, because strictly speaking I am not "dependent" on excalidraw, but what I want is a canvas where I can draw?! And the text-note is not really text, and not called ckeditor :-)
• Should we stick with Canvas Note?
• Should we include all libraries as files in the libraries folder, or use node-dependencies? Most libraries I added already statically, however I have one dependency (excalidraw-to-svg), that is a node-module. With sub-dependencies, and so on. I personally think it would be cleaner to have node-modules with a fixed version, which might make it more easy to upgrade?!
• I am rendering SVGs on the fly, so that we are able to include it directly into other notes (which I think is awesome) (https://github.com/thfrei/trilium/commit/345d4ffe35c79602228a2d195b39914ac81176b0#diff-84feae3021f52eb576d88e62f416ce922ef2e772e44ba117d36ff2b82919f8baR22) (Sidenote: can we do this with mermaid yet?). However, this has two issues:
  • It's another dependency: https://www.npmjs.com/package/excalidraw-to-svg. I must say, it is extremly well done, with a test suite, and only very minimal dependencies. Just excalidraw and jsdom to be able to render the canvas in node.js. What do you think? Due to this node module, it might make sense to call at least excalidraw directly?
  • Rendering takes ~1-2s. I guess it is due to loading this huge canvas/jsdom object in node. Do you think, that's a problem? I am not sure if we can do something about it.
• For the book view, should we change the width to the container, so that we see the whole svg-image, or use it's original zoom factor?

Thank you for your time!

Some impressions:

• Book view:
• Just the image, rendered to svg on the fly:
• Canvas-note referenced in Text-note:

[thfrei]

@zadam

Slightly off-topic:

• Shouldn't we make the :filename in the image api optional? Whatever you type in as a filename is not used. Opening the link without the filename however gives an error.
  • default image api: route(GET, '/api/images/:noteId/:filename', [auth.checkApiAuthOrElectron], imageRoute.returnImage); https://github.com/thfrei/trilium/blob/excalidraw/src/routes/routes.js#L287
  • share api: https://github.com/thfrei/trilium/blob/excalidraw/src/share/routes.js#L91

The solution as of express 4.x would be to use array format for routes (backwards compatible): https://github.com/thfrei/trilium/commit/692f910862db3403750fe87f2aafb1bfe096b5e2

Appendix:

• Just some random filename:
• No filename:

[zadam]

Hi @thfrei, very impressive work! Here are some of my answers:

I guess we should include all assets, so that no external request is being done?

Yes, I think there shouldn't be any external dependency, Trilium should be fully offline capable.

Just a side-note - revisions and sharing support are not necessarily mandatory for the initial PR. Some notes types are not supported in either. It can also be backfilled later.

Should we stick internally with the type: canvas-note

I agree. "mermaid" is named as such because there's no generic name for such diagram type and also the syntax is mermaid specific (although partially compatible with plantuml).

Should we include all libraries as files in the libraries folder, or use node-dependencies?

I'm not sure. I've so far stuck to the old-school manual dependency management, but I don't have a strong preference. If it's an issue to extract it into the libraries, keep it as a node dependency.

I am rendering SVGs on the fly, so that we are able to include it directly into other notes (which I think is awesome)

In the end, I would like to have the rendered SVG stored in the database. Complex libraries like excalidraw might not work in browsers in 30 years, but SVG as a standard format will, so it's good to have that as a sort of insurance against bit rot. But I'm not really sure how to do this (when to render and where to store the rendered SVG).

But IMHO this is not a critical problem now and can be optimized later. On the fly approach is fine for now.

For the book view, should we change the width to the container, so that we see the whole svg-image, or use it's original zoom factor?

In book view the existing approach is to show the whole shrunk image fitting the box.

Shouldn't we make the :filename in the image api optional? Whatever you type in as a filename is not used. Opening the link without the filename however gives an error.

The reason for this is when you try to save the image in browser, it will offer you the last segment of the URL as a filename.

[thfrei]

Happy Easter. Thank you for your feedback! I'll use it to update my branch and then make a PR.

Just a side-note - revisions and sharing support are not necessarily mandatory for the initial PR. Some notes types are not supported in either. It can also be backfilled later.

I'd like to have this in the first PR. It just looks so nice :-)

One more thing: I found a couple of issues with excalidraw-to-svg and excalidraw/utils. The issues are quite severe (in the backend, no freedraw paths are rendered. This is some limitation in the jsdom/node-canvas library with missing Path2D support - more details in the code later).

However, I was thinking rendering an SVG alongside the excalidraw-format, so that we save both in parallel to the note. It would increase the note size, and maybe bandwith usage, however sharing and embedding would be way faster. It would also combat bitrot.

[thfrei]

Suggestion: close this, since Feature branch was merged? #2798