search

lukesmurray / linked-notes-vscode

vscode extension for taken Zettelkasten inspired markdown notes.
https://marketplace.visualstudio.com/items?itemName=lukesmurray.linked-notes-vscode
0 stars 0 forks source link

Adoption Issues #49

Closed lukesmurray closed 3 years ago

lukesmurray commented 4 years ago

I've found that this extension is hard to use in my daily note taking. I do want to adopt the extension but there are some issues which are important to address when I have time.

Performance Issues

The extension occasionally freezes my workspace which is completely unacceptable. There is no situation where I am ok with having my workspace frozen while waiting for the extension to start. Sometimes typing becomes painfully slow, there is a long delay between key presses and rendering on the screen. Typing should always be fluid. The typing issue is more prevalent on longer documents. Sometimes I open up a new document during meetings to avoid delay while typing. I then copy over the text afterwards.

The extension takes a long time to boot. Especially on longer workspaces. Instead of processing all files the extension can adopt principles of progressive enhancement and enable features on the fly. An example of 5 lines which are probably causing a bunch of pain is the following. Waiting for all files to be parsed by remark is not feasible.

https://github.com/lukesmurray/linked-notes-vscode/blob/01eda3b4fe1867c2bd1266a2bb0c211a977dc15a/src/extension.ts#L59-L68

The extension should definitely respect .gitignore patterns. Opening the extension in a workspace with a node_modules files essentially crashes vscode. I've disabled linked notes by default to avoid this behavior.

Organization Issues

While Zettlekasten is a great idea I often find that some sense of file hierarchy is beneficial. Some common use cases are

Ideally I would like to be able to link to any file by its title so [[this is the page title]] links to a document with the title [[this is a page title]] anywhere in the workspace. The downside of this approach is there must be a one to one mapping between documents and titles, but the upside is I can have nested folders which is something that happens in the real world.

Exporting Notes

The more I get into writing documents with markdown the more I feel limited by most implementations. I've recently found that pandoc supports citations and very rich output formats. I plan on writing all my notes for a pandoc output format but pandoc has very very bad support for [[wikilinks]]. In general it seems almost impossible to implement. That being said I do not want to implement much support for exporting documents or even previewing documents in this extension. I think that would be boiling the ocean. I can however materialize documents in a parseable form for other markdown libraries.

Solutions

Performance

For performance while typing we can try to use profiling to determine the performance issues. Luckily there is essentially a single path for causing code to run (with a small caveat)

  1. A markdown file is changed.
  2. Remark is used to parse the markdown contents into an abstract syntax tree.
  3. The updated abstract syntax tree is searched for file references.
  4. The new file references are passed into the store.
  5. Various reselect selectors are updated.

In general the hot paths seem to be (2) and (5). (2) takes about 20-30ms per document and (5) takes about 10ms per document. Unfortunately these steps are run for all documents when the extension loads. For a workspace with 1000 documents this is unacceptable.

One way to avoid the start up cost is to index results of (2) and store the results with a file hash in a memento (such as the workspace state in the extension context). Then the store can be completely hydrated with minimal changes.

Within (2) the hot path is visit-parents which is used for providing context for backlinks. visit parents takes about 10ms. Within (5) there is almost no obvious hot path other than immer.

The caveat I mentioned is citation parsing currently utilizes aho corasick to match citations from the csl json file. This means that if the CSL JSON file changes, all workspace files are invalidated. When workspace files are invalidated steps 1-5 must be run again. The simple fix is to remove any reliance on citation parsing to the CSL JSON and instead store parsed ids which are matched lazily when needed.

Organization

The organization problem is solved by creating a mapping from fileReference to title to filepath. The fileReference to title is easy. The title to filepath is hard. I ran into issues in #44 while trying to solve it. Currently all backlinks are resolved based on the fsPath so the fsPath needs to be consistent.

A new solution is to create a fileManager which has a mapping from title to fsPath. When files are parsed their titles are added to the fileManager. Duplicate titles are considered an error. When the fileManager resolves a reference it can either return the real path, or it returns a fake path. The fake path can be indicated by some prefix but the fake path is consistent and contains the real title. When a file has to be materialized, for example in go to definition or, follow link, the fsPath is checked. If the fsPath is fake the fspath is replaced by a real fspath. The real fsPath will be created in the current directory. The basename can be a slug or the full title or a timestamp. The key is the basename doesn't matter. If the user does not like the location of the file the user can move the file. The store will automatically update all references based on the new location of the file.

Along with this change I think the title of the document should be stored in the frontmatter and not in the first level heading. Title file references would then exist in front matter. TBD if parsing is easy.

Export

In order to reduce the scope of this extension I don't think the extension should try to preview files or export them to html. Instead the extension can export a copy of the workspace with all file reference materialized.

lukesmurray commented 4 years ago

One thing to be aware of is the new fileManager solution makes titles the key to files rather than fsPath. It would be worthwhile to embrace that shift in perspective by making title the id field in LinkedFiles

lukesmurray commented 4 years ago

Some profiling information. On a random file

On a 500 line file of bullets which is pretty common for me

lukesmurray commented 4 years ago

Some more profiling information. For my notes v1. 54 files. 2758 lines of text

Parse Times
0.03339513502
0.004175510019
0.0005511009991
0.002248964995
0.006719399005
0.0008735980093
0.004659835994
0.009994465977
0.001601568997
0.004112870008
0.0004316779971
0.005089417011
0.0005862330198
0.002761716992
0.004867209017
0.002547047019
0.004743729979
0.0003197840154
0.0003495500088
0.0003584190011
0.0002724289894
0.003586726993
0.02842760599
0.0006132580042
0.0008112640083
0.001105188012
0.0003248640001
0.00904597798
0.006073327005
0.004826602995
0.0003878070116
0.0007841020226
0.0008388699889
0.0004696129858
0.101936202
0.0006372579932
0.000331265986
0.000893724978
0.005055361986
0.004454500973
0.0003116099834
0.002639479011
0.0003658370078
0.0004711079895
0.004856160998
0.000777583003
0.002436902016
0.004312792003
0.0008638699949
0.0003763740063
0.0003555440009
0.01105740598
0.0005006510019
0.0005578649938
0.291146365

Total in bold. Really not an issue! Loading all files takes 9 seconds though.

lukesmurray commented 4 years ago 
[info       - 17:00:07.677] delay tasks.md 1.007820388019085 seconds
[info       - 17:00:07.679] ready to update tasks.md 1.01023332798481 seconds
[info       - 17:00:07.825] syntax tree tasks.md 0.1455620429813862 seconds
[info       - 17:00:07.857] references tasks.md 0.032539633005857466 seconds
[info       - 17:00:08.140] delay integration-notes.md 1.076525098979473 seconds
[info       - 17:00:08.140] ready to update integration-notes.md 1.0771143670082093 seconds
[info       - 17:00:08.160] syntax tree integration-notes.md 0.0019356209933757782 seconds
[info       - 17:00:08.160] references integration-notes.md 0.0002028679847717285 seconds
[info       - 17:00:08.190] delay bidmc.md 1.251646685987711 seconds
[info       - 17:00:08.190] ready to update bidmc.md 1.252209139019251 seconds
[info       - 17:00:08.191] syntax tree bidmc.md 0.0004845280051231384 seconds
[info       - 17:00:08.191] references bidmc.md 0.00009163299202919007 seconds
[info       - 17:00:08.385] delay charticulator-architecture.md 1.4448570820093154 seconds
[info       - 17:00:08.385] ready to update charticulator-architecture.md 1.445453399002552 seconds
[info       - 17:00:08.387] syntax tree charticulator-architecture.md 0.0019802899956703184 seconds
[info       - 17:00:08.387] references charticulator-architecture.md 0.00019640201330184938 seconds
[info       - 17:00:08.578] delay vscode-wiki.md 1.6187998809814452 seconds
[info       - 17:00:08.578] ready to update vscode-wiki.md 1.6194040499925613 seconds
[info       - 17:00:08.583] syntax tree vscode-wiki.md 0.005170408010482788 seconds
[info       - 17:00:08.584] references vscode-wiki.md 0.0006351799964904785 seconds
[info       - 17:00:08.761] delay http-cookies.md 1.7957636369764804 seconds
[info       - 17:00:08.761] ready to update http-cookies.md 1.7964642199873924 seconds
[info       - 17:00:08.762] syntax tree http-cookies.md 0.0007940400242805481 seconds
[info       - 17:00:08.762] references http-cookies.md 0.0001121239960193634 seconds
[info       - 17:00:09.800] delay bidmc-cards.md 2.0408786729872226 seconds
[info       - 17:00:09.800] ready to update bidmc-cards.md 2.0415933509767057 seconds
[info       - 17:00:09.120] syntax tree bidmc-cards.md 0.004146737992763519 seconds
[info       - 17:00:09.120] references bidmc-cards.md 0.00036553800106048584 seconds
[info       - 17:00:09.187] delay javascript-events.md 2.2021869010031225 seconds
[info       - 17:00:09.187] ready to update javascript-events.md 2.203121811002493 seconds
[info       - 17:00:09.201] syntax tree javascript-events.md 0.013945254027843475 seconds
[info       - 17:00:09.202] references javascript-events.md 0.0006574880182743072 seconds
[info       - 17:00:09.361] delay caption.md 2.3738028840124605 seconds
[info       - 17:00:09.361] ready to update caption.md 2.3754030520021914 seconds
[info       - 17:00:09.362] syntax tree caption.md 0.001176012009382248 seconds
[info       - 17:00:09.362] references caption.md 0.0000814480185508728 seconds
[info       - 17:00:09.539] delay nlp.md 2.5307672879993914 seconds
[info       - 17:00:09.539] ready to update nlp.md 2.5315483480095864 seconds
[info       - 17:00:09.543] syntax tree nlp.md 0.003967295974493027 seconds
[info       - 17:00:09.543] references nlp.md 0.0003402199745178223 seconds
[info       - 17:00:09.723] delay dftw-chat-client.md 2.7121261139810087 seconds
[info       - 17:00:09.723] ready to update dftw-chat-client.md 2.7134541690051557 seconds
[info       - 17:00:09.723] syntax tree dftw-chat-client.md 0.00047124698758125303 seconds
[info       - 17:00:09.723] references dftw-chat-client.md 0.0000857519805431366 seconds
[info       - 17:00:09.887] delay dice-game.md 2.8750003509819506 seconds
[info       - 17:00:09.887] ready to update dice-game.md 2.875861010015011 seconds
[info       - 17:00:09.892] syntax tree dice-game.md 0.005013489991426468 seconds
[info       - 17:00:09.893] references dice-game.md 0.0006507509946823121 seconds
[info       - 17:00:10.540] delay writing-inbox.md 3.0404752530157566 seconds
[info       - 17:00:10.540] ready to update writing-inbox.md 3.041322483986616 seconds
[info       - 17:00:10.550] syntax tree writing-inbox.md 0.0006119489967823028 seconds
[info       - 17:00:10.550] references writing-inbox.md 0.00014343801140785217 seconds
[info       - 17:00:10.221] delay rent.md 3.2054381110072137 seconds
[info       - 17:00:10.221] ready to update rent.md 3.2063484679758547 seconds
[info       - 17:00:10.224] syntax tree rent.md 0.0028754329979419706 seconds
[info       - 17:00:10.225] references rent.md 0.0007015469968318939 seconds
[info       - 17:00:10.395] delay meeting-bidmc-biweekly-sync.md 3.35474944499135 seconds
[info       - 17:00:10.395] ready to update meeting-bidmc-biweekly-sync.md 3.355754116982222 seconds
[info       - 17:00:10.398] syntax tree meeting-bidmc-biweekly-sync.md 0.0036782499849796297 seconds
[info       - 17:00:10.399] references meeting-bidmc-biweekly-sync.md 0.0005108100175857544 seconds
[info       - 17:00:10.567] delay performance-optimizations-for-bidmc.md 3.5252580280005934 seconds
[info       - 17:00:10.567] ready to update performance-optimizations-for-bidmc.md 3.526269024014473 seconds
[info       - 17:00:10.569] syntax tree performance-optimizations-for-bidmc.md 0.0015535120069980622 seconds
[info       - 17:00:10.569] references performance-optimizations-for-bidmc.md 0.00016791102290153502 seconds
[info       - 17:00:10.735] delay meeting-with-karger.md 3.6913926270008086 seconds
[info       - 17:00:10.735] ready to update meeting-with-karger.md 3.6924635689854624 seconds
[info       - 17:00:10.744] syntax tree meeting-with-karger.md 0.008822290003299713 seconds
[info       - 17:00:10.745] references meeting-with-karger.md 0.0009709540009498597 seconds
[info       - 17:00:10.916] delay bidmc-templates.md 3.78678481400013 seconds
[info       - 17:00:10.916] ready to update bidmc-templates.md 3.7882022030055524 seconds
[info       - 17:00:10.919] syntax tree bidmc-templates.md 0.003124290019273758 seconds
[info       - 17:00:10.920] references bidmc-templates.md 0.00026681700348854065 seconds
[info       - 17:00:11.890] delay personal.md 3.9571327349841594 seconds
[info       - 17:00:11.890] ready to update personal.md 3.9583810819983483 seconds
[info       - 17:00:11.890] syntax tree personal.md 0.00039391100406646726 seconds
[info       - 17:00:11.890] references personal.md 0.00007993599772453308 seconds
[info       - 17:00:11.258] delay linked-notes-vscode.md 4.123089783996344 seconds
[info       - 17:00:11.258] ready to update linked-notes-vscode.md 4.124324607998132 seconds
[info       - 17:00:11.258] syntax tree linked-notes-vscode.md 0.0004762980043888092 seconds
[info       - 17:00:11.258] references linked-notes-vscode.md 0.0001224360167980194 seconds
[info       - 17:00:11.996] delay reading-inbox.md 4.85887821701169 seconds
[info       - 17:00:11.996] ready to update reading-inbox.md 4.860291317999363 seconds
[info       - 17:00:11.996] syntax tree reading-inbox.md 0.0004135569930076599 seconds
[info       - 17:00:11.997] references reading-inbox.md 0.00008939400315284729 seconds
[info       - 17:00:12.184] delay project.md 5.042708422988653 seconds
[info       - 17:00:12.184] ready to update project.md 5.045735481023788 seconds
[info       - 17:00:12.185] syntax tree project.md 0.0005482890009880066 seconds
[info       - 17:00:12.185] references project.md 0.0000809049904346466 seconds
[info       - 17:00:12.387] delay document-reference-interface.md 5.238288653999567 seconds
[info       - 17:00:12.387] ready to update document-reference-interface.md 5.239430886000394 seconds
[info       - 17:00:12.392] syntax tree document-reference-interface.md 0.004857942998409272 seconds
[info       - 17:00:12.393] references document-reference-interface.md 0.0006201450228691101 seconds
[info       - 17:00:12.587] delay nlp-lecture.md 5.419673675000667 seconds
[info       - 17:00:12.587] ready to update nlp-lecture.md 5.420840876996517 seconds
[info       - 17:00:12.624] syntax tree nlp-lecture.md 0.03662895497679711 seconds
[info       - 17:00:12.626] references nlp-lecture.md 0.0020377199947834014 seconds
[info       - 17:00:12.824] delay vscode-extension-patterns.md 5.654472966998815 seconds
[info       - 17:00:12.824] ready to update vscode-extension-patterns.md 5.655594137012958 seconds
[info       - 17:00:12.825] syntax tree vscode-extension-patterns.md 0.0011916399896144867 seconds
[info       - 17:00:12.825] references vscode-extension-patterns.md 0.00012084999680519104 seconds
[info       - 17:00:13.280] delay dftw-recitation-7.md 5.8558513070046905 seconds
[info       - 17:00:13.280] ready to update dftw-recitation-7.md 5.85699592101574 seconds
[info       - 17:00:13.290] syntax tree dftw-recitation-7.md 0.0012955900132656096 seconds
[info       - 17:00:13.290] references dftw-recitation-7.md 0.00011346802115440369 seconds
[info       - 17:00:13.218] delay markdown-tools.md 6.042003464996815 seconds
[info       - 17:00:13.218] ready to update markdown-tools.md 6.043978040009737 seconds
[info       - 17:00:13.223] syntax tree markdown-tools.md 0.004558729976415634 seconds
[info       - 17:00:13.223] references markdown-tools.md 0.0001568410098552704 seconds
[info       - 17:00:13.408] delay dftw.md 6.22865945699811 seconds
[info       - 17:00:13.408] ready to update dftw.md 6.230747727990151 seconds
[info       - 17:00:13.408] syntax tree dftw.md 0.0003745729923248291 seconds
[info       - 17:00:13.409] references dftw.md 0.00008076497912406922 seconds
[info       - 17:00:13.631] delay future.md 6.443566246002913 seconds
[info       - 17:00:13.631] ready to update future.md 6.444717689007521 seconds
[info       - 17:00:13.645] syntax tree future.md 0.013809196025133133 seconds
[info       - 17:00:13.649] references future.md 0.004078332006931305 seconds
[info       - 17:00:13.863] delay journal-2020.md 6.6736204850077625 seconds
[info       - 17:00:13.863] ready to update journal-2020.md 6.674879760980606 seconds
[info       - 17:00:13.868] syntax tree journal-2020.md 0.004716496020555496 seconds
[info       - 17:00:13.871] references journal-2020.md 0.0032588119804859163 seconds
[info       - 17:00:14.159] delay linked-notes.md 6.946685202002525 seconds
[info       - 17:00:14.159] ready to update linked-notes.md 6.947958348989487 seconds
[info       - 17:00:14.164] syntax tree linked-notes.md 0.004917934983968735 seconds
[info       - 17:00:14.166] references linked-notes.md 0.0017552520036697387 seconds
[info       - 17:00:14.497] delay luke.md 7.28209061101079 seconds
[info       - 17:00:14.497] ready to update luke.md 7.283242828011513 seconds
[info       - 17:00:14.497] syntax tree luke.md 0.0005393339991569519 seconds
[info       - 17:00:14.498] references luke.md 0.00008925899863243103 seconds
[info       - 17:00:14.794] delay utilities.md 7.575755688995123 seconds
[info       - 17:00:14.794] ready to update utilities.md 7.5773227600157265 seconds
[info       - 17:00:14.795] syntax tree utilities.md 0.0009843290150165559 seconds
[info       - 17:00:14.795] references utilities.md 0.00010015898942947387 seconds
[info       - 17:00:15.880] delay wiki-links.md 7.867271858006716 seconds
[info       - 17:00:15.890] ready to update wiki-links.md 7.868696177005768 seconds
[info       - 17:00:15.890] syntax tree wiki-links.md 0.0009076640009880066 seconds
[info       - 17:00:15.900] references wiki-links.md 0.00011118802428245545 seconds
[info       - 17:00:15.389] delay andy-matuschak.md 8.165009543001652 seconds
[info       - 17:00:15.389] ready to update andy-matuschak.md 8.166418005019427 seconds
[info       - 17:00:15.390] syntax tree andy-matuschak.md 0.0006144919991493225 seconds
[info       - 17:00:15.390] references andy-matuschak.md 0.00007733699679374694 seconds
[info       - 17:00:15.698] delay scratch-nlp-paper.md 8.471100980997086 seconds
[info       - 17:00:15.698] ready to update scratch-nlp-paper.md 8.473162781000138 seconds
[info       - 17:00:15.699] syntax tree scratch-nlp-paper.md 0.0005544349849224091 seconds
[info       - 17:00:15.699] references scratch-nlp-paper.md 0.00008112099766731262 seconds
[info       - 17:00:16.180] delay karger.md 8.788050220012664 seconds
[info       - 17:00:16.180] ready to update karger.md 8.789369559019804 seconds
[info       - 17:00:16.180] syntax tree karger.md 0.0005198630094528198 seconds
[info       - 17:00:16.190] references karger.md 0.00017416200041770936 seconds
[info       - 17:00:16.359] delay inspiration.md 9.126143437981606 seconds
[info       - 17:00:16.359] ready to update inspiration.md 9.127475268989802 seconds
[info       - 17:00:16.360] syntax tree inspiration.md 0.001060470014810562 seconds
[info       - 17:00:16.360] references inspiration.md 0.00014901798963546753 seconds
[info       - 17:00:16.672] delay meeting-personal-karger-weekly-sync.md 9.433948690980673 seconds
[info       - 17:00:16.672] ready to update meeting-personal-karger-weekly-sync.md 9.435593152999878 seconds
[info       - 17:00:16.678] syntax tree meeting-personal-karger-weekly-sync.md 0.005509042978286743 seconds
[info       - 17:00:16.678] references meeting-personal-karger-weekly-sync.md 0.0004667400121688843 seconds
[info       - 17:00:17.420] delay dftw-recitation-10.md 9.777423293977975 seconds
[info       - 17:00:17.420] ready to update dftw-recitation-10.md 9.778968997001648 seconds
[info       - 17:00:17.470] syntax tree dftw-recitation-10.md 0.00455171000957489 seconds
[info       - 17:00:17.470] references dftw-recitation-10.md 0.0002943429946899414 seconds
[info       - 17:00:17.344] delay guess-the-market.md 10.076307039022446 seconds
[info       - 17:00:17.344] ready to update guess-the-market.md 10.077800796985626 seconds
[info       - 17:00:17.345] syntax tree guess-the-market.md 0.00036762002110481265 seconds
[info       - 17:00:17.345] references guess-the-market.md 0.00007654699683189392 seconds
[info       - 17:00:17.604] delay evergreen-notes.md 10.333514546006917 seconds
[info       - 17:00:17.604] ready to update evergreen-notes.md 10.335018400996923 seconds
[info       - 17:00:17.607] syntax tree evergreen-notes.md 0.002500799000263214 seconds
[info       - 17:00:17.607] references evergreen-notes.md 0.00022488200664520262 seconds
[info       - 17:00:17.867] delay bob-sutton.md 10.592393810987472 seconds
[info       - 17:00:17.867] ready to update bob-sutton.md 10.594215402007103 seconds
[info       - 17:00:17.867] syntax tree bob-sutton.md 0.00036609101295471193 seconds
[info       - 17:00:17.867] references bob-sutton.md 0.00008353298902511597 seconds
[info       - 17:00:18.136] delay fallback-ui.md 10.858917524009943 seconds
[info       - 17:00:18.136] ready to update fallback-ui.md 10.860516872018575 seconds
[info       - 17:00:18.137] syntax tree fallback-ui.md 0.0004200960099697113 seconds
[info       - 17:00:18.137] references fallback-ui.md 0.00007807502150535583 seconds
[info       - 17:00:18.400] delay bidmc-metrics-document.md 11.119906026989222 seconds
[info       - 17:00:18.401] ready to update bidmc-metrics-document.md 11.121846403002738 seconds
[info       - 17:00:18.401] syntax tree bidmc-metrics-document.md 0.000698175996541977 seconds
[info       - 17:00:18.401] references bidmc-metrics-document.md 0.00009192198514938354 seconds
[info       - 17:00:18.666] delay student-chat-client-work.md 11.381311989992858 seconds
[info       - 17:00:18.666] ready to update student-chat-client-work.md 11.383105753004552 seconds
[info       - 17:00:18.668] syntax tree student-chat-client-work.md 0.0019080789983272552 seconds
[info       - 17:00:18.668] references student-chat-client-work.md 0.00014218899607658386 seconds
[info       - 17:00:18.933] delay nlp-caption.md 11.63075924000144 seconds
[info       - 17:00:18.933] ready to update nlp-caption.md 11.633836094975472 seconds
[info       - 17:00:18.936] syntax tree nlp-caption.md 0.0031902649998664856 seconds
[info       - 17:00:18.937] references nlp-caption.md 0.00017360597848892212 seconds
[info       - 17:00:19.202] delay msr.md 11.89280514600873 seconds
[info       - 17:00:19.202] ready to update msr.md 11.897238793998957 seconds
[info       - 17:00:19.202] syntax tree msr.md 0.0004899519979953766 seconds
[info       - 17:00:19.202] references msr.md 0.00008799999952316284 seconds
[info       - 17:00:19.491] delay scratch.md 12.17889023399353 seconds
[info       - 17:00:19.491] ready to update scratch.md 12.180544381976128 seconds
[info       - 17:00:19.495] syntax tree scratch.md 0.004307626008987426 seconds
[info       - 17:00:19.495] references scratch.md 0.0002415819764137268 seconds
[info       - 17:00:19.766] delay steve.md 12.450398887991906 seconds
[info       - 17:00:19.766] ready to update steve.md 12.45218401402235 seconds
[info       - 17:00:19.766] syntax tree steve.md 0.00035243999958038333 seconds
[info       - 17:00:19.766] references steve.md 0.0000683859884738922 seconds
[info       - 17:00:20.240] delay bidmc-autocomplete.md 12.689406501024962 seconds
[info       - 17:00:20.240] ready to update bidmc-autocomplete.md 12.691094211995601 seconds
[info       - 17:00:20.380] syntax tree bidmc-autocomplete.md 0.014275559991598128 seconds
[info       - 17:00:20.390] references bidmc-autocomplete.md 0.0005032289922237396 seconds
[info       - 17:00:20.307] delay divya.md 12.968814812004567 seconds
[info       - 17:00:20.307] ready to update divya.md 12.97049341699481 seconds
[info       - 17:00:20.307] syntax tree divya.md 0.0003542919754981995 seconds
[info       - 17:00:20.307] references divya.md 0.0000689769983291626 seconds
[info       - 17:00:20.575] delay haystack.md 13.232484225004912 seconds
[info       - 17:00:20.575] ready to update haystack.md 13.234214065998792 seconds
[info       - 17:00:20.576] syntax tree haystack.md 0.00035224300622940064 seconds
[info       - 17:00:20.576] references haystack.md 0.00007457399368286133 seconds
[info       - 17:00:20.853] delay instructions.md 13.506415098994971 seconds
[info       - 17:00:20.853] ready to update instructions.md 13.508129177004099 seconds
[info       - 17:00:20.853] syntax tree instructions.md 0.0005231010019779205 seconds
[info       - 17:00:20.853] references instructions.md 0.000043557018041610716 second

Pretty sure I've figured it out. When we call updateLinkedFileSyntaxTree we delay for a debounce period. We then parse the file, get all the file references, and call dispatch.

The issue is because we're single threaded we get a huge stack of async calls and overall those calls take 10-15 seconds to resolve. The solution is to create a persistent index.

If you look at the delay times you can see they are increasing linearly.

lukesmurray commented 4 years ago

I added a cache in a224d4f but the cache has some weird behavior.

Loading from the cache takes miniscule amounts of time. But overall the cache end to end takes about 3 seconds. It is a good reduction but I think we could go down to .1 seconds with some work.

lukesmurray commented 4 years ago

A lot of improvements recently.

Next steps. Focus on materializing. Create a boundary between vscode and the materializing code. Ideally we can materialize as a processing step while publishing markdown.m docs with other tools.

Second step. Focus on cache invalidation. Brute force version. Reindex the entire workspace. Cleaner version. Reindex only files which could be invalid. Instances with need for cache invalidation. Right after materialize file. After csl json is updated. Currently offline changes break cache. Headed in a good direction though.

lukesmurray commented 4 years ago

ok cache invalidation is working but could be improved. its still pretty slow.

switch to materializing.

updates are still pretty slow

lukesmurray commented 4 years ago

convert wikilinks to links convert links to wikilinks

[[testing]] goes to [[testing]]

[[testing]] goes to [[testing]]

lukesmurray commented 3 years ago

ok three weeks in and I'm actually quite happy with this. I'm going to create a separate issue for future steps.