search

souvikinator / notion-to-md

Convert notion pages, block and list of blocks to markdown (supports nesting and custom parsing)
https://www.npmjs.com/package/notion-to-md
MIT License
1.08k stars 89 forks source link

Significant Performance Improvement in Markdown Conversion Process #95

Open lucas-almeida026 opened 1 year ago

lucas-almeida026 commented 1 year ago

Hi Souvik Kar Mahapatra and contributors,

Firstly, I'd like to thank you for your work on the notion-to-md library. It's been incredibly useful for personal and professional projects.

While using your library, I noticed some opportunities for performance optimizations and implemented a different architecture that led to significant improvements in the markdown conversion process. However, this new architecture diverges substantially from the original and does not support all the features of the current version, such as custom transformers.

Despite these differences, I wanted to share the work with you because I believe some elements could potentially benefit the notion-to-md library and its users.

The new approach involves three main steps:

  1. We request all blocks from a page in a single operation, storing them in a 2D array (layers of blocks).
  2. Reconstruct the block tree using parent and blockId properties.
  3. Finally, traverse the tree recursively and covert all blocks into a valid markdown string. - gist with the code here

I did some local tests to measure the improvement, I'm aware that this kind of testing might not be precise enough and can be biased, however I believe it can serve as a starting point.

In this table you can see some key statistics from these tests, the overall result is approximate 40% faster execution time for the entire process described in previous steps.

I understand that the approach taken is probably not suitable for direct integration into the library due to its architectural differences and reduced feature set. However, I thought that sharing it may result in more improvements for the future.

Please let me know if you're interested in exploring these improvements further. I would be happy to share more details or collaborate in integrating some of these optimizations into the library.

Best, Lucas Almeida

souvikinator commented 1 year ago

Hi Lucas, Thanks for the effort and contribution, really appreciate it. I'll go through the code and then I guess we can discuss on what can be done and how can we make the features compatible with the new changes.