[Feature/Request]: Local Storage

BigBoyBarney commented 1 year ago

Hi! First of all, thank you so much! This is an amazing tool that I'm still learning to utilise to its fullest!

With that said, I can't seem to find where / how all the data is stored. I can export individual projects as json for example, but can I somehow backup my entire knowledge-base?

Even if it's not a built in feature, it could be substituted with the local storage + some sync tool like syncthing. So my question is, where / how does Knowledge store the data, and is there any way to back it up?

Thanks in advance! Barney

RobRoyce commented 1 year ago

Thanks for the feedback! Happy you are finding it useful.

Tl;dr at the bottom

Local Storage

The app uses localStorage to save most data (esp. projects and sources). You can see that at work in storage.service.ts

Chromium stores localStorge on-disk in a LevelDB file in an OS-dependent location:

On macOS, it is stored for me in /Users/<USER-NAME>/Library/Application Support/knowledge-canvas/Local Storage/leveldb.
On Windows, this post suggests it uses C:\Users\<USER-NAME>\AppData\Roaming\<APP-NAME>\Local Storage\leveldb\

Theoretically you could backup your data by backing up the appropriate LevelDB directories.

Future Improvements

I did build an import/export mechanism into the settings menu, but it is currently disabled. You can reenable it by uncommenting the lines in settings.component.ts and rebuilding. Note that there are some issues that have not been worked out yet with this approach (in particular, not all of the Source icons are backed up properly), which is why it is commented out in the latest release.

That said, I know this is less than ideal, and I am all too familiar with the pitfalls of using localStorage (esp. for larger datasets). I've been really busy with work recently, but once I get time, my next major update will be to change the storage layer to something more efficient and scalable. I have a few ideas on how to accomplish that, but nothing is final yet.

Tl;dr

You can backup the LevelDB folder for now
You can enable the storage settings menu and rebuild (but this method is error-prone and needs fixing)
Eventually the storage layer will be migrated and import/export/backup will be made easier

I'll keep this issue open while I address the problems/solutions above.

Thanks again!

BigBoyBarney commented 1 year ago

That's wonderful! the LevelDB folder does indeed back everything up and removing / readding it recovers everything, including settings seemlessly.

Thank you for your work! Looking forward to future updates :)

Have a nice week!

silopolis commented 1 year ago

That said, I know this is less than ideal, and I am all too familiar with the pitfalls of using localStorage (esp. for larger datasets). I've been really busy with work recently, but once I get time, my next major update will be to change the storage layer to something more efficient and scalable. I have a few ideas on how to accomplish that, but nothing is final yet.

Please, if possible, make storage shareable between Knowledge instances, extra super bonus if solution opens opportunity for NextCloud integration... Thanks in advance for considering 🙏

KnowledgeCanvas / knowledge