search

poleguy / cherrybud

cherrytree viewer
Other
29 stars 1 forks source link

Large files cause crash #1

Open poleguy opened 3 years ago

poleguy commented 3 years ago

360mb file crashes file after 10secs of unresponsiveness. See report on https://github.com/giuspen/cherrytree/issues/328

metal450 commented 3 years ago

Tried it with an 18mb file, & it loaded OK (no crash).

poleguy commented 3 years ago

I just tried with 40, 80, 150, and 310 MB files. It loaded all but the 310 MB file successfully, and yes, it takes a long time to load with no 'loading' message.

Interestingly, these huge files take a long time to load and save on my PC as well, and also show no loading message.

If anybody is loading files bigger than 300 MB, I'd like to know. Please reply here with the size and type of your biggest files. I'd like to handle 99% and gracefully fail at the 1% case. I don't know if it makes sense to architect this to handle really large ctb files.

I have not read through the entire cherrytree repo or history, although as I build this app I probably will have to become familiar with a lot more of it.

@metal450 How big is your file on disk as a ctb? I do intend to support that format as well once I am happy with the UI and stability.

My 310 MB file crashed the program. So I should be able to debug and fix this issue. Thank you for reporting.

metal450 commented 3 years ago

ctb is 273mb. Large files are very common due to how it stores every attachment as stream data within the database itself - so i.e. a few large images or a video attachment or something, and you're there. This is only a problem with ctd though, since ctb is a "proper" database & doesn't require loading the entire file in memory upfront. Pretty sure that's the main reason ctb is default - ctd is prohibitively slow & memory hog due to having to load the entire file preemptively.

metal450 commented 2 years ago

Just curious, i this project no longer under development?

poleguy commented 2 years ago

It's not dead, but I'm at a plateau. I'm trying to decide what the next step should be. I could add ctb support next. But even with ctb the architecture of the app right now tries to load the whole file into memory. So if whether I use ctb or ctd, I really should update the code to only load the currently displayed tab.

But then, it works for my very simple case already, and more important to me would be a search feature?

Select/copy text?

Editing? (If editing will eventually be supported, I'd not like to re-create an editor from scratch, so I'm trying to identify a open source editor codebase that I can start from.)

Maybe better support to open ctb files via various sources properly and re-open the last file (ssh/url/local file)... it seems Android is antagonistic to file viewer type apps... or at least there are not great examples I can find on how to do it well.

metal450 commented 2 years ago

Personally I'd definitely start with CTD first, as that's the default & what most people use (including myself). Followed by incremental loading. Then you've at least got a viewer!

Editing I'd definitely bump till later, as I think having a functional reader first is a much more important step than a reader/writer. And opening from various sources an even lesser priority. Just my 2c. I'd LOVE to be able to just view my notes on my phone, which I presently cannot :)

poleguy commented 2 years ago

I did a little search for salute reader with source as a reference. I'm able to open my ctb file with asqlitemanager and read the text. https://f-droid.org/en/packages/dk.andsen.asqlitemanager I may try to leverage this code. Can you try opening your big ctb on your phone with this viewer? I'd like to know that it works with a big file with real content before pushing forward.

metal450 commented 2 years ago

I installed that apk & oddly, it crashes the moment I launch it - however, I found the same app on the Play Store (https://play.google.com/store/apps/details?id=dk.andsen.asqlitemanager&hl=en_US&gl=US), & that one works. I can indeed oen my ctb file - opens instantly, & can flip through & see my content in the database :)

poleguy commented 2 years ago

Happy New Year. I did a little work and am now successfully compiling the asqlitemanager source on my machine and it runs in the emulator and opens my example ctb. That's a first step toward doing the same in the cherrybud app.

metal450 commented 2 years ago

Awesome!! Man how amazing it would be to have even read-only access to my notes on the go :D

poleguy commented 2 years ago

Just wondering, but where do you actually store your notes? Or maybe the better question is how will you sync them with your phone?

I'm 100% linux and have been using syncthing, or sometimes sshfs, which is generally okay, but once android is in the mix I'm wondering what I will settle on. I do use syncthing on android as well, but I'm interested what other people tend to use.

On Sun, Jan 2, 2022 at 7:28 PM metal450 @.***> wrote:

Awesome!! Man how amazing it would be to have even read-only access to my notes on the go :D

— Reply to this email directly, view it on GitHub https://github.com/poleguy/cherrybud/issues/1#issuecomment-1003825045, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGNA72WTZAEQYBHGFDVLWTUUD3TDANCNFSM4UENL27A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were assigned.Message ID: @.***>

metal450 commented 2 years ago

SyncThing is exactly what I use, across Linux, Windows, & Android :)