search

TiddlyWiki / TiddlyWiki5

A self-contained JavaScript wiki for the browser, Node.js, AWS Lambda etc.
https://tiddlywiki.com/
Other
7.98k stars 1.18k forks source link

text slicer out of memory error #3570

Closed rcrath closed 5 years ago

rcrath commented 5 years ago

Node.js TW5.1.17 on manjaro, 8 GB memory up to date with text slicer plugin. TW is as follows:

THis does not look sustainable as it will produce thousands of inodes...maybe sliced texts should have their own subdirectories.

BTW, found a neat solution to keeping them organized but out of the way...naming them with the name of the original tiddler prefixed with an Omega (Ω) sorts them all after z, keeping them out of the way in searches, which will find stuff in the original text first.

to reproduce

rich[~]$ tiddlywiki /home/rich/Dropbox/FiddlyWiki --server
Serving on 127.0.0.1:8080
(press ctrl-C to exit)
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList 
 filesystem: Saved file /home/rich/Dropbox/FiddlyWiki/tiddlers/$__StoryList.tid 

<--- Last few GCs --->

[2700:0x5594b1669210] 24418040 ms: Mark-sweep 901.1 (945.3) -> 901.0 (914.3) MB, 79.4 / 0.0 ms  (average mu = 1.000, current mu = 0.000) last resort GC in old space requested
[2700:0x5594b1669210] 24418128 ms: Mark-sweep 901.0 (914.3) -> 901.0 (914.3) MB, 87.3 / 0.0 ms  (average mu = 1.000, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x2841df64fb5d]
Security context: 0x3ed38631d949 <JSObject>
    1: DoJoin(aka DoJoin) [0x12e84a924979] [native array.js:~89] [pc=0x2841df6de0a9](this=0x3bc4fb3825b1 <undefined>,0x022e30cd3069 <JSArray[36]>,36,0x3bc4fb382741 <true>,0x3bc4fb382851 <String[0]: >,0x3bc4fb382801 <false>,0x3bc4fb3825b1 <undefined>,0x3bc4fb3825b1 <undefined>)
    2: Join(aka Join) [0x12e84a9249c9] [native array.js:1] [bytecode=0x18f1454e309...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0x5594af5ea361 node::Abort() [node]
 2: 0x5594af5ea3af  [node]
 3: 0x5594af79c842 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0x5594af79ca9b v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0x5594afb323a3  [node]
 6: 0x5594afb44d15 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
 7: 0x5594afb1314a v8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::PretenureFlag, v8::internal::Map*, v8::internal::AllocationAlignment) [node]
 8: 0x5594afb1a551 v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node]
 9: 0x5594afdce410 v8::internal::Runtime_StringBuilderConcat(int, v8::internal::Object**, v8::internal::Isolate*) [node]
10: 0x2841df64fb5d 
Aborted (core dumped)
rich[~]$
Jermolene commented 5 years ago

Hi @rcrath I'm delighted you're experimenting with text-slicer, it's great to get feedback.

It sounds possible that a big part of the problem is the 292MB of images and audio. In the upcoming 5.1.18 release you could shift those to become file attachments, rather than tiddlers:

https://tiddlywiki.com/prerelease/#Using%20the%20integrated%20static%20file%20server

rcrath commented 5 years ago

@Jermolene, ok, I'll give it a try. alpha tester here I go...how do I install alpha under node on Linux? Is there an easy path to changing the folder? Or do I need to go in and reconnect all the links manually? That would be an hour's work or so.

I had a suggestion on the inner anchor-TOC million post thread. I can start a new issue with those suggestions and use the issue for feedback on the text slicer. I have a couple more things that have been working for me, including formatting sliced filenames as "Ω ORIG TTL" the Ω keeps the sliced tiddlers all after the main sort in search results, which I find helpful.

Jermolene commented 5 years ago

Hi @rcrath

The instructions for installing the prerelease under Node.js are here: https://tiddlywiki.com/prerelease/#Installing%20TiddlyWiki%20Prerelease%20on%20Node.js

Please do go ahead and create a new thread for any new text-slicer issues.

rcrath commented 5 years ago
  1. moved audio and large image files to ./tiddlers/files
  2. installed and launched TW5 from git successfully.
  3. Same error.

I am back home on my 16 gig machine instead of the 8 Gb laptop. It was working fine when I left but now gives the same error. It gave the error before I switched to git, too (i.e. TW5 5.1.17)

Here is the cmd line running up-to-date manjaro linux, same as laptop.

Thanks for looking at this. I'd be happy to send the tiddlers folder file list if that would shed any insight

rich[TiddlyWiki5]$ tiddlywiki /home/rich/Dropbox/FiddlyWiki --listen
Serving on http://127.0.0.1:8080
(press ctrl-C to exit)
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList 
 filesystem: Saved file /home/rich/Dropbox/FiddlyWiki/tiddlers/$__StoryList.tid 

<--- Last few GCs --->

[12965:0x564aed2a9210]     9508 ms: Mark-sweep 906.3 (951.9) -> 906.2 (919.9) MB, 48.8 / 0.0 ms  (average mu = 0.931, current mu = 0.000) last resort GC in old space requested
[12965:0x564aed2a9210]     9571 ms: Mark-sweep 906.2 (919.9) -> 906.2 (919.9) MB, 63.5 / 0.0 ms  (average mu = 0.843, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x3b3aa69cfb5d]
Security context: 0x25524ef1d949 <JSObject>
    1: DoJoin(aka DoJoin) [0x33254500dc59] [native array.js:~89] [pc=0x3b3aa6c97e29](this=0x22f0aa6025b1 <undefined>,0x0e2433383881 <JSArray[38]>,38,0x22f0aa602741 <true>,0x22f0aa602851 <String[0]: >,0x22f0aa602801 <false>,0x22f0aa6025b1 <undefined>,0x22f0aa6025b1 <undefined>)
    2: Join(aka Join) [0x33254500dca9] [native array.js:1] [bytecode=0x7db6dbad011...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0x564aebb0a361 node::Abort() [node]
 2: 0x564aebb0a3af  [node]
 3: 0x564aebcbc842 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0x564aebcbca9b v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0x564aec0523a3  [node]
 6: 0x564aec064d15 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
 7: 0x564aec03314a v8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::PretenureFlag, v8::internal::Map*, v8::internal::AllocationAlignment) [node]
 8: 0x564aec03a551 v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node]
 9: 0x564aec2ee410 v8::internal::Runtime_StringBuilderConcat(int, v8::internal::Object**, v8::internal::Isolate*) [node]
10: 0x3b3aa69cfb5d 
Aborted (core dumped)
rich[TiddlyWiki5]$
rcrath commented 5 years ago

hmmm, one more thing. The one thing that changed on both systems before the breakage was running an update which included nodejs, npm, and related stuff among other things. Could there be some introduced problem from node? Cuz it was updating node that may have marked the beginning of the problem, though I updated TW right after updating Node and npm. Just a thought to try and home in on what changed on both systems. I Have a clean TW5 and will start adding stuff back in to see if it breaks at some particular increment.

BurningTreeC commented 5 years ago

Hi @rcath, I've been running the current prerelease for the last week or so and kept it up-to-date constantly, without issues. Could you provide a command line example how you run it so that more people can test with a similar setting?

Jermolene commented 5 years ago

moved audio and large image files to ./tiddlers/files The "files" folder should be in the root of the wiki folder, as a sibling of the "tiddlers" folder, not a child of it.

Best wishes

Jeremy

installed and launched TW5 from git successfully. Same error. I am back home on my 16 gig machine instead of the 8 Gb laptop. It was working fine when I left but now gives the same error. It gave the error before I switched to git, too (i.e. TW5 5.1.17)

Here is the cmd line running up-to-date manjaro linux, same as laptop.

Thanks for looking at this. I'd be happy to send the tiddlers folder file list if that would shed any insight

rich[TiddlyWiki5]$ tiddlywiki /home/rich/Dropbox/FiddlyWiki --listen Serving on http://127.0.0.1:8080 (press ctrl-C to exit) syncer-server-filesystem: Dispatching 'save' task: $:/StoryList filesystem: Saved file /home/rich/Dropbox/FiddlyWiki/tiddlers/$__StoryList.tid

<--- Last few GCs --->

[12965:0x564aed2a9210] 9508 ms: Mark-sweep 906.3 (951.9) -> 906.2 (919.9) MB, 48.8 / 0.0 ms (average mu = 0.931, current mu = 0.000) last resort GC in old space requested [12965:0x564aed2a9210] 9571 ms: Mark-sweep 906.2 (919.9) -> 906.2 (919.9) MB, 63.5 / 0.0 ms (average mu = 0.843, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x3b3aa69cfb5d]

Security context: 0x25524ef1d949 1: DoJoin(aka DoJoin) [0x33254500dc59] [native array.js:~89] [pc=0x3b3aa6c97e29](this=0x22f0aa6025b1 ,0x0e2433383881 <JSArray[38]>,38,0x22f0aa602741 ,0x22f0aa602851 <String[0]: >,0x22f0aa602801 ,0x22f0aa6025b1 ,0x22f0aa6025b1 ) 2: Join(aka Join) [0x33254500dca9] [native array.js:1] [bytecode=0x7db6dbad011...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: 0x564aebb0a361 node::Abort() [node] 2: 0x564aebb0a3af [node] 3: 0x564aebcbc842 v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [node] 4: 0x564aebcbca9b v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [node] 5: 0x564aec0523a3 [node] 6: 0x564aec064d15 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node] 7: 0x564aec03314a v8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::PretenureFlag, v8::internal::Map*, v8::internal::AllocationAlignment) [node] 8: 0x564aec03a551 v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node] 9: 0x564aec2ee410 v8::internal::Runtime_StringBuilderConcat(int, v8::internal::Object*, v8::internal::Isolate) [node] 10: 0x3b3aa69cfb5d Aborted (core dumped) rich[TiddlyWiki5]$

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rcrath commented 5 years ago

Hi @BurningTreeC, the cmd line is the first line of the code excerpt above. tiddlywiki /home/rich/Dropbox/FiddlyWiki --listen

rcrath commented 5 years ago

@Jermolene moving the files folder brought the wiki back up. It looks like I need to manually re-link the moved files. Having trouble with that.

and oddly, the image files continue to work after they have been moved with no ./files prefix in the links to them. Also, should *.meta files follow the image files to the new files folder or stay put in the tiddlers folder?

But I'm glad to get the wiki back up and operational so I can write tomorrow! thnx!

rcrath commented 5 years ago

also, lazy loading does not seem to work with the new --listen command, but still works with --server

@BurningTreeC the command which is working now is tiddlywiki /home/rich/Dropbox/FiddlyWiki --server $:/core/save/lazy-all

BurningTreeC commented 5 years ago

Hi @rcrath , for lazy-loading have you tried the root-tiddler parameter --root-tiddler="$:/core/save/lazy-all" ?

As far as I know, embedding external images works using the [img[path]] syntax

for links to files, does [ext[./files/...]] work?

to embed external files, best practice is to use the _canonical_uri field

rcrath commented 5 years ago

@BurningTreeC using --root-tiddler="$:/core/save/lazy-all" fails to start under 5.1.18. The way I had it, using just the path with no root= option, and --server rather than the newer --listen is the only way I have tried that loads the wiki. is there a way to check if lazy loading is in fact on? the [ext[PATH]] method worked. Thank you for reading the manual to me! I looked at that page but failed to register the ext thing. :)

What am I embedding an external URL in? a pre-existing tiddler? or a new one dedicated to the file?

rcrath commented 5 years ago

I answered the second part through experimentation and it worked great! Thanks. I think we can close this one.