Nexrender-server datafile size problem.

benjioh5 commented 3 years ago

Describe the bug

RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at Immediate.save [as _onImmediate] (/home/some/project/node_modules/@ne
xrender/server/src/helpers/database.js:24:35)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)

./nexrender-server -h

<--- Last few GCs --->

[10993:0x41cdaf0]    10412 ms: Mark-sweep 2049.2 (2051.4) -> 2049.2 (2051.4) MB, 4.2 / 0.0 ms  (average mu = 0.958, current mu = 0.009) last resort GC in old space requested
[10993:0x41cdaf0]    10416 ms: Mark-sweep 2049.2 (2051.4) -> 2049.2 (2051.4) MB, 4.0 / 0.0 ms  (average mu = 0.936, current mu = 0.006) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x18c584d008d1 <JSObject>
    0: builtin exit frame: parse(this=0x18c584d1ec89 <Object map = 0x240bb1a83549>,0x1303bec40119 <Very long string[980291692]>,0x18c584d1ec89 <Object map = 0x240bb1a83549>)

    1: /* anonymous */ [0x113acfbd50f1] [/snapshot/nexrender-server/src/helpers/database.js:0] [bytecode=0x3015e9fb27a9 offset=217](this=0x113acfbd4ca1 <Object map = 0x240bb1a80439>,0x113acfbd4ca1 <Object map = 0x240bb1a804...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0x9cc540 node::Abort() [./nexrender-server]
 2: 0x9ce6c0 node::OnFatalError(char const*, char const*) [./nexrender-server]
 3: 0xaeb86e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [./nexrender-server]
 4: 0xaebbe7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [./nexrender-server]
 5: 0xc809c5  [./nexrender-server]
 6: 0xc8fb10 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [./nexrender-server]
 7: 0xc5d924 v8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [./nexrender-server]
 8: 0xd63cb9 v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [./nexrender-server]
 9: 0xd6619d v8::internal::JsonParser<unsigned short>::ParseJsonValue() [./nexrender-server]
10: 0xd66acf v8::internal::JsonParser<unsigned short>::ParseJson() [./nexrender-server]
11: 0xb8f252  [./nexrender-server]
12: 0xb8f946 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [./nexrender-server]
13: 0x12cbad9  [./nexrender-server]

When ~/nexrender/database.js's size exceeds nodejs (v8)'s heap size or JSON.stringify's length spec, nexrender-server always fail until remove ~/nexrender/database.js file manually.

Information about environment

Which version of nexrender are you using? 1.26.0
What kind of the setup do you have? (server+worker or cli) server + worker
Are you using pre-built binaries or globally installed npm packages? npm
Which kind (and version) of operating system you are running it in? Ubuntu 18.02 LTS
Any additional information that might be helpful

Expected behavior Use JSONStream or automatically remove database.js's finished jobs.

Screenshots

nothing -

benjioh5 commented 3 years ago

I think nexrender-server use db or seperated files for each jobs for efficient management. Nexrender-server is too simple to manage lots of render works (~1000 jobs/daily).

Can I contribute by adding db or patching these code on nexrender-server?

inlife commented 3 years ago

Yeah, for sure. Seems to me that the simple file-based read/write for database outlived itself. At the very least for projects with high amounts of render jobs.

I think it would be great to add something like a switch allowing the user to choose which database backend he'd like to use, with the current single-file-based one being the default (at least for some time, for adoption purposes). And adding some abstraction layer within nexrender so that other users would be able to contribute by adding their database drives of choice as well. What do you think @benjioh5 ?

ivansenic commented 3 years ago

@inlife Is there any progress on this issue? What DB would be the first one to be supported?

inlife commented 3 years ago

@ivansenic first thing that would be probably is the interface and basic file-based storage system, as well as legacy single-file database. From my side I might add some basic Mongodb support, SQLite, and maybe even Redis.

But I have to warn you, no work has been started on this issue yet, so to answer your question: no progress yet.

Thiago-Dantas commented 3 years ago

Been thinking about working on something like this lately, but life still hasn't let me

For my project I'll end up dumping server data in a postgres database, eventually, just don't know how to structure the code.

Interface DB operations or a DB class and let user choose? Is this a server-side config file, environment variable or something else?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

inlife / nexrender

Nexrender-server datafile size problem. #553