FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[ghost]

When changing the channel number of existing channels a crash occurs when attempting to change the second channel.

Using the latest master branch in Docker (19.03.12)

<--- Last few GCs --->

[1:0x4042d70]    44016 ms: Mark-sweep 2058.4 (2063.8) -> 2047.8 (2059.6) MB, 119.1 / 0.0 ms  (+ 230.2 ms in 107 steps since start of marking, biggest step 9.8 ms, walltime since start of marking 395 ms) (average mu = 0.193, current mu = 0.178) allocation [1:0x4042d70]    44443 ms: Mark-sweep 2064.3 (2070.0) -> 2040.3 (2051.4) MB, 391.0 / 0.0 ms  (average mu = 0.139, current mu = 0.084) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1c24ce3c08d1 <JSObject>
    0: builtin exit frame: parse(this=0x1c24ce3dec89 <Object map = 0x2720f4f03549>,0x1d67e04e18d9 <Uint8Array map = 0x2720f4f245b9>,0x1c24ce3dec89 <Object map = 0x2720f4f03549>)

    1: /* anonymous */(aka /* anonymous */) [0x3fd26ce100d9] [/home/node/app/src/dao/channel-db.js:21] [bytecode=0x3747048fe271 offset=32](this=0x0b566a7804b1 <undefined>,0x0b566a7801b9 <null>,0x1d67e04e18d9 <Uint8Array...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa37f40 node::Abort() [./dizquetv]
 2: 0xa38381 node::OnFatalError(char const*, char const*) [./dizquetv]
 3: 0xbb4a4e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [./dizquetv]
 4: 0xbb4ca2 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [./dizquetv]
 5: 0xd73935  [./dizquetv]
 6: 0xd73fc7 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [./dizquetv]
 7: 0xd814e2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [./dizquetv]
 8: 0xd823de v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [./dizquetv]
 9: 0xd8512d v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [./dizquetv]
10: 0xd55a94 v8::internal::Factory::NewJSObjectFromMap(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationType, v8::internal::Handle<v8::internal::AllocationSite>) [./dizquetv]
11: 0xe6673e v8::internal::JsonParser<unsigned short>::BuildJsonObject(v8::internal::JsonParser<unsigned short>::JsonContinuation const&, std::vector<v8::internal::JsonProperty, std::allocator<v8::internal::JsonProperty> > const&, v8::internal::Handle<v8::internal::Map>) [./dizquetv]
12: 0xe68063 v8::internal::JsonParser<unsigned short>::ParseJsonValue() [./dizquetv]
13: 0xe68f0f v8::internal::JsonParser<unsigned short>::ParseJson() [./dizquetv]
14: 0xc6deb8 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [./dizquetv]
15: 0x141ae99  [./dizquetv]

[vexorian]

I've been changing channel numbers for a while and it doesn't reproduce. Maybe it really is running out of memory in your case? My biggest channel json has 20 MB, but I don't really have that many shows. I wonder how big is the channel json for the channel you are trying to rename? I see that the process got to 2 GB

[vexorian]

Okay, I limited my container's memory to 1 GB, and tried moving around some channels and it reached out of memory. It definitely shouldn't need that much memory for that (or should it?) will try to look for leaks.

[ghost]

The size of the json doesn't seem to be related as the crash occurs with any channel. My largest json is 6.8M and a total size of all is 229M.

I've just tried changing some channels again and it isn't crashing if I allow some time between each attempt.

The system has 64GB in total and normal ram usage of the container is around 764MB. However, after changing the first channel number the usage peaks to 2.4GB and then settles to 1.64GB. When the memory usage reaches >2.7GB then the crash occurs.

[vexorian]

I've been testing for a bit. The size of the specific channel being renamed doesn't matter. But the total size of the channels does. Because the memory issue is when generating the TV guide.

What I've found is that the TV guide process is quite wasteful and creates multiple copies of the channel db when there are many TV guide updates pending in the queue. I've been tweaking this a bit and now am able to keep dizquetv running below 500MB when constantly updating my channels (in total my channels are 50MB and the container needs 250 initially, so just 5 extra copies of the channels db should be enough to kill the container).

@morlanth are you able to build docker images from a github branch? If so I can send you a branch that you can test with and confirm that the memory issue is fixed with the changes I am trying.

[ghost]

Yes, I am able to do that but it won’t be until tomorrow morning.

[vexorian]

The current dizquetv:development image should have the fix now.

[ghost]

I've built an image from the 20200909_dev branch.

Initial memory use is lower than before at 318M. After the first channel number change, peak was 1.5G and settled down to 900M. There were no crashes on any of the following attempts and the largest memory peak observed was 1.8G. The memory usage settled down to around 700M on average.