search

DEgITx / rats-search

BitTorrent P2P multi-platform search engine for Desktop and Web servers with integrated torrent client.
MIT License
1.61k stars 174 forks source link

Out of memory error #145

Open Pheromon opened 3 years ago

Pheromon commented 3 years ago

I leave my client running for days (I suppose this should be the default usage), but after a few hours I get out of heap memory errors.

Can you provide a way to configure the usage of bigger heaps?

DEgITx commented 3 years ago

can yo send rats.log?

Pheromon commented 3 years ago

Sorry I missed your request. I tried to add --max-old-space-size=16000 to the node options and it worked for longer time, but at the end it crashed anyway and now it will not start anymore.

I can't find any rats.log file anywhere: do I need to enable debugging somewhere?

DEgITx commented 3 years ago

it's must be in the same folder where db and other config files, what OS do you using?

Pheromon commented 3 years ago

debian 11 (bullseye)

Pheromon commented 3 years ago

Anyway even if augmenting the memory size in that way seems to help , after a few days of running I still get plenty of errors and I need to restart the server. Maybe I need to give more memory to sphinx or other background tools, too?

Pheromon commented 3 years ago

This issue is till marked with "waiting for reply", do you need more informations?

DEgITx commented 3 years ago

yes, if possible, when it's happens, and if possible attach rats.log, it's must be in Rats on the Boat folder, on unix systems somehre on ~/.config or ~/.share directories

Pheromon commented 3 years ago

There is no such file on my system, I found the directory you say: /home/sw/.config/Rats on The Boat but it has only an empty directory named "Dictionaries"

DEgITx commented 3 years ago

can you send which version of app you are running, web or GUI, the gui version folder must be there and have such struct

image

Pheromon commented 3 years ago

In the directory where I clone the repo and run th server (web version) I have some of the files you show here (for example version.vrs, sphinx.conf, query.og and searchd.log), but no rats.log.

Pheromon commented 3 years ago

BTW, I have no clue how to run the GUI version, you don't explain in the readme, and I'm interested

DEgITx commented 3 years ago

n the directory where I clone the repo and run th server (web version) I have some of the files you show here (for example version.vrs, sphinx.conf, query.og and searchd.log), but no rats.log.

for web version it's in the same directory as sources

DEgITx commented 3 years ago

BTW, I have no clue how to run the GUI version, you don't explain in the readme, and I'm interested

GUI version it's https://github.com/DEgITx/rats-search/releases/tag/v1.7.1 AppImage for example, as desktop application, not web one

Pheromon commented 3 years ago

This morning my instance was not working and on the console I had millions of lines like this one: I had to kill -9 the server to make it stop. A simple kill signal didn't work.

[replication] replicate remote torrent [H-Manga] COMIC Ananga Ranga Vol. 9 [Digital] - COMIC アナンガ・ランガvol.9 [DL版] [Japanese]
[add] Error: Cannot enqueue Query after fatal error.
    at Protocol._validateEnqueue (/home/user/rats-search/node_modules/mysql/lib/protocol/Protocol.js:212:16)
    at Protocol._enqueue (/home/user/rats-search/node_modules/mysql/lib/protocol/Protocol.js:138:13)
[...]
  code: 'PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR',
  fatal: false
}

If it useful (I doubt) I'll open a separate issue, please tell me.

DEgITx commented 3 years ago

for each problem open separate, open and attach log

Pheromon commented 3 years ago

the problem btw is still I have no rats.log anywhere. I deleted the ".config/Rats on The Boat" directory and restarted the program, it didn't get recreated.

Any hint?

DEgITx commented 3 years ago

I already told that rats.log in the same folder with sources for web version

DEgITx commented 3 years ago

I will recheck this

Pheromon commented 3 years ago

I know, but the file doesn't exist anywhere and, believe me, I looked everywhere

so, how can I check why the file is not created?

DEgITx commented 3 years ago

this functionality was missing in web version, I restored in master

Pheromon commented 3 years ago

Can I run master or I risk losing the db again?

DEgITx commented 3 years ago

yes you can

Pheromon commented 3 years ago

So, I run master with 16GB of ram and everything went fine for a couple days but eventually started to get these errors:

[4:26:50 AM] Rejection: Error: Cannot enqueue Query after fatal error.
    at Protocol._validateEnqueue (/home/user/rats-search/node_modules/mysql/lib/protocol/Protocol.js:212:16)
    at Protocol._enqueue (/home/user/rats-search/node_modules/mysql/lib/protocol/Protocol.js:138:13)
[...]
  code: 'PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR',
  fatal: false
}

And even pressing CTRl-C multiple times the server was unable to stop.

At the end I pressed CTRL-\ to kill it and was able to re-run the server.

I don't seem to find something interesting in the log file, do you want me to attach it?

DEgITx commented 3 years ago

please provide me also logs/screen of memory usage of rats processes after few days, also will help

Pheromon commented 3 years ago

The master seems much more stable, it is now able to run for a couple of days without issues. Anyway then it start to hang and I need to kill it, this is the memory situation:

USER      PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
user     4003  0.0 27.1 17581300 16759636 pts/5 Rl+ Jul18 3296:42  |           \_ node --max-old-space-size=16000 src/background/server.js
user     4055  0.0  5.1 6677572 3195356   pts/5 Sl+ Jul18 4046:46  |               \_ /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach

Checking the log I see also many of these errors:

[6:40:24 AM] Error: ER_PARSE_ERROR: sphinxql: syntax error, unexpected $undefined near '`add�ed`,`trackerschecked`,`hash`,`name`,`ipv4`,`info`,`contentType`,`added`,`nameIndex`) VALUES (1488384,20,2097152,666,1,0,0,3,0,2095411200,1534949782,1543831256,'50fa5c
a510d32fe674e608c8e599c70d364a1ef1','Scene','127.0.0.1','{\"name\":\"Сцена / The.Scene [1-20 из 20] [США, 2004, интернет-детектив, Web-Rip 600p]\",\"poster\":\"http://i42.fastpic.ru/big/2012/0701/40/6b90d0f0dbbb07141b3747c342f41840.jpg\",\"description\":\"\\.
..
[...]
  sqlMessage: 'sphinxql: syntax error, unexpected $undefined near \'`add�ed`,`trackerschecked`,`hash`,`name`,`ipv4`,`info`,`contentType`,`added`,`nameIndex`) VALUES (1488384,20,2097152,666,1,0,0,3,0,2095411200,1534949782,1543831256,\'50fa5ca510d32fe674e608c8e599c70d364a1ef1\',\'Scene\',\'127.0.0.1\',\'{\\"name\\":\\"Сцена / The.Scene [1-20 из 20] [США, 2004, интернет-детектив, Web-Rip 600p]\\",\\"poster\\":\\"http://i42.fastpic.ru/big/2012/0701/40/6b90d0f0dbbb07141b3747c342f41840.jpg\\",\\"description\\":\\"\\\\...',
  sqlState: '42000',
  index: 0,
  sql: 'INSERT INTO torrents(`id`,`files`,`piecelength`,`port`,`seeders`,`leechers`,`completed`,`good`,`bad`,`size`,`add�ed`,`trackerschecked`,`hash`,`name`,`ipv4`,`info`,`contentType`,`added`,`nameIndex`) VALUES (1488384,20,2097152,666,1,0,0,3,0,2095411200,1534949782,1543831256,\'50fa5ca510d32fe674e608c8e599c70d364a1ef1\',\'Scene\',\'127.0.0.1\',\'{\\"name\\":\\"Сцена / The.Scene [1-20 из 20] [США, 2004, интернет-детектив, Web-Rip 600p]\\",\\"poster\\":\\"http://i42.fastpic.ru/big/2012/0701/40/6b90d0f0dbbb07141b3747c342f41840.jpg\\",\\"description\\":\\"\\\\n\\\\t\\\\t\\\\t\\\\tСцена / The.Scene \\\\n\\\\n\\\\nГод выпуска: 2004-2006\\\\nСтрана: США\\\\nЖанр: интернет-детектив\\\\nРежиссёр: Mitchell Reichgut\\\\nВ ролях: Joe Testa, Trice Able, Laura Minarich, Dinarte Freitas, Noah Rothman, Jill Howell, Curt Rosloff, Nick White\\\\nПеревод: �юбительский (многоголосый закадровый)\\\\nРусские су�титры: нет\\\\n�родолжительность: 20 серий по 15-25 минут\\\\nОп�сание: В этом первом в своем роде ф�льме действие происходит на э
кране ком�ьютеров его геро�в, где мы видим, �ак они общаются в чате. Содержание сериала лучше всего описывают слова гла�ного героя, произносимы� за кадром:\\\\n�олее 200 миллионов людей в мире скачивают музыку, фильмы, сериалы и шоу, игры и другие электронные
 развл�чения. Одна только музыка оценивается в более чем миллиард скачиваний в неделю. Большинство людей которые участв�ют в файлообмене, не имеют понятия, откуда они берутся. Они понятия не имеют, что есть целый мир, огромная глобальная инфраструктура, котор
ая существует исклю�ительно для того, чтобы доставать и поглощать новый контент. На верхнем уровне этого механизма - место, откуда берутс� фактически все эти �айлы, которые потом расползаются по всему миру. Это �есто, где заключаются сделки, где доступ и иден
тификац�� личности ревностно о�раняются, и где э�итная группа технократов дергает за н�видимые рычаги. Те из нас, кто облюбовали себе это место, придумали для него название. Мы называем его \\\\\\"сцена\\\\\\".\\\\n\\\\nДополнитель�ая информация: В тексте при
сутствует ненормативная лексика.\\\\nПодробная информа�ия о сериале: http://rutracker.org/forum/viewtopic.php?t=4079882\\\\n\\\\nПѵревод: ID-Daemon\\\\nОзвучивание: Sad Kit, Zack, lanz, Skaner, Kineza, Legonaft, ARiS, ID-Daemon, Uncle_Michael, Joss\\\\nРабота
 со звуком: Joss\\\\nРабота � видео: ARiS, ID-Daemon, SckraT\\\\n\\\\n \\\\n\\\\n\\\\nРелиз от: \\\\n\\\\nФормат видео: AVI\\\\nКачество: Web-Rip\\\\nВидео кодек: XviD\\\\n\\\\nВидео\\\\nЭпи�од 1: Xvid, 800x600, 299 кбит/сек, 14.984 fps\\\\nЭпизод 2: Xvid, 800x600, 249 кбит/сек, 9.989 fps\\\\nЭпизод 3: Xvid, 800x600, 434 кбит/сек, 24 fps\\\\nЭпизод 4: Xvid, 800x600, 522 кбит/сек, 24 fps\\\\nЭпизод 5: Xvid, 800x600, 520 кбит/сек, 23.976 fps\\\\nЭпизод 6: Xvid, 800x600, 517 кбит/сек, 23.976 fps\\\\nЭпиз�д 7: Xvid, 800x600, 517 кбит/сек, 23.976 fps\\\\nЭпизод 8: Xvid, 800x600, 606 кбит/сек, 29.970 fps\\\\nЭпизод 9: Xvid, 800x600, 511 кбит/сек, 15 fps\\\\nЭпизод 10: Xvid, 800x600, 619 кбит/сек, 29.970 fps\\\\n�пизод 11: Xvid, 800x600, 599 кбит/сек, 29.970 fps\\\\nЭпизод 12: Xvid, 800x600, 399 кбит/сек, 15 fps\\\\nЭпизод 13: Xvid, 800x600, 650 кбит/сек, 29.970 fps\\\\nЭпизод 14: Xvid, 800x600, 600 кби�/с�к, 29.970 fps\\\\nЭпизод 15: Xvid, 800x600, 600 кбит/сек, 29.970 fps\\\\nЭпизод 16: Xvid, 800x600, 599 кбит/сек, 29.970 fps\\\\nЭпизод 17: Xvid, 800x600, 738 �бит/сек, 29.970 fps\\\\nЭп�зод 18: Xvid, 800x600, 599 кбит/сек, 29.970 fps\\\\nЭпизод 19: X�vid, 800x600, 5�99 кбит/сек, 29.970 fps\\\\nЭпизод 20: Xvid, 800x600, 599 кбит/сек, 29.970 fps\\\\n\\\\nАудио: MP3 2ch, 128 кбит/се
к, 48 КГц\\\\nHD-версия 720p: http://rutracker.org/forum/viewtopic.php?t=4064846\\\\n\\\\nДополнительные сведения о сериале\\\\nСериал уникаль�ый по формату, это н� кино, как мы его привыкли видеть, а какой-то �ругой, новый вид искусства. За вс� эти годы никт
о, никто не снял ничего подобного. Персонажей фильма мы видим лишь в маленьком о�не, основное место в видеоряде занима�т и�ображение экрана компьютера. Основные события происходят именно там.\\\\n\\\\nАтмосфера производит неизгладимое впечатление. Зритель сле
дит за происходящим на экране ком�ьютера, слышит стук клавиш, музыку, к�торую с�уш�ет персонаж. М� види� то, что сегодня привыкли делать каждый день, и уже не замечаем, как это выгля�ит. Зритель как бы оказывается на месте героев филь�а.\\\\n\\\\nГлавн��е гер
ои - участники пиратской сценовой рип-группы, распространяющей фильмы-нови��ки, до их официального выхода. Сюжет закручивается вокруг того, что бывает, если в процесс файлообмена вклиниваются деньги, и чем это может кончиться.\\\\nСериал распространяется по л
ицен�ии Creative Commons, то есть это свободный сер�ал, снятый независимыми продюсерами.\\\\n\\\\n\\\\nСкри�шоты\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n Раздача саундтрека \\\\n\\\\t\\\\t\\\\t\\",\\"rutrackerThreadId\\":3600572,\\"contentCategory\\":\\"Главна
я » Сериалы » Зарубежные сериалы » Веб-сериалы, Вебизоды к �ериалам и Пилотные с�рии сериалов\\",\\"trackers\\":[\\"rutracker\\",\\"rutor\\"],\\"rutorThreadId\\":237539}\',\'video\',1534949782,\'Scene Сцена / The.Scene [1-20 из 20] [США, 2004, интернет-детект
ив, Web-Rip 600p]\')'
}

[...]

[6:40:24 AM] Error: ER_PARSE_ERROR: sphinxql: syntax error, unexpected $undefined near '`add�ed`,`trackerschecked`,`hash`,`name`,`ipv4`,`info`,`contentType`,`added`,`nameIndex`) VALUES (1488384,20,2097152,666,1,0,0,3,0,2095411200,1534949782,1543831256,'50fa5c
a510d32fe674e608c8e599c70d364a1ef1','Scene','127.0.0.1','{\"name\":\"Сцена / The.Scene [1-20 из 20] [США, 2004, интернет-детектив, Web-Rip 600p]\",\"poster\":\"http://i42.fastpic.ru/big/2012/0701/40/6b90d0f0dbbb07141b3747c342f41840.jpg\",\"description\":\"\\.
..
    at Query.Sequence._packetToError (/home/user/rats-search/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)

[...]

[6:40:29 AM] Error: ER_PARSE_ERROR: sphinxql: syntax error, unexpected $undefined near '`leech�ers`,`completed`,`good`,`bad`,`size`,`added`,`trackerschecked`,`hash`,`name`,`ipv4`,`contentType`,`nameIndex`) VALUES (1488528,1273,524288,46608,0,0,0,1,0,468999661,1523186269,1537414329,'1145b14f201b4f691c6f5f1d31bf6a122e744811','Navitel 9.5.30 [Android OS]','128.72.164.180','application','Navitel 9.5.30 [Android OS]')'

[...]

[6:40:31 AM] Error: ER_PARSE_ERROR: sphinxql: syntax error, unexpected $undefined near '`nam�e`,`ipv4`,`contentType`,`nameIndex`) VALUES (1488589,1,2097152,14065,93,6,174,1,0,1699558733,1535360297,1535360429,'c4f7b0b1bd53d0a477cbd5c790141c0b4f9c1555','Wolf.Gi
rl.With.You.v1.0.0.6.zip','77.53.161.204','archive',NULL)'
    at Query.Sequence._packetToError (/home/user/rats-search/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)

[...]

[6:40:33 AM] Error: ER_PARSE_ERROR: sphinxql: syntax error, unexpected $undefined near '`piecelength�`,`port`,`seeders`,`leechers`,`completed`,`good`,`bad`,`size`,`added`,`trackerschecked`,`hash`,`name`,`contentType`,`nameIndex`) VALUES (1488637,448,8388608,0,0,0,0,1,1,81709046123,1520588553,1532789442,'7e0cfe168f7a956dfaa7bd65622db53ce11fd6ca','БМ','video','БМ')'

[...]

[1:51:05 PM] Error: ER_PARSE_ERROR: index torrents: syntax error, unexpected '~' near ' ~到了异世界就拿出真本事'
    at Query.Sequence._packetToError (/home/user/rats-search/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)
    at Query.ErrorPacket (/home/user/rats-search/node_modules/mysql/lib/protocol/sequences/Query.js:79:18)

[...]

If useful I can attach full logs

Pheromon commented 3 years ago

Memory situation before today's crash

22174  0.0  0.0   2424   536 pts/5    S+   Jul23   0:00  |       \_ sh -c node --max-old-space-size=16000 src/background/server.js
22175  0.0 32.0 20572424 19756860 pts/5 Rl+ Jul23 3735:23  |           \_ node --max-old-space-size=16000 src/background/server.js
22229  0.0  6.2 7119344 3855224 pts/5 Sl+  Jul23 6887:09  |               \_ /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
Pheromon commented 3 years ago

today it crashed by itself and these are the latest rows on the console:

[tracker] found Nachalo.2010. BDRip.avi on rutracker
[system] Rejection: Error: Cannot enqueue Query after fatal error.
    at Protocol._validateEnqueue (/home/user/rats-search/node_modules/mysql/lib/protocol/Protocol.js:212:16)
    at Protocol._enqueue (/home/user/rats-search/node_modules/mysql/lib/protocol/Protocol.js:138:13)
    at Connection.query (/home/user/rats-search/node_modules/mysql/lib/Connection.js:198:25)                                                                                                                                                                          at queryCall (/home/user/rats-search/src/background/mysql.js:12:10)
    at /home/user/rats-search/src/background/mysql.js:18:4                                                                                                                                                                                                            at new Promise (<anonymous>)
    at Connection.sphinx.query (/home/user/rats-search/src/background/mysql.js:15:42)                                                                                                                                                                                 at Object.randomTorrents (/home/user/rats-search/src/background/api.js:221:17)
    at Socket.<anonymous> (/home/user/rats-search/src/background/p2p.js:96:40)                                                                                                                                                                                        at Socket.emit (node:events:394:28) {
  code: 'PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR',
  fatal: false
}                                                                                                                                                                                                                                                                  

<--- Last few GCs --->                                                                                                                                                                                                                                             
[22446:0x6652f60] 256979190 ms: Scavenge (reduce) 15603.5 (16043.0) -> 15603.1 (16043.0) MB, 44.9 / 0.0 ms  (average mu = 0.385, current mu = 0.384) allocation failure
[22446:0x6652f60] 256979242 ms: Scavenge (reduce) 15603.7 (16043.6) -> 15603.4 (16043.4) MB, 43.7 / 0.0 ms  (average mu = 0.385, current mu = 0.384) allocation failure
[22446:0x6652f60] 256979375 ms: Scavenge (reduce) 15605.1 (16044.2) -> 15604.4 (16044.2) MB, 47.9 / 0.0 ms  (average mu = 0.385, current mu = 0.384) allocation failure

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb17ec0 node::Abort() [node]
 2: 0xa341f4 node::FatalError(char const*, char const*) [node]
 3: 0xcfe71e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xcfea97 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xee8d35  [node]
 6: 0xee987c  [node]
 7: 0xef77b1 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0xefad0c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xebebb5 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
10: 0xeb9a59 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [node]
11: 0xeb9b14 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>, int, v8::internal::Handle<v8::internal::Oddball>, v8::internal::AllocationType) [node]
12: 0x110269d v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::New<v8::internal::Isolate>(v8::internal::Isolate*, int, v8::internal::AllocationType, v8::internal::Mini
mumCapacity) [node]
13: 0x11028d3 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDic
tionary>, int, v8::internal::AllocationType) [node]
14: 0x1102e96 v8::internal::Handle<v8::internal::NameDictionary> v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>,
v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [node]
15: 0x110665a v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8:
:internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [node]
16: 0x10d5989 v8::internal::LookupIterator::ApplyTransitionToDataProperty(v8::internal::Handle<v8::internal::JSReceiver>) [node]
17: 0x10f4003 v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [node]
18: 0x10abdd9 v8::internal::JSObject::DefineOwnPropertyIgnoreAttributes(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::JSObject::AccessorInfoHand
ling) [node]
19: 0x10ac0f9 v8::internal::JSObject::DefineOwnPropertyIgnoreAttributes(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::JSObject::AccessorInfoHandling) [node]
20: 0xfdf18a v8::internal::JsonParser<unsigned char>::BuildJsonObject(v8::internal::JsonParser<unsigned char>::JsonContinuation const&, std::vector<v8::internal::JsonProperty, std::allocator<v8::internal::JsonProperty> > const&, v8::internal::Handle<v8::inter
nal::Map>) [node]
21: 0xfe0729 v8::internal::JsonParser<unsigned char>::ParseJsonValue() [node]
22: 0xfe15ef v8::internal::JsonParser<unsigned char>::ParseJson() [node]
23: 0xda3638  [node]
24: 0xda3c06 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [node]
25: 0x16148d9  [node]
Aborted
Pheromon commented 3 years ago

BTW, the issues is still marked with "awaiting reply", may I give you more informations?

DEgITx commented 3 years ago

if possible also make screen of htop after few days of usage and attach here

Pheromon commented 3 years ago

I will collect the relevant statistics of the process, in the meantime version 1.8 after a few days of great working crashed in this way. The server was launched with this parameter to make it last longer: --max-old-space-size=26000

<--- Last few GCs --->

[7687:0x5318f60] 292367133 ms: Scavenge 16029.4 (16511.9) -> 16027.9 (16510.6) MB, 78.5 / 0.0 ms  (average mu = 0.624, current mu = 0.000) allocation failure
[7687:0x5318f60] 292367331 ms: Scavenge 16037.1 (16516.3) -> 16035.3 (16516.3) MB, 56.2 / 0.1 ms  (average mu = 0.624, current mu = 0.000) allocation failure
[7687:0x5318f60] 292367401 ms: Scavenge 16035.3 (16516.3) -> 16035.2 (16516.3) MB, 70.3 / 0.0 ms  (average mu = 0.624, current mu = 0.000) allocation failure

<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 0xb17ec0 node::Abort() [node]
 2: 0xa341f4 node::FatalError(char const*, char const*) [node]
 3: 0xcfe71e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xcfea97 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xee8d35  [node]
 6: 0xf16858 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [node]
 7: 0xf21bfe v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [node]
 8: 0xf10edf v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [node]
 9: 0xf1120f v8::internal::PageEvacuationJob::Run(v8::JobDelegate*) [node]
10: 0x18fedf4 v8::platform::DefaultJobWorker::Run() [node]
11: 0xb83345  [node]
12: 0x7fa05b72fea7  [/lib/x86_64-linux-gnu/libpthread.so.0]
13: 0x7fa05b65fdef clone [/lib/x86_64-linux-gnu/libc.so.6]
Aborted
Pheromon commented 3 years ago

This a part of the htop situation after a couple days

  PID USER      PRI  NI  VIRT   RES   SHR S CPU%▽MEM%   TIME+  Command 
28454 user      20   0 3930M 3230M 2680M S 472.  5.4     515h /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28463 user      20   0 3930M 3230M 2680M S 75.8  5.4 51h19:47 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
 9007 user      20   0 12.7G 11.9G 40584 S 72.2 19.8 21h33:49 node --max-old-space-size=26000 src/background/server.js
28462 user      20   0 3930M 3230M 2680M S 61.4  5.4 51h18:23 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28458 user      20   0 3930M 3230M 2680M S 50.5  5.4 51h17:55 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28459 user      20   0 3930M 3230M 2680M S 50.5  5.4 51h21:08 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28461 user      20   0 3930M 3230M 2680M S 46.9  5.4 51h18:06 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28456 user      20   0 3930M 3230M 2680M S 43.3  5.4 51h19:05 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28457 user      20   0 3930M 3230M 2680M R 43.3  5.4 51h18:38 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28455 user      20   0 3930M 3230M 2680M S 39.7  5.4 51h18:25 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28464 user      20   0 3930M 3230M 2680M S 32.5  5.4 51h21:25 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28460 user      20   0 3930M 3230M 2680M R 25.3  5.4 51h19:22 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
Pheromon commented 3 years ago

htop dump a day later: as you can see the used memory grows until it reaches the maximum allowed

28454 user      20   0 4082M 3341M 2780M S 198.  5.5     582h /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
 9007 user      20   0 18.7G 17.9G 40584 R 95.5 29.8 41h54:03 node --max-old-space-size=26000 src/background/server.js
28461 user      20   0 4082M 3341M 2780M S 18.1  5.5 57h56:13 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28456 user      20   0 4082M 3341M 2780M S 21.3  5.5 57h58:13 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28459 user      20   0 4082M 3341M 2780M S 18.7  5.5 57h59:40 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28464 user      20   0 4082M 3341M 2780M R 29.0  5.5 58h00:32 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28462 user      20   0 4082M 3341M 2780M S 18.7  5.5 57h57:10 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28463 user      20   0 4082M 3341M 2780M S 15.5  5.5 57h58:42 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28457 user      20   0 4082M 3341M 2780M S 17.4  5.5 57h57:03 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28455 user      20   0 4082M 3341M 2780M S 16.8  5.5 57h57:03 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28458 user      20   0 4082M 3341M 2780M S 21.9  5.5 57h56:42 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
28460 user      20   0 4082M 3341M 2780M S 20.6  5.5 57h58:13 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
 9009 user      20   0 18.7G 17.9G 40584 S  0.6 29.8 26:43.55 node --max-old-space-size=26000 src/background/server.js
 9011 user      20   0 18.7G 17.9G 40584 S  0.0 29.8 26:57.65 node --max-old-space-size=26000 src/background/server.js
 9010 user      20   0 18.7G 17.9G 40584 S  0.0 29.8 26:55.70 node --max-old-space-size=26000 src/background/server.js
 9012 user      20   0 18.7G 17.9G 40584 S  1.3 29.8 27:05.50 node --max-old-space-size=26000 src/background/server.js
 3419 user      20   0 4082M 3341M 2780M S  0.6  5.5  2h26:56 /home/user/rats-search/imports/linux/x64/searchd --config /home/user/rats-search/sphinx.conf --nodetach
 9056 user      20   0 18.7G 17.9G 40584 S  0.0 29.8  4:38.06 node --max-old-space-size=26000 src/background/server.js