Fatal process out of memory: DateTimePatternGeneratorCache::CreateGenerator

[umizaki]

Hello,

I'm encountering an issue when I pass a specific URL as an argument to thug and run it, an error message is displayed.

Specifically, when I run:

$ venv/bin/thug https://www.yahoo.co.jp

I get the error message Fatal process out of memory: DateTimePatternGeneratorCache::CreateGenerator (see details below). The data is saved in MongoDB.

The same result occurs when using NODE_OPTIONS:

$ NODE_OPTIONS="--max-old-space-size=4096" venv/bin/thug https://www.yahoo.co.jp

However, if I adjust the number of pages to fetch using -t, the error message does not appear:

$ venv/bin/thug -t 15 https://www.yahoo.co.jp

Here is the version of Thug and host information:

$ venv/bin/thug -V
Thug 6.8 (JS Engine: Google V8 v12.6.228.19)
$ uname -a
Linux vagrant 5.15.0-112-generic #122-Ubuntu SMP Thu May 23 07:48:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Here are the details of the error message:

$ venv/bin/thug https://www.yahoo.co.jp
[2024-06-25 07:06:00] [window open redirection] about:blank -> https://www.yahoo.co.jp
[2024-06-25 07:06:00] [HTTP] URL: https://www.yahoo.co.jp (Status: 200,  Referer: None)

(snip)

[2024-06-25 07:06:03] [HTTP] URL: https://s.yimg.jp/images/ds/yas/ual-2.10.2.min.js (Status: 200,  Referer: https://www.yahoo.co.jp)
[2024-06-25 07:06:03] [HTTP] URL: https://s.yimg.jp/images/ds/yas/ual-2.10.2.min.js (Content-type: application/javascript, MD5: 163718d1f89949da8baaa4d2f18efdca)

<--- Last few GCs --->

[23795:0x56307c90e000]     3455 ms: Scavenge 4.5 (6.8) -> 4.2 (9.1) MB, pooled: 0 MB, 1.87 / 0.23 ms  (average mu = 1.000, current mu = 1.000) allocation failure;

<--- JS stacktrace --->

#
# Fatal process out of memory: DateTimePatternGeneratorCache::CreateGenerator
#
==== C stack trace ===============================

    /home/vagrant/venv/lib/python3.12/site-packages/_STPyV8.cpython-312-x86_64-linux-gnu.so(v8::base::debug::StackTrace::StackTrace()+0x13) [0x7efc28c296c3]
    /home/vagrant/venv/lib/python3.12/site-packages/_STPyV8.cpython-312-x86_64-linux-gnu.so(+0xdb52fb) [0x7efc28c2b2fb]
    /home/vagrant/venv/lib/python3.12/site-packages/_STPyV8.cpython-312-x86_64-linux-gnu.so(v8::base::FatalOOM(v8::base::OOMType, char const*)+0x38) [0x7efc29673fd8]
    /home/vagrant/venv/lib/python3.12/site-packages/_STPyV8.cpython-312-x86_64-linux-gnu.so(v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&)+0x2ac) [0x7efc28c2e96c]
    /home/vagrant/venv/lib/python3.12/site-packages/_STPyV8.cpython-312-x86_64-linux-gnu.so(+0x10ec8ca) [0x7efc28f628ca]
    /home/vagrant/venv/lib/python3.12/site-packages/_STPyV8.cpython-312-x86_64-linux-gnu.so(v8::internal::JSDateTimeFormat::CreateDateTimeFormat(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::JSDateTimeFormat::RequiredOption, v8::internal::JSDateTimeFormat::DefaultsOption, char const*)+0x68b) [0x7efc28f60bfb]
    /home/vagrant/venv/lib/python3.12/site-packages/_STPyV8.cpython-312-x86_64-linux-gnu.so(v8::internal::JSDateTimeFormat::ToLocaleDateTime(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::JSDateTimeFormat::RequiredOption, v8::internal::JSDateTimeFormat::DefaultsOption, char const*)+0x18a) [0x7efc28f6044a]
    /home/vagrant/venv/lib/python3.12/site-packages/_STPyV8.cpython-312-x86_64-linux-gnu.so(v8::internal::Builtin_DatePrototypeToLocaleTimeString(int, unsigned long*, v8::internal::Isolate*)+0x13d) [0x7efc296b2dbd]
    /home/vagrant/venv/lib/python3.12/site-packages/_STPyV8.cpython-312-x86_64-linux-gnu.so(+0x16c37f6) [0x7efc295397f6]
Trace/breakpoint trap (core dumped)

[buffer]

Thanks, I am aware of this issue. This is a Google V8 bug and I already opened a ticket about it to V8 developers a few weeks ago. Unfortunately all we can do is waiting for them to fix it

[umizaki]

I appreciate your prompt response and explanation on this issue. I have understood the content. I’m wondering if I should also wait for the bug in Google V8 to be fixed. Since this does not seem to be a thug issue, I’m considering closing this issue. As I’m not familiar with GitHub, could you please advise me on this?

[buffer]

Sorry for the late response. Yes, I think the issue could be closed as I have no other option than waiting for a Google V8 fix to be merged. Thanks!