RareBodhi
commented
1 month ago Interestingly we just had all of our production services crash all at once, running on AWS EC2 instances with Node v20 and v22, all at once, all with the same error.
The other information as provided by Gabba holds true for us, but it out of the blue affected everything all at once after all our AWS k8s nodes got restarted by AWS all at the same time.
We're still investigating the cause and solution, but the timing of this issue being created just a few hours ago seems rather suspiciously coincided with our own, makes me wonder what's going on here.
We also run Java services and they were also affected by this, so I don't believe this to be an issue with NodeJS or V8 itself.
The Image where our issue started v1.30.4-eks-16b398d
The Java error was less verbose but looks like
Error occurred during initialization of VM
Failed to mark memory page as executable - check if grsecurity/PaX is enabledNodeJS Error
#
# Fatal error in , line 0
# Check failed: 12 == (*__errno_location ()).
#
#
#
#FailureMessage Object: 0xffffee1205a0
1: 0xceb064 [node]
2: 0x1f43eb0 V8_Fatal(char const*, ...) [node]
3: 0x1f4e5e8 v8::base::OS::SetPermissions(void*, unsigned long, v8::base::OS::MemoryPermission) [node]
4: 0x10e1974 v8::internal::MemoryAllocator::SetPermissionsOnExecutableMemoryChunk(v8::internal::VirtualMemory*, unsigned long, unsigned long, unsigned long) [node]
5: 0x10e1cb4 v8::internal::MemoryAllocator::AllocateAlignedMemory(unsigned long, unsigned long, unsigned long, v8::internal::AllocationSpace, v8::internal::Executability, void*, v8::internal::VirtualMemory*) [node]
6: 0x10e1eb8 v8::internal::MemoryAllocator::AllocateUninitializedChunkAt(v8::internal::BaseSpace*, unsigned long, v8::internal::Executability, unsigned long, v8::internal::PageSize) [node]
7: 0x10e2488 v8::internal::MemoryAllocator::AllocatePage(v8::internal::MemoryAllocator::AllocationMode, v8::internal::Space*, v8::internal::Executability) [node]
8: 0x10f6e78 v8::internal::PagedSpaceBase::TryExpandImpl() [node]
9: 0x10f98c0 [node]
10: 0x10f9e54 v8::internal::PagedSpaceBase::RefillLabMain(int, v8::internal::AllocationOrigin) [node]
11: 0x1070988 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
12: 0x1050230 v8::internal::Factory::CodeBuilder::AllocateInstructionStream(bool) [node]
13: 0x1050604 v8::internal::Factory::CodeBuilder::BuildInternal(bool) [node]
14: 0xed068c v8::internal::baseline::BaselineCompiler::Build(v8::internal::LocalIsolate*) [node]
15: 0xee2a04 v8::internal::GenerateBaselineCode(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>) [node]
16: 0xf3c1b0 v8::internal::Compiler::CompileSharedWithBaseline(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
17: 0xf3c734 v8::internal::Compiler::CompileBaseline(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
18: 0xece3cc v8::internal::baseline::BaselineBatchCompiler::CompileBatch(v8::internal::Handle<v8::internal::JSFunction>) [node]
19: 0xf4708c v8::internal::Compiler::Compile(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
20: 0x14587a8 v8::internal::Runtime_CompileLazy(int, unsigned long*, v8::internal::Isolate*) [node]
21: 0x1862a84 [node]
anmol-nura
RedYetiDev
rarecrumb
gireeshpunathil
bnoordhuis
Version
v23.0.0
Platform
Subsystem
No response
What steps will reproduce the bug?
By repeatedly running Node, as simply installing dependencies of a project through the command
node $(which npm) installin a while loop (see the script used for testing here).For instance:
How often does it reproduce? Is there a required condition?
Very often on specific platforms such as the one shown above.
What is the expected behavior? Why is that the expected behavior?
Node should not fail and crash.
What do you see instead?
At same point Node crashes giving the following error, extracted from here:
Additional information
Similar issue https://github.com/nodejs/help/issues/4465.
The output of
stracecan be found here.