search

pulumi / pulumi

Pulumi - Infrastructure as Code in any programming language 🚀
https://www.pulumi.com
Apache License 2.0
21.77k stars 1.12k forks source link

Strange error messages when performing `pulumi up` or `pulumi preview` #17040

Closed smith558 closed 2 months ago

smith558 commented 2 months ago

What happened?

Every 2nd time (sometimes more often) I run pulumi up or pulumi preview I get these errors. It's getting incredibly frustrating and is extremely slowing down my work. There seems to be no clear cause because sometimes the same code runs without a problem and then identical code next time fails with the error. Waiting a minute or longer between performing the commands seems to be the only thing helping so far.

pulumi up
Previewing update (staging)

View in Browser (Ctrl+O): https://app.pulumi.com/*******/hub-infrastructure/staging/previews/d150547f-879c-479a-9fe5-8e50382e2b14

     Type                 Name                        Plan     Info
     pulumi:pulumi:Stack  hub-infrastructure-staging           1 error; 25 messages

Diagnostics:
  pulumi:pulumi:Stack (hub-infrastructure-staging):
    #
    # Fatal error in , line 0
    # Check failed: needs_context && current_scope_ == closure_scope_ && current_scope_->is_function_scope() && !function_.is_null() implies function_->context() != *context_.
    #
    #
    #
    #FailureMessage Object: 0x7ffd27527630
    ----- Native stack trace -----
     1: 0x100d701  [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
     2: 0x28d98cb V8_Fatal(char const*, ...) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
     3: 0x1342f81 v8::internal::ScopeIterator::VisitLocalScope(std::function<bool (v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::Object>, v8::internal::ScopeIterator::ScopeType)> const&, v8::internal::ScopeIterator::Mode, v8::internal::ScopeIterator::ScopeType) const [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
     4: 0x134343e v8::internal::ScopeIterator::ScopeObject(v8::internal::ScopeIterator::Mode) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
     5: 0x133fa82 v8::internal::DebugScopeIterator::GetObject() [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
     6: 0x194e869 v8_inspector::V8DebuggerAgentImpl::currentCallFrames(std::unique_ptr<std::vector<std::unique_ptr<v8_inspector::protocol::Debugger::CallFrame, std::default_delete<v8_inspector::protocol::Debugger::CallFrame> >, std::allocator<std::unique_ptr<v8_inspector::protocol::Debugger::CallFrame, std::default_delete<v8_inspector::protocol::Debugger::CallFrame> > > >, std::default_delete<std::vector<std::unique_ptr<v8_inspector::protocol::Debugger::CallFrame, std::default_delete<v8_inspector::protocol::Debugger::CallFrame> >, std::allocator<std::unique_ptr<v8_inspector::protocol::Debugger::CallFrame, std::default_delete<v8_inspector::protocol::Debugger::CallFrame> > > > > >*) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
     7: 0x1953e85 v8_inspector::V8DebuggerAgentImpl::didPause(int, v8::Local<v8::Value>, std::vector<int, std::allocator<int> > const&, v8::debug::ExceptionType, bool, v8::base::EnumSet<v8::debug::BreakReason, int>) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
     8: 0x195fba9  [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
     9: 0x1975f32 v8_inspector::V8InspectorImpl::forEachSession(int, std::function<void (v8_inspector::V8InspectorSessionImpl*)> const&) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
    10: 0x1962471 v8_inspector::V8Debugger::BreakProgramRequested(v8::Local<v8::Context>, std::vector<int, std::allocator<int> > const&, v8::base::EnumSet<v8::debug::BreakReason, int>) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
    11: 0x1353c62 v8::internal::Debug::OnDebugBreak(v8::internal::Handle<v8::internal::FixedArray>, v8::internal::StepAction, v8::base::EnumSet<v8::debug::BreakReason, int>) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
    12: 0x1353fb0 v8::internal::Debug::HandleDebugBreak(v8::internal::IgnoreBreakMode, v8::base::EnumSet<v8::debug::BreakReason, int>) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
    13: 0x13385fe v8::debug::BreakRightNow(v8::Isolate*, v8::base::EnumSet<v8::debug::BreakReason, int>) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
    14: 0x13bf329 v8::internal::Isolate::InvokeApiInterruptCallbacks() [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
    15: 0x13daf1f v8::internal::StackGuard::HandleInterrupts(v8::internal::StackGuard::InterruptLevel) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
    16: 0x1898d07 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [/home/stanley/.nvm/versions/node/v22.6.0/bin/node]
    17: 0x7d8f194ac576

    error: an unhandled error occurred: Program exited with non-zero exit code: -1

Example

N/A

Output of pulumi about

pulumi about
CLI          
Version      3.129.0
Go Version   go1.22.6
Go Compiler  gc

Plugins
KIND      NAME          VERSION
resource  azure-native  2.56.0
language  nodejs        unknown

Host     
OS       ubuntu
Version  24.04
Arch     x86_64

This project is written in nodejs: executable='/home/stanley/.nvm/versions/node/v22.6.0/bin/node' version='v22.6.0'

Current Stack: *******/hub-infrastructure/staging

TYPE                                  URN
pulumi:pulumi:Stack                   urn:pulumi:staging::hub-infrastructure::pulumi:pulumi:Stack::hub-infrastructure-staging
pulumi:providers:azure-native         urn:pulumi:staging::hub-infrastructure::pulumi:providers:azure-native::default_2_56_0
azure-native:resources:ResourceGroup  urn:pulumi:staging::hub-infrastructure::azure-native:resources:ResourceGroup::staging-group
azure-native:app:ManagedEnvironment   urn:pulumi:staging::hub-infrastructure::azure-native:app:ManagedEnvironment::managed-environment
azure-native:resources:ResourceGroup  urn:pulumi:staging::hub-infrastructure::azure-native:resources:ResourceGroup::staging
azure-native:web:StaticSite           urn:pulumi:staging::hub-infrastructure::azure-native:web:StaticSite::web-app
azure-native:app:ManagedEnvironment   urn:pulumi:staging::hub-infrastructure::azure-native:app:ManagedEnvironment::managed-environment
azure-native:app:ContainerApp         urn:pulumi:staging::hub-infrastructure::azure-native:app:ContainerApp::hub-backend

Found no pending operations associated with staging

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/*******
User           smodrak
Organizations  smodrak
Token type     personal

Dependencies:
NAME                              VERSION
@pulumi/pulumi                    3.129.0
@types/eslint__js                 8.42.3
eslint                            8.57.0
@eslint/js                        9.9.0
@pulumi/eslint-plugin             0.2.0
@typescript-eslint/eslint-plugin  8.2.0
@typescript-eslint/parser         8.2.0
eslint-config-prettier            9.1.0
eslint-plugin-prettier            5.2.1
prettier                          3.3.3
typescript                        5.5.4
@types/node                       18.19.45
@pulumi/azure-native              2.56.0
typescript-eslint                 8.2.0

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

justinvp commented 2 months ago

Hi @smith558. Thanks for opening the issue and sorry for the trouble. This looks like a possible OOM (related: https://github.com/pulumi/pulumi/issues/16596).

You may be able to workaround this by explicitly setting the heap size for Node.js in Pulumi.yaml:

runtime:
  name: nodejs
  options:
    nodeargs: --max-old-space-size=4096
justinvp commented 2 months ago

@julienp, I would expect with https://github.com/pulumi/pulumi/pull/16700 we'd detect this and offer the suggested workaround. Any ideas why that's not happening?

smith558 commented 2 months ago

@justinvp Thank you. Will try this.

julienp commented 2 months ago

Bot mistakenly closed this because a PR mentioning this got merged, but I don't think this is fixed.

smith558 commented 2 months ago

Hi @smith558. Thanks for opening the issue and sorry for the trouble. This looks like a possible OOM (related: #16596).

You may be able to workaround this by explicitly setting the heap size for Node.js in Pulumi.yaml:

runtime: name: nodejs options: nodeargs: --max-old-space-size=4096

This has resolved my issue. I will leave the GitHub issue open as you seem to have related tasks to do.

pulumi-bot commented 2 months ago

This issue has been addressed in PR #17045 and shipped in release v3.130.0.