search

KnapsackPro / knapsack_pro-ruby

Knapsack Pro gem splits tests across parallel CI nodes and makes sure that tests run in optimal time
https://knapsackpro.com
MIT License
131 stars 27 forks source link

log threads unable to give the backtrace #270

Open devinburnette opened 1 week ago

devinburnette commented 1 week ago

I think there's a bug in the log_threads method introduced here where it will blow up if the backtrace for a thread is nil for whatever reason.

I got this stacktrace on a test run:

NoMethodError: undefined method `join' for nil:NilClass
/home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/knapsack_pro-7.7.0/lib/knapsack_pro/runners/queue/base_runner.rb:83:in `block in log_threads': undefined method `join' for nil:NilClass (NoMethodError)
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/knapsack_pro-7.7.0/lib/knapsack_pro/runners/queue/base_runner.rb:75:in `each'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/knapsack_pro-7.7.0/lib/knapsack_pro/runners/queue/base_runner.rb:75:in `log_threads'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/knapsack_pro-7.7.0/lib/knapsack_pro/runners/queue/base_runner.rb:62:in `block (2 levels) in trap_signals'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/type/hash_lookup_type_map.rb:44:in `key?'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/type/hash_lookup_type_map.rb:44:in `key?'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/postgresql_adapter.rb:702:in `get_oid_type'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:60:in `block (2 levels) in exec_query'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:57:in `each'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:57:in `each_with_index'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:57:in `block in exec_query'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/postgresql_adapter.rb:750:in `execute_and_clear'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:54:in `exec_query'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/database_statements.rb:560:in `select'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `select_all'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/abstract/query_cache.rb:110:in `select_all'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/querying.rb:54:in `_query_by_sql'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:942:in `block in exec_main_query'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:928:in `exec_main_query'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:914:in `block in exec_queries'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:908:in `exec_queries'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/association_relation.rb:44:in `exec_queries'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:695:in `load'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:250:in `records'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/relation.rb:245:in `to_ary'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/associations/association.rb:224:in `find_target'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/associations/collection_association.rb:264:in `load_target'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.4/lib/active_record/associations/collection_proxy.rb:43:in `load_target'
    from /home/betterment/retail/investing/vendor/bundle/ruby/3.2.0/gems/ac
    ...

Build ID: a525cf31-784f-44f0-baa8-f881e0fa4715

ArturT commented 1 week ago

@devinburnette Thanks for reporting this.

Could you work around the issue while we look into it? Or is this preventing you from using Knapsack Pro?

devinburnette commented 1 week ago

No it doesn't block usage. From what I can tell, it just prevents us from getting the backtrace to determine why a build is hanging.

devinburnette commented 1 week ago

Another data point here, I saw another couple of these today and it looked like these were the result of cancelled builds, so certain trapped signals may not result in backtraces which makes sense.

ArturT commented 1 week ago

This makes sense. Thank you for clarifying it.

devinburnette commented 2 days ago

more data on this.. I think it's possible that because of this error, RSpec does not catch the signal and therefore other downstream things that rely on RSpec receiving the signal and going into RSpec.world.wants_to_quit do not happen.