search

shakacode / shakapacker

Use Webpack to manage app-like JavaScript modules in Rails
MIT License
430 stars 95 forks source link

Configuration issues when running in pipeline #409

Closed alecslupu closed 10 months ago

alecslupu commented 10 months ago

When running in pipeline, Shakapacker is sometimes failing to compile the assets, due to error.

Expected behavior:

Not to fail occasionally

Actual behavior:

Failing with the following message / stack trace, but when the pipeline is being retried, is passing without any issues. There is no config change between retries. 

NoMethodError:
  undefined method `[]' for nil:NilClass

        symbolized_config = config[env].deep_symbolize_keys
                                  ^^^^^
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/shakapacker-7.1.0/lib/shakapacker/configuration.rb:130:in `load'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/shakapacker-7.1.0/lib/shakapacker/configuration.rb:121:in `data'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/shakapacker-7.1.0/lib/shakapacker/configuration.rb:106:in `fetch'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/shakapacker-7.1.0/lib/shakapacker/configuration.rb:34:in `ensure_consistent_versioning?'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/shakapacker-7.1.0/lib/shakapacker/version_checker.rb:30:in `raise_if_gem_and_node_package_versions_differ'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/shakapacker-7.1.0/lib/shakapacker/railtie.rb:13:in `block in <class:Engine>'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/railties-6.1.7.6/lib/rails/initializable.rb:32:in `instance_exec'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/railties-6.1.7.6/lib/rails/initializable.rb:32:in `run'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/railties-6.1.7.6/lib/rails/initializable.rb:61:in `block in run_initializers'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/railties-6.1.7.6/lib/rails/initializable.rb:60:in `run_initializers'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/railties-6.1.7.6/lib/rails/application.rb:391:in `initialize!'
# /home/runner/work/decidim/decidim/spec/decidim_dummy_app/config/environment.rb:5:in `<top (required)>'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:332:in `require'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:332:in `block in require'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:299:in `load_dependency'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:332:in `require'
# /home/runner/work/decidim/decidim/decidim-dev/lib/decidim/dev/test/base_spec_helper.rb:21:in `<top (required)>'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:332:in `require'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:332:in `block in require'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:299:in `load_dependency'
# /home/runner/work/decidim/decidim/vendor/bundle/ruby/3.1.0/gems/activesupport-6.1.7.6/lib/active_support/dependencies.rb:332:in `require'
# ./spec/spec_helper.rb:9:in `<top (required)>'
# ./spec/system/admin/admin_edits_proposal_spec.rb:3:in `require'
# ./spec/system/admin/admin_edits_proposal_spec.rb:3:in `<top (required)>'

Env variables:

   env:
    DATABASE_USERNAME: postgres
    DATABASE_PASSWORD: postgres
    DATABASE_HOST: localhost
    RUBYOPT: -W:no-deprecated
    VALIDATOR_HTML_URI: http://localhost:8888/
    ELECTIONS_BULLETIN_BOARD_SERVER: http://localhost:8000/api
    RUBY_VERSION: 3.1.1
    DECIDIM_MODULE: decidim-proposals
    CODECOV_TOKEN: bc15b944-6b42-420a-b3f9-a5a8fb214326
    DECIDIM_BULLET_ENABLED: true
    DECIDIM_BULLET_N_PLUS_ONE: false
    DECIDIM_BULLET_COUNTER_CACHE: true
    DECIDIM_BULLET_UNUSED_EAGER: false
    DISPLAY: :99
    CI: true
    SIMPLECOV: true
    PARALLEL_TEST_PROCESSORS: 2
    SHAKAPACKER_RUNTIME_COMPILE: false
    NODE_ENV: test

Small, reproducible repo:

It happens occasionally, so I do not have reproducible steps.

Setup environment:

ahangarha commented 10 months ago

It happens occasionally, so I do not have reproducible steps.

This is strange, and we have never faced such issues. Hard to say anything unless we can reproduce the issue.

Please add further details as soon as you get any.

alecslupu commented 10 months ago

I am trying to see if i can replicate it locally. The pipeline output is : failing passing

G-Rath commented 10 months ago

So config is being set by loading YAML:

config = begin
  YAML.load_file(config_path.to_s, aliases: true)
rescue ArgumentError
  YAML.load_file(config_path.to_s)
end
symbolized_config = config[env].deep_symbolize_keys

The only way I can see config being nil is if the underlying file is empty because contrary to its documentation YAML.load_file looks to return nil by default if the file is empty:

  ###
  # Loads the document contained in +filename+.  Returns the yaml contained in
  # +filename+ as a Ruby object, or if the file is empty, it returns
  # the specified +fallback+ return value, which defaults to +false+.
  # See load for options.
  def self.load_file filename, **kwargs
    File.open(filename, 'r:bom|utf-8') { |f|
      self.load f, filename: filename, **kwargs
    }
  end

  def self.load yaml, permitted_classes: [Symbol], permitted_symbols: [], aliases: false, filename: nil, fallback: nil, symbolize_names: false, freeze: false, strict_integer: false
    safe_load # ...
  end

  def self.safe_load yaml, permitted_classes: [], permitted_symbols: [], aliases: false, filename: nil, fallback: nil, symbolize_names: false, freeze: false, strict_integer: false
    result = parse(yaml, filename: filename)
    return fallback unless result

   # ...
end

(Note how fallback: nil - hence, unless I've missed something, it'll return nil instead of false)

I'm not familiar with decidim but doing a quick search of the codebase for Shakapacker it does look like you're potentially overriding parts of Shakapacker specifically related to config generation - if those are relevant, I'd be checking you don't have a race condition in your config generation that could be resulting in an empty config file.

(on an aside, it looks like decidim is using NPM rather than Yarn - I'd be really grateful if you could check out https://github.com/shakacode/shakapacker/pull/349 if you've not already, as your codebase seems like it would be a really good smoketest before we make it the default behaviour in the upcoming major!)

alecslupu commented 10 months ago

@G-Rath , indeed, we are overriding shakapacker config, so that we generate a file like the one below: Also, the configuration file is being loaded from here:

Indeed, the issue that we are facing may or may not be related to shakapacker itself. due to size of decidim dest suite we use both matrix and parallels gem to run 2 suites at once. Thanks for pointing me to the race condition issue.

On decidim, we are using since like beginning the NPM, and now we already work on moving the shakapacker to use esbuild instead of babel.

I have noticed the package_json gem while checkingh #349, and I really think that we could use that in our code base as well. I personally got hit by the struggle to manually add to Package.json various node packages so that i make other decidim modules work. 

---
default:
  source_path: app/packs
  source_entry_path: entrypoints
  nested_entries: true
  css_extract_ignore_order_warnings: false
  public_root_path: public
  public_output_path: decidim-packs
  cache_path: tmp/shakapacker
  webpack_compile_output: true
  extract_css: true
  shakapacker_precompile: true
  additional_paths: &1
  - node_modules
  - app/packs
  - "/home/alecslupu/Sites/decidim/redesign/decidim-api/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-comments/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-accountability/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-admin/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-assemblies/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-blogs/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-budgets/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-debates/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-forms/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-meetings/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-pages/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-participatory_processes/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-proposals/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-sortitions/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-templates/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-surveys/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-system/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-verifications/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-conferences/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-design/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-dev/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-elections/app/packs"
  - "/home/alecslupu/Sites/decidim/redesign/decidim-initiatives/app/packs"
  cache_manifest: false
  webpack_loader: babel
  ensure_consistent_versioning: true
  compiler_strategy: digest
  useContentHash: false
  entrypoints: &2
    decidim_api_docs: "/home/alecslupu/Sites/decidim/redesign/decidim-api/app/packs/entrypoints/decidim_api_docs.js"
    decidim_api_graphiql: "/home/alecslupu/Sites/decidim/redesign/decidim-api/app/packs/entrypoints/decidim_api_graphiql.js"
    decidim_core: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_core.js"
    decidim_sw: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_sw.js"
    decidim_conference_diploma: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_conference_diploma.js"
    decidim_editor: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_editor.js"
    decidim_email: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_email.js"
    decidim_map: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_map.js"
    decidim_geocoding_provider_photon: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_geocoding_provider_photon.js"
    decidim_geocoding_provider_here: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_geocoding_provider_here.js"
    decidim_map_provider_default: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_map_provider_default.js"
    decidim_map_provider_here: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_map_provider_here.js"
    decidim_widget: "/home/alecslupu/Sites/decidim/redesign/decidim-core/app/packs/entrypoints/decidim_widget.js"
    decidim_comments: "/home/alecslupu/Sites/decidim/redesign/decidim-comments/app/packs/entrypoints/decidim_comments.js"
    decidim_accountability: "/home/alecslupu/Sites/decidim/redesign/decidim-accountability/app/packs/entrypoints/decidim_accountability.js"
    decidim_accountability_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-accountability/app/packs/entrypoints/decidim_accountability_admin.js"
    decidim_accountability_admin_imports: "/home/alecslupu/Sites/decidim/redesign/decidim-accountability/app/packs/entrypoints/decidim_accountability_admin_imports.js"
    decidim_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-admin/app/packs/entrypoints/decidim_admin.js"
    decidim_assemblies: "/home/alecslupu/Sites/decidim/redesign/decidim-assemblies/app/packs/entrypoints/decidim_assemblies.js"
    decidim_assemblies_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-assemblies/app/packs/entrypoints/decidim_assemblies_admin.js"
    decidim_blogs: "/home/alecslupu/Sites/decidim/redesign/decidim-blogs/app/packs/entrypoints/decidim_blogs.js"
    decidim_budgets: "/home/alecslupu/Sites/decidim/redesign/decidim-budgets/app/packs/entrypoints/decidim_budgets.js"
    decidim_budgets_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-budgets/app/packs/entrypoints/decidim_budgets_admin.js"
    decidim_debates_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-debates/app/packs/entrypoints/decidim_debates_admin.js"
    decidim_forms: "/home/alecslupu/Sites/decidim/redesign/decidim-forms/app/packs/entrypoints/decidim_forms.js"
    decidim_forms_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-forms/app/packs/entrypoints/decidim_forms_admin.js"
    decidim_questionnaire_answers_pdf: "/home/alecslupu/Sites/decidim/redesign/decidim-forms/app/packs/entrypoints/decidim_questionnaire_answers_pdf.js"
    decidim_meetings: "/home/alecslupu/Sites/decidim/redesign/decidim-meetings/app/packs/entrypoints/decidim_meetings.js"
    decidim_meetings_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-meetings/app/packs/entrypoints/decidim_meetings_admin.js"
    decidim_pages: "/home/alecslupu/Sites/decidim/redesign/decidim-pages/app/packs/entrypoints/decidim_pages.js"
    decidim_participatory_processes: "/home/alecslupu/Sites/decidim/redesign/decidim-participatory_processes/app/packs/entrypoints/decidim_participatory_processes.js"
    decidim_participatory_processes_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-participatory_processes/app/packs/entrypoints/decidim_participatory_processes_admin.js"
    decidim_proposals: "/home/alecslupu/Sites/decidim/redesign/decidim-proposals/app/packs/entrypoints/decidim_proposals.js"
    decidim_proposals_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-proposals/app/packs/entrypoints/decidim_proposals_admin.js"
    decidim_sortitions: "/home/alecslupu/Sites/decidim/redesign/decidim-sortitions/app/packs/entrypoints/decidim_sortitions.js"
    decidim_templates: "/home/alecslupu/Sites/decidim/redesign/decidim-templates/app/packs/entrypoints/decidim_templates.js"
    decidim_templates_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-templates/app/packs/entrypoints/decidim_templates_admin.js"
    decidim_system: "/home/alecslupu/Sites/decidim/redesign/decidim-system/app/packs/entrypoints/decidim_system.js"
    decidim_verifications: "/home/alecslupu/Sites/decidim/redesign/decidim-verifications/app/packs/entrypoints/decidim_verifications.js"
    decidim_conferences: "/home/alecslupu/Sites/decidim/redesign/decidim-conferences/app/packs/entrypoints/decidim_conferences.js"
    decidim_conferences_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-conferences/app/packs/entrypoints/decidim_conferences_admin.js"
    decidim_design: "/home/alecslupu/Sites/decidim/redesign/decidim-design/app/packs/entrypoints/decidim_design.js"
    decidim_dev: "/home/alecslupu/Sites/decidim/redesign/decidim-dev/app/packs/entrypoints/decidim_dev.js"
    decidim_dev_test_custom_map: "/home/alecslupu/Sites/decidim/redesign/decidim-dev/app/packs/entrypoints/decidim_dev_test_custom_map.js"
    decidim_elections: "/home/alecslupu/Sites/decidim/redesign/decidim-elections/app/packs/entrypoints/decidim_elections.js"
    decidim_elections_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-elections/app/packs/entrypoints/decidim_elections_admin.js"
    decidim_votings: "/home/alecslupu/Sites/decidim/redesign/decidim-elections/app/packs/entrypoints/decidim_votings.js"
    decidim_votings_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-elections/app/packs/entrypoints/decidim_votings_admin.js"
    decidim_initiatives: "/home/alecslupu/Sites/decidim/redesign/decidim-initiatives/app/packs/entrypoints/decidim_initiatives.js"
    decidim_initiatives_admin: "/home/alecslupu/Sites/decidim/redesign/decidim-initiatives/app/packs/entrypoints/decidim_initiatives_admin.js"
    decidim_initiatives_print: "/home/alecslupu/Sites/decidim/redesign/decidim-initiatives/app/packs/entrypoints/decidim_initiatives_print.js"
    decidim_initiatives_initiatives_votes: "/home/alecslupu/Sites/decidim/redesign/decidim-initiatives/app/packs/entrypoints/decidim_initiatives_initiatives_votes.js"
  stylesheet_imports: &3 {}
  compile: false
development:
  source_path: app/packs
  source_entry_path: entrypoints
  nested_entries: true
  css_extract_ignore_order_warnings: false
  public_root_path: public
  public_output_path: decidim-packs
  cache_path: tmp/shakapacker
  webpack_compile_output: true
  extract_css: true
  shakapacker_precompile: true
  additional_paths: *1
  cache_manifest: false
  webpack_loader: babel
  ensure_consistent_versioning: true
  compiler_strategy: mtime
  useContentHash: false
  compile: false
  dev_server:
    https: false
    host: localhost
    port: 3035
    hmr: false
    inline_css: true
    client:
      overlay: false
    compress: true
    allowed_hosts: all
    pretty: true
    headers:
      Access-Control-Allow-Origin: "*"
    static:
      watch:
        ignored: "**/node_modules/**"
  entrypoints: *2
  stylesheet_imports: *3
test:
  source_path: app/packs
  source_entry_path: entrypoints
  nested_entries: true
  css_extract_ignore_order_warnings: false
  public_root_path: public
  public_output_path: packs-test
  cache_path: tmp/shakapacker
  webpack_compile_output: true
  extract_css: true
  shakapacker_precompile: true
  additional_paths: *1
  cache_manifest: false
  webpack_loader: babel
  ensure_consistent_versioning: true
  compiler_strategy: digest
  useContentHash: false
  compile: false
  entrypoints: *2
  stylesheet_imports: *3
production:
  source_path: app/packs
  source_entry_path: entrypoints
  nested_entries: true
  css_extract_ignore_order_warnings: false
  public_root_path: public
  public_output_path: decidim-packs
  cache_path: tmp/shakapacker
  webpack_compile_output: true
  extract_css: true
  shakapacker_precompile: true
  additional_paths: *1
  cache_manifest: true
  webpack_loader: babel
  ensure_consistent_versioning: true
  compiler_strategy: digest
  useContentHash: true
  compile: false
  entrypoints: *2
  stylesheet_imports: *3
justin808 commented 10 months ago

From what I can tell, this is a "question" and not a "bug."

alecslupu commented 10 months ago

@G-Rath , @ahangarha and @justin808, thanks for your input. Yes, now after clarification, it looks like a question, and based on your input, is not even related to shakapacker. Thanks for input!