Consistent way to handle compiled resources

alecslupu commented 1 year ago

Notice: A bug is a crash or incorrect behavior. If you have a debugging or troubleshooting question, please open a discussion on the Discussions Tab. Otherwise, remove this line and fill out the following sections.

Expected behavior:

Asset compilation should respect the RAILS environment in which the assets are being compiled.

Actual behavior:

Currently the behavior varies on multiple factors.

Running the command :

 RAILS_ENV=test bundle exec rake assets:precompile

will compile the js assets generating the following structure:

decidim_core (1.24 MiB)
      js/249-ec2e3ebada8025a94315.js
      js/1230-2d2276329353b825cfcd.js
      js/9732-67c0f69bfb7f23cf64af.js
      js/5671-e1a7c8e971fa73e54b50.js
      js/1787-63187eccb0f404616589.js
      js/2449-aa50519c7ac79df1aea9.js
      js/3784-0293ffaa7902d4e20046.js
      js/3667-7efa4b10e61b95ad879b.js
      js/4930-dd2b126045ec1c942aa9.js
      css/decidim_core-00a3663a.css
      js/decidim_core-3351b76e64ec549f7e08.js

Running

NODE_ENV=test RAILS_ENV=test bundle exec rake shakapacker:compile

will compile the js assets generating the following structure:

decidim_core (1.22 MiB)
      js/249.js
      js/1230.js
      js/9732.js
      js/5671.js
      js/1787.js
      js/2449.js
      js/3784.js
      js/3667.js
      js/4930.js
      css/decidim_core.css
      js/decidim_core.js

Running the following command

NODE_ENV=test RAILS_ENV=test bundle exec rspec spec/system/editor_spec.rb -e "with full toolbar controls image"

will generate a lot of files that follow the previous response.

Running the following command

RAILS_ENV=test bundle exec rspec spec/system/editor_spec.rb -e "with full toolbar controls image"

will generate the files having the following structure:

decidim-core_app_packs_images_sync_recursive_-decidim-core_app_packs_src_decidim_account_form-f1f256.js
decidim-core_app_packs_images_sync_recursive_-decidim-core_app_packs_src_decidim_account_form-f1f256.js.map
decidim-core_app_packs_src_decidim_autocomplete_js.js
decidim-core_app_packs_src_decidim_autocomplete_js.js.map
decidim-core_app_packs_src_decidim_direct_uploads_upload_field_js.js
decidim-core_app_packs_src_decidim_direct_uploads_upload_field_js.js.map
decidim-core_app_packs_src_decidim_form_datepicker_js-decidim-core_app_packs_src_decidim_form-55cc57.js
decidim-core_app_packs_src_decidim_form_datepicker_js-decidim-core_app_packs_src_decidim_form-55cc57.js.map
decidim-core_app_packs_src_decidim_input_emoji_js-decidim-core_app_packs_src_decidim_redesign-fe0013.js
decidim-core_app_packs_src_decidim_input_emoji_js-decidim-core_app_packs_src_decidim_redesign-fe0013.js.map
decidim-core_app_packs_src_decidim_map_controller_markers_js.js
decidim-core_app_packs_src_decidim_map_controller_markers_js.js.map
decidim-core_app_packs_src_decidim_map_js.js
decidim-core_app_packs_src_decidim_map_js.js.map
decidim-core_app_packs_src_decidim_sw_index_js.js
decidim-core_app_packs_src_decidim_sw_index_js.js.map
decidim_core.js
decidim_core.js.map

our shackapacker.yml ( generated)

---
default:
  source_path: app/packs
  source_entry_path: entrypoints
  nested_entries: true
  css_extract_ignore_order_warnings: false
  public_root_path: public
  public_output_path: decidim-packs
  cache_path: tmp/shakapacker
  webpack_compile_output: true
  extract_css: true
  shakapacker_precompile: true
  additional_paths: &1
  - node_modules
  - app/packs
  - "/path/to/project/decidim-api/app/packs"
  - "/path/to/project/decidim-core/app/packs"
  - "/path/to/project/decidim-comments/app/packs"
  - "/path/to/project/decidim-accountability/app/packs"
  - "/path/to/project/decidim-admin/app/packs"
  - "/path/to/project/decidim-assemblies/app/packs"
  - "/path/to/project/decidim-blogs/app/packs"
  - "/path/to/project/decidim-budgets/app/packs"
  - "/path/to/project/decidim-debates/app/packs"
  - "/path/to/project/decidim-forms/app/packs"
  - "/path/to/project/decidim-meetings/app/packs"
  - "/path/to/project/decidim-pages/app/packs"
  - "/path/to/project/decidim-participatory_processes/app/packs"
  - "/path/to/project/decidim-proposals/app/packs"
  - "/path/to/project/decidim-sortitions/app/packs"
  - "/path/to/project/decidim-templates/app/packs"
  - "/path/to/project/decidim-surveys/app/packs"
  - "/path/to/project/decidim-system/app/packs"
  - "/path/to/project/decidim-verifications/app/packs"
  - "/path/to/project/decidim-conferences/app/packs"
  - "/path/to/project/decidim-dev/app/packs"
  - "/path/to/project/decidim-elections/app/packs"
  - "/path/to/project/decidim-initiatives/app/packs"
  cache_manifest: false
  webpack_loader: babel
  ensure_consistent_versioning: true
  compiler_strategy: digest
  useContentHash: false
  entrypoints: &2
    decidim_api_docs: "/path/to/project/decidim-api/app/packs/entrypoints/decidim_api_docs.js"
    decidim_api_graphiql: "/path/to/project/decidim-api/app/packs/entrypoints/decidim_api_graphiql.js"
    decidim_core: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_core.js"
    decidim_sw: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_sw.js"
    redesigned_decidim_core: "/path/to/project/decidim-core/app/packs/entrypoints/redesigned_decidim_core.js"
    decidim_conference_diploma: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_conference_diploma.js"
    decidim_editor: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_editor.js"
    decidim_email: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_email.js"
    decidim_map: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_map.js"
    decidim_geocoding_provider_photon: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_geocoding_provider_photon.js"
    decidim_geocoding_provider_here: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_geocoding_provider_here.js"
    decidim_map_provider_default: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_map_provider_default.js"
    decidim_map_provider_here: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_map_provider_here.js"
    decidim_widget: "/path/to/project/decidim-core/app/packs/entrypoints/decidim_widget.js"
    decidim_comments: "/path/to/project/decidim-comments/app/packs/entrypoints/decidim_comments.js"
    decidim_accountability: "/path/to/project/decidim-accountability/app/packs/entrypoints/decidim_accountability.js"
    decidim_accountability_admin: "/path/to/project/decidim-accountability/app/packs/entrypoints/decidim_accountability_admin.js"
    decidim_accountability_admin_imports: "/path/to/project/decidim-accountability/app/packs/entrypoints/decidim_accountability_admin_imports.js"
    decidim_admin: "/path/to/project/decidim-admin/app/packs/entrypoints/decidim_admin.js"
    decidim_assemblies: "/path/to/project/decidim-assemblies/app/packs/entrypoints/decidim_assemblies.js"
    decidim_assemblies_admin: "/path/to/project/decidim-assemblies/app/packs/entrypoints/decidim_assemblies_admin.js"
    decidim_blogs: "/path/to/project/decidim-blogs/app/packs/entrypoints/decidim_blogs.js"
    decidim_budgets: "/path/to/project/decidim-budgets/app/packs/entrypoints/decidim_budgets.js"
    decidim_budgets_admin: "/path/to/project/decidim-budgets/app/packs/entrypoints/decidim_budgets_admin.js"
    decidim_debates_admin: "/path/to/project/decidim-debates/app/packs/entrypoints/decidim_debates_admin.js"
    decidim_forms: "/path/to/project/decidim-forms/app/packs/entrypoints/decidim_forms.js"
    decidim_forms_admin: "/path/to/project/decidim-forms/app/packs/entrypoints/decidim_forms_admin.js"
    decidim_questionnaire_answers_pdf: "/path/to/project/decidim-forms/app/packs/entrypoints/decidim_questionnaire_answers_pdf.js"
    decidim_meetings: "/path/to/project/decidim-meetings/app/packs/entrypoints/decidim_meetings.js"
    decidim_meetings_admin: "/path/to/project/decidim-meetings/app/packs/entrypoints/decidim_meetings_admin.js"
    decidim_pages: "/path/to/project/decidim-pages/app/packs/entrypoints/decidim_pages.js"
    decidim_participatory_processes: "/path/to/project/decidim-participatory_processes/app/packs/entrypoints/decidim_participatory_processes.js"
    decidim_participatory_processes_admin: "/path/to/project/decidim-participatory_processes/app/packs/entrypoints/decidim_participatory_processes_admin.js"
    decidim_proposals: "/path/to/project/decidim-proposals/app/packs/entrypoints/decidim_proposals.js"
    decidim_proposals_admin: "/path/to/project/decidim-proposals/app/packs/entrypoints/decidim_proposals_admin.js"
    decidim_sortitions: "/path/to/project/decidim-sortitions/app/packs/entrypoints/decidim_sortitions.js"
    decidim_templates: "/path/to/project/decidim-templates/app/packs/entrypoints/decidim_templates.js"
    decidim_templates_admin: "/path/to/project/decidim-templates/app/packs/entrypoints/decidim_templates_admin.js"
    decidim_system: "/path/to/project/decidim-system/app/packs/entrypoints/decidim_system.js"
    decidim_verifications: "/path/to/project/decidim-verifications/app/packs/entrypoints/decidim_verifications.js"
    decidim_conferences: "/path/to/project/decidim-conferences/app/packs/entrypoints/decidim_conferences.js"
    decidim_conferences_admin: "/path/to/project/decidim-conferences/app/packs/entrypoints/decidim_conferences_admin.js"
    decidim_dev: "/path/to/project/decidim-dev/app/packs/entrypoints/decidim_dev.js"
    decidim_dev_test_custom_map: "/path/to/project/decidim-dev/app/packs/entrypoints/decidim_dev_test_custom_map.js"
    decidim_elections: "/path/to/project/decidim-elections/app/packs/entrypoints/decidim_elections.js"
    decidim_elections_admin: "/path/to/project/decidim-elections/app/packs/entrypoints/decidim_elections_admin.js"
    decidim_votings: "/path/to/project/decidim-elections/app/packs/entrypoints/decidim_votings.js"
    decidim_votings_admin: "/path/to/project/decidim-elections/app/packs/entrypoints/decidim_votings_admin.js"
    decidim_initiatives: "/path/to/project/decidim-initiatives/app/packs/entrypoints/decidim_initiatives.js"
    decidim_initiatives_admin: "/path/to/project/decidim-initiatives/app/packs/entrypoints/decidim_initiatives_admin.js"
    decidim_initiatives_print: "/path/to/project/decidim-initiatives/app/packs/entrypoints/decidim_initiatives_print.js"
    decidim_initiatives_initiatives_votes: "/path/to/project/decidim-initiatives/app/packs/entrypoints/decidim_initiatives_initiatives_votes.js"
  stylesheet_imports: &3
    imports:
      app:
      - stylesheets/decidim/initiatives/initiatives
  compile: false
development:
  source_path: app/packs
  source_entry_path: entrypoints
  nested_entries: true
  css_extract_ignore_order_warnings: false
  public_root_path: public
  public_output_path: decidim-packs
  cache_path: tmp/shakapacker
  webpack_compile_output: true
  extract_css: true
  shakapacker_precompile: true
  additional_paths: *1
  cache_manifest: false
  webpack_loader: babel
  ensure_consistent_versioning: true
  compiler_strategy: mtime
  useContentHash: false
  compile: false
  dev_server:
    https: false
    host: localhost
    port: 3035
    hmr: false
    inline_css: true
    client:
      overlay: false
    compress: true
    allowed_hosts: all
    pretty: true
    headers:
      Access-Control-Allow-Origin: "*"
    static:
      watch:
        ignored: "**/node_modules/**"
  entrypoints: *2
  stylesheet_imports: *3
test:
  source_path: app/packs
  source_entry_path: entrypoints
  nested_entries: true
  css_extract_ignore_order_warnings: false
  public_root_path: public
  public_output_path: packs-test
  cache_path: tmp/shakapacker
  webpack_compile_output: true
  extract_css: true
  shakapacker_precompile: true
  additional_paths: *1
  cache_manifest: false
  webpack_loader: babel
  ensure_consistent_versioning: true
  compiler_strategy: digest
  useContentHash: false
  compile: false
  entrypoints: *2
  stylesheet_imports: *3
production:
  source_path: app/packs
  source_entry_path: entrypoints
  nested_entries: true
  css_extract_ignore_order_warnings: false
  public_root_path: public
  public_output_path: decidim-packs
  cache_path: tmp/shakapacker
  webpack_compile_output: true
  extract_css: true
  shakapacker_precompile: true
  additional_paths: *1
  cache_manifest: true
  webpack_loader: babel
  ensure_consistent_versioning: true
  compiler_strategy: digest
  useContentHash: true
  compile: false
  entrypoints: *2
  stylesheet_imports: *3

Small, reproducible repo:

https://github.com/decidim/decidim ( branch chore/upgrade-shakapacker )

Setup environment:

Ruby version: 3.1
Rails version: 6.1
Shakapacker version: 7.10

alecslupu commented 1 year ago

At decidim we pass through a major redesign process, in which we have ended up replacing the WYSIWYG from QuillJS to Tip-tap, change that led to have some specs. We also, changed our tag system from foundation-datapicker to tom-select.

While working with the dev-server we could not see any issue. We did not see any issues either when running in CI the specs, as there was no NODE_ENV variable being set. We only had "RAILS_ENV".

One of my colleagues tried to run the editor specs locally, case when observed that the specs were failing. I could observe, while investigating on my end, that my script actually was failing with:

packs-test/js/1230.js 1:12730 Uncaught SyntaxError: Invalid regular expression: /ß|€|â||„|ˆ|•|±|¥|É| "

During my investigation, i could observe that 'rake webpacker:compile' and the asset compiling during the test are behaving differently.

The Invalid regular expression error, is being caused by tom-select dependency @orchidjs/unicode-variants, and it seems that the compiler is actually replacing accent_pat from

https://github.com/orchidjs/unicode-variants/blob/1e0f5e89c843750200dfe5cbc8610cb424dae7a8/lib/index.mjs#L86

const convert_pat = new RegExp(Object.keys(latin_convert).join('|') + '|' + accent_pat, 'gu');

to

new RegExp(Object.keys(j).join("|")+"|[̀-ͯ·ʾʼ]","gu")

I could observe in my specs that i had a node_env variable that was being set, but removing it, would have no effect.

I ended up, overriding the webpackConfig variable in my config.js to always use the TerserPlugin on all the environments, as being defined in https://github.com/shakacode/shakapacker/blob/master/package/environments/production.js , which in the end fixed the above issue.

justin808 commented 1 year ago

@alecslupu if you want a custom NODE_ENV, then why not just set it before running the scripts that invoke this command?

justin808 commented 1 year ago

Closed as #366 is also closed.

shakacode / shakapacker