search

paketo-buildpacks / rails-assets

Cloud Native Buildpack for precompiling rails assets for Ruby applications.
Apache License 2.0
7 stars 4 forks source link

Missing manifest.json after precompiling Rails 6.0 app #35

Closed matiasgarcia closed 3 years ago

matiasgarcia commented 3 years ago

What happened?

We created a vanilla Rails 6 application that uses webpacker as its assets pipeline. After deploying, the error we see is:

Oops!. Something went wrong. Webpacker can't find application.js in /workspace/public/packs/manifest.json. Possible causes: 1. You want to set webpacker.yml value of compile to true for your environment unless you are using the `webpack -w` or the webpack-dev-server. 2. webpack has not yet re-run to reflect updates. 3. You have misconfigured Webpacker's config/webpacker.yml file. 4. Your webpack configuration is not creating a manifest. Your manifest contains: { }

Build Configuration

# Note: You must restart bin/webpack-dev-server for changes to take effect

default: &default
  source_path: app/javascript
  source_entry_path: packs
  public_root_path: public
  public_output_path: packs
  cache_path: tmp/cache/webpacker
  webpack_compile_output: true

  # Additional paths webpack should lookup modules
  # ['app/assets', 'engine/foo/app/assets']
  additional_paths: []

  # Reload manifest.json on all requests so we reload latest compiled packs
  cache_manifest: false

  # Extract and emit a css file
  extract_css: false

  static_assets_extensions:
    - .jpg
    - .jpeg
    - .png
    - .gif
    - .tiff
    - .ico
    - .svg
    - .eot
    - .otf
    - .ttf
    - .woff
    - .woff2

  extensions:
    - .mjs
    - .js
    - .sass
    - .scss
    - .css
    - .module.sass
    - .module.scss
    - .module.css
    - .png
    - .svg
    - .gif
    - .jpeg
    - .jpg

development:
  <<: *default
  compile: true

  # Reference: https://webpack.js.org/configuration/dev-server/
  dev_server:
    https: false
    host: localhost
    port: 3035
    public: localhost:3035
    hmr: false
    # Inline should be set to true if using HMR
    inline: true
    overlay: true
    compress: true
    disable_host_check: true
    use_local_ip: false
    quiet: false
    pretty: false
    headers:
      'Access-Control-Allow-Origin': '*'
    watch_options:
      ignored: '**/node_modules/**'

test:
  <<: *default
  compile: true

  # Compile test packs to a separate directory
  public_output_path: packs-test

production: &production
  <<: *default

  # Production depends on precompilation of packs prior to booting for performance.
  compile: false

  # Extract and emit a css file
  extract_css: true

  # Cache manifest.json for performance
  cache_manifest: true

staging:
  <<: *production

Environment variables:

RAILS_ENV: production
RAILS_LOG_TO_STDOUT: true
RAILS_SERVE_STATIC_FILES: true
SECRET_KEY_BASE: some_key_base
sophiewigmore commented 3 years ago

@matiasgarcia Hi! Did you see this error while you were building the app image, or did this occur when you tried to run the container?

matiasgarcia commented 3 years ago

@sophiewigmore this happens when running the container. I believe that for some reason the assets are not carried after precompiling them.

sophiewigmore commented 3 years ago

@matiasgarcia I'm struggling to reproduce this error on my end. I used the Rails 6.0 app with the same webpacker.yml file as you, and then ran pack build with the environment variables you mentioned.

Could you give me a bit more information about your app, and the kpack command you ran with?

I saw in this post, that many people with the same error had version mismatches between the Gemfile.lock and the yarn.lock. Are both of those files up to date?

matiasgarcia commented 3 years ago

@sophiewigmore In the Rails 6.0 app you are using webpacker 4.3.0. We are using 5.2.1.

I am still checking if there are more differences.

Here is our Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    actioncable (6.1.0)
      actionpack (= 6.1.0)
      activesupport (= 6.1.0)
      nio4r (~> 2.0)
      websocket-driver (>= 0.6.1)
    actionmailbox (6.1.0)
      actionpack (= 6.1.0)
      activejob (= 6.1.0)
      activerecord (= 6.1.0)
      activestorage (= 6.1.0)
      activesupport (= 6.1.0)
      mail (>= 2.7.1)
    actionmailer (6.1.0)
      actionpack (= 6.1.0)
      actionview (= 6.1.0)
      activejob (= 6.1.0)
      activesupport (= 6.1.0)
      mail (~> 2.5, >= 2.5.4)
      rails-dom-testing (~> 2.0)
    actionpack (6.1.0)
      actionview (= 6.1.0)
      activesupport (= 6.1.0)
      rack (~> 2.0, >= 2.0.9)
      rack-test (>= 0.6.3)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.2.0)
    actiontext (6.1.0)
      actionpack (= 6.1.0)
      activerecord (= 6.1.0)
      activestorage (= 6.1.0)
      activesupport (= 6.1.0)
      nokogiri (>= 1.8.5)
    actionview (6.1.0)
      activesupport (= 6.1.0)
      builder (~> 3.1)
      erubi (~> 1.4)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.1, >= 1.2.0)
    activeadmin (2.9.0)
      arbre (~> 1.2, >= 1.2.1)
      formtastic (>= 3.1, < 5.0)
      formtastic_i18n (~> 0.4)
      inherited_resources (~> 1.7)
      jquery-rails (~> 4.2)
      kaminari (~> 1.0, >= 1.2.1)
      railties (>= 5.2, < 6.2)
      ransack (~> 2.1, >= 2.1.1)
    activejob (6.1.0)
      activesupport (= 6.1.0)
      globalid (>= 0.3.6)
    activemodel (6.1.0)
      activesupport (= 6.1.0)
    activerecord (6.1.0)
      activemodel (= 6.1.0)
      activesupport (= 6.1.0)
    activestorage (6.1.0)
      actionpack (= 6.1.0)
      activejob (= 6.1.0)
      activerecord (= 6.1.0)
      activesupport (= 6.1.0)
      marcel (~> 0.3.1)
      mimemagic (~> 0.3.2)
    activesupport (6.1.0)
      concurrent-ruby (~> 1.0, >= 1.0.2)
      i18n (>= 1.6, < 2)
      minitest (>= 5.1)
      tzinfo (~> 2.0)
      zeitwerk (~> 2.3)
    addressable (2.7.0)
      public_suffix (>= 2.0.2, < 5.0)
    appsignal (2.11.1)
      rack
    arbre (1.4.0)
      activesupport (>= 3.0.0, < 6.2)
      ruby2_keywords (>= 0.0.2, < 1.0)
    bcrypt (3.1.16)
    bindex (0.8.1)
    bootsnap (1.5.1)
      msgpack (~> 1.0)
    builder (3.2.4)
    byebug (11.1.3)
    concurrent-ruby (1.1.7)
    connection_pool (2.2.3)
    crack (0.4.3)
      safe_yaml (~> 1.0.0)
    crass (1.0.6)
    devise (4.7.3)
      bcrypt (~> 3.0)
      orm_adapter (~> 0.1)
      railties (>= 4.1.0)
      responders
      warden (~> 1.2.3)
    diff-lcs (1.4.4)
    docile (1.3.2)
    erubi (1.10.0)
    et-orbi (1.2.4)
      tzinfo
    exception_notification (4.4.3)
      actionmailer (>= 4.0, < 7)
      activesupport (>= 4.0, < 7)
    factory_bot (6.1.0)
      activesupport (>= 5.0.0)
    factory_bot_rails (6.1.0)
      factory_bot (~> 6.1.0)
      railties (>= 5.0.0)
    faraday (1.1.0)
      multipart-post (>= 1.2, < 3)
      ruby2_keywords
    ffi (1.13.1)
    formtastic (3.1.5)
      actionpack (>= 3.2.13)
    formtastic_i18n (0.6.0)
    fugit (1.4.1)
      et-orbi (~> 1.1, >= 1.1.8)
      raabro (~> 1.4)
    globalid (0.4.2)
      activesupport (>= 4.2.0)
    has_scope (0.7.2)
      actionpack (>= 4.1)
      activesupport (>= 4.1)
    hashdiff (0.3.2)
    hashie (4.1.0)
    httparty (0.18.0)
      mime-types (~> 3.0)
      multi_xml (>= 0.5.2)
    i18n (1.8.5)
      concurrent-ruby (~> 1.0)
    inherited_resources (1.12.0)
      actionpack (>= 5.2, < 6.2)
      has_scope (~> 0.6)
      railties (>= 5.2, < 6.2)
      responders (>= 2, < 4)
    jbuilder (2.10.1)
      activesupport (>= 5.0.0)
    jquery-rails (4.4.0)
      rails-dom-testing (>= 1, < 3)
      railties (>= 4.2.0)
      thor (>= 0.14, < 2.0)
    jwt (2.2.2)
    kaminari (1.2.1)
      activesupport (>= 4.1.0)
      kaminari-actionview (= 1.2.1)
      kaminari-activerecord (= 1.2.1)
      kaminari-core (= 1.2.1)
    kaminari-actionview (1.2.1)
      actionview
      kaminari-core (= 1.2.1)
    kaminari-activerecord (1.2.1)
      activerecord
      kaminari-core (= 1.2.1)
    kaminari-core (1.2.1)
    listen (3.3.3)
      rb-fsevent (~> 0.10, >= 0.10.3)
      rb-inotify (~> 0.9, >= 0.9.10)
    loofah (2.8.0)
      crass (~> 1.0.2)
      nokogiri (>= 1.5.9)
    mail (2.7.1)
      mini_mime (>= 0.1.1)
    marcel (0.3.3)
      mimemagic (~> 0.3.2)
    method_source (1.0.0)
    mime-types (3.3.1)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2020.1104)
    mimemagic (0.3.5)
    mini_mime (1.0.2)
    mini_portile2 (2.4.0)
    minitest (5.14.2)
    msgpack (1.3.3)
    multi_json (1.15.0)
    multi_xml (0.6.0)
    multipart-post (2.1.1)
    mysql2 (0.5.3)
    nio4r (2.5.4)
    nokogiri (1.10.10)
      mini_portile2 (~> 2.4.0)
    oauth2 (1.4.4)
      faraday (>= 0.8, < 2.0)
      jwt (>= 1.0, < 3.0)
      multi_json (~> 1.3)
      multi_xml (~> 0.5)
      rack (>= 1.2, < 3)
    omniauth (1.9.1)
      hashie (>= 3.4.6)
      rack (>= 1.6.2, < 3)
    omniauth-google-oauth2 (0.8.1)
      jwt (>= 2.0)
      oauth2 (~> 1.1)
      omniauth (>= 1.1.1)
      omniauth-oauth2 (>= 1.6)
    omniauth-oauth2 (1.7.0)
      oauth2 (~> 1.4)
      omniauth (~> 1.9)
    orm_adapter (0.5.0)
    public_suffix (4.0.6)
    puma (5.1.1)
      nio4r (~> 2.0)
    raabro (1.4.0)
    rack (2.2.3)
    rack-mini-profiler (2.2.0)
      rack (>= 1.2.0)
    rack-proxy (0.6.5)
      rack
    rack-test (1.1.0)
      rack (>= 1.0, < 3)
    rails (6.1.0)
      actioncable (= 6.1.0)
      actionmailbox (= 6.1.0)
      actionmailer (= 6.1.0)
      actionpack (= 6.1.0)
      actiontext (= 6.1.0)
      actionview (= 6.1.0)
      activejob (= 6.1.0)
      activemodel (= 6.1.0)
      activerecord (= 6.1.0)
      activestorage (= 6.1.0)
      activesupport (= 6.1.0)
      bundler (>= 1.15.0)
      railties (= 6.1.0)
      sprockets-rails (>= 2.0.0)
    rails-dom-testing (2.0.3)
      activesupport (>= 4.2.0)
      nokogiri (>= 1.6)
    rails-html-sanitizer (1.3.0)
      loofah (~> 2.3)
    railties (6.1.0)
      actionpack (= 6.1.0)
      activesupport (= 6.1.0)
      method_source
      rake (>= 0.8.7)
      thor (~> 1.0)
    rake (13.0.1)
    ransack (2.4.1)
      activerecord (>= 5.2.4)
      activesupport (>= 5.2.4)
      i18n
    rb-fsevent (0.10.4)
    rb-inotify (0.10.1)
      ffi (~> 1.0)
    redis (4.2.5)
    responders (3.0.1)
      actionpack (>= 5.0)
      railties (>= 5.0)
    rspec-core (3.10.0)
      rspec-support (~> 3.10.0)
    rspec-expectations (3.10.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.10.0)
    rspec-mocks (3.10.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.10.0)
    rspec-rails (4.0.1)
      actionpack (>= 4.2)
      activesupport (>= 4.2)
      railties (>= 4.2)
      rspec-core (~> 3.9)
      rspec-expectations (~> 3.9)
      rspec-mocks (~> 3.9)
      rspec-support (~> 3.9)
    rspec-support (3.10.0)
    rspec_junit_formatter (0.4.1)
      rspec-core (>= 2, < 4, != 2.12.0)
    ruby2_keywords (0.0.2)
    safe_yaml (1.0.5)
    sass-rails (6.0.0)
      sassc-rails (~> 2.1, >= 2.1.1)
    sassc (2.4.0)
      ffi (~> 1.9)
    sassc-rails (2.1.2)
      railties (>= 4.0.0)
      sassc (>= 2.0)
      sprockets (> 3.0)
      sprockets-rails
      tilt
    semantic_range (2.3.1)
    settingslogic (2.0.9)
    shoulda-matchers (4.4.1)
      activesupport (>= 4.2.0)
    sidekiq (6.1.2)
      connection_pool (>= 2.2.2)
      rack (~> 2.0)
      redis (>= 4.2.0)
    sidekiq-cron (1.2.0)
      fugit (~> 1.1)
      sidekiq (>= 4.2.1)
    simplecov (0.19.1)
      docile (~> 1.1)
      simplecov-html (~> 0.11)
    simplecov-html (0.12.3)
    spring (2.1.1)
    sprockets (4.0.2)
      concurrent-ruby (~> 1.0)
      rack (> 1, < 3)
    sprockets-rails (3.2.2)
      actionpack (>= 4.0)
      activesupport (>= 4.0)
      sprockets (>= 3.0.0)
    thor (1.0.1)
    tilt (2.0.10)
    turbolinks (5.2.1)
      turbolinks-source (~> 5.2)
    turbolinks-source (5.2.0)
    tzinfo (2.0.3)
      concurrent-ruby (~> 1.0)
    warden (1.2.9)
      rack (>= 2.0.9)
    web-console (4.1.0)
      actionview (>= 6.0.0)
      activemodel (>= 6.0.0)
      bindex (>= 0.4.0)
      railties (>= 6.0.0)
    webmock (3.0.1)
      addressable (>= 2.3.6)
      crack (>= 0.3.2)
      hashdiff
    webpacker (5.2.1)
      activesupport (>= 5.2)
      rack-proxy (>= 0.6.1)
      railties (>= 5.2)
      semantic_range (>= 2.3.0)
    websocket-driver (0.7.3)
      websocket-extensions (>= 0.1.0)
    websocket-extensions (0.1.5)
    zeitwerk (2.4.2)

PLATFORMS
  ruby

DEPENDENCIES
  activeadmin
  appsignal (~> 2.11)
  bootsnap (>= 1.4.4)
  byebug
  devise
  exception_notification
  factory_bot_rails
  httparty (= 0.18.0)
  jbuilder (~> 2.7)
  listen (~> 3.3)
  mysql2 (~> 0.5)
  omniauth-google-oauth2
  puma (~> 5.0)
  rack-mini-profiler (~> 2.0)
  rails (~> 6.1.0)
  rspec-rails (~> 4.0.1)
  rspec_junit_formatter
  sass-rails (>= 6)
  settingslogic
  shoulda-matchers (~> 4.4)
  sidekiq (~> 6.1)
  sidekiq-cron (~> 1.2)
  simplecov
  spring
  turbolinks (~> 5)
  tzinfo-data
  web-console (>= 4.1.0)
  webmock
  webpacker (~> 5.0)

RUBY VERSION
   ruby 2.5.7p206

BUNDLED WITH
   1.17.3

yarn.lock webpacker

"@rails/webpacker@5.2.1":
  version "5.2.1"
  resolved "https://registry.yarnpkg.com/@rails/webpacker/-/webpacker-5.2.1.tgz#87cdbd4af2090ae2d74bdc51f6f04717d907c5b3"
  integrity sha512-rO0kOv0o4ESB8ZnKX+b54ZKogNJGWSMULGmsJacREfm9SahKEQwXBeHNsqSGtS9NAPsU6YUFhGKRd4i/kbMNrQ==
  dependencies:
    "@babel/core" "^7.11.1"
    "@babel/plugin-proposal-class-properties" "^7.10.4"
    "@babel/plugin-proposal-object-rest-spread" "^7.10.1"
    "@babel/plugin-syntax-dynamic-import" "^7.8.3"
    "@babel/plugin-transform-destructuring" "^7.10.1"
    "@babel/plugin-transform-regenerator" "^7.10.1"
    "@babel/plugin-transform-runtime" "^7.11.0"
    "@babel/preset-env" "^7.11.0"
    "@babel/runtime" "^7.11.2"
    babel-loader "^8.1.0"
    babel-plugin-dynamic-import-node "^2.3.3"
    babel-plugin-macros "^2.8.0"
    case-sensitive-paths-webpack-plugin "^2.3.0"
    compression-webpack-plugin "^4.0.0"
    core-js "^3.6.5"
    css-loader "^3.5.3"
    file-loader "^6.0.0"
    flatted "^3.0.4"
    glob "^7.1.6"
    js-yaml "^3.14.0"
    mini-css-extract-plugin "^0.9.0"
    node-sass "^4.14.1"
    optimize-css-assets-webpack-plugin "^5.0.3"
    path-complete-extname "^1.0.0"
    pnp-webpack-plugin "^1.6.4"
    postcss-flexbugs-fixes "^4.2.1"
    postcss-import "^12.0.1"
    postcss-loader "^3.0.0"
    postcss-preset-env "^6.7.0"
    postcss-safe-parser "^4.0.2"
    regenerator-runtime "^0.13.7"
    sass-loader "^8.0.2"
    style-loader "^1.2.1"
    terser-webpack-plugin "^4.0.0"
    webpack "^4.44.1"
    webpack-assets-manifest "^3.1.1"
    webpack-cli "^3.3.12"
    webpack-sources "^1.4.3"

As you see, we are using webpacker 5.2.1

matiasgarcia commented 3 years ago

package.json

{
  "name": "withdrawals",
  "private": true,
  "dependencies": {
    "@activeadmin/activeadmin": "^2.9.0",
    "@rails/actioncable": "^6.0.0",
    "@rails/activestorage": "^6.0.0",
    "@rails/ujs": "^6.0.0",
    "@rails/webpacker": "5.2.1",
    "turbolinks": "^5.2.0"
  },
  "version": "0.1.0",
  "devDependencies": {
    "webpack-dev-server": "^3.11.0"
  }
}
sophiewigmore commented 3 years ago

Even when I modify the app to use webpacker 5.2.1 I don't see this issue. I'll continue to look into how I can reproduce this error so we can pinpoint what's going on. If you're able to provide an app that you're using that could be helpful

genevieve commented 3 years ago

Hey @matiasgarcia. Not sure if it will resolve it for you, but what fixed the issue for us was to have the webpack-cli added to our package.json in the dependencies group (it used to be in the devDependencies group.) Keep in mind, this is what solved our issue on an application with an older webpacker (v4). cc @josh-lauer

I'm actually not sure I would recommend adding the webpack-cli if it's not already included in your application. I think this worked for our application because it's much older and configured very differently from our newer rails applications. Since webpacker is supposed to be included in the rails asset pipeline generation, its assets (like public/packs and manifest.json) should be covered by the behavior of this buildpack which simply runs bundle exec rails assets:precompile and shouldn't require a separate command.

genevieve commented 3 years ago

We found this issue helpful: https://github.com/rails/webpacker/issues/1561#issuecomment-395672374

matiasgarcia commented 3 years ago

@genevieve @sophiewigmore We found something interesting.

Let me do a quick recap:

We added a new environment called staging, both in webpacker.yml and config/environments.

config/environments/staging.rb

require_relative './production'

config/webpacker.yml

...

production: &production
  <<: *default

  # Production depends on precompilation of packs prior to booting for performance.
  compile: false

  # Extract and emit a css file
  extract_css: true

  # Cache manifest.json for performance
  cache_manifest: true

staging:
  <<: *production

So we initially initialized our app with RAILS_ENV=staging. We noticed that this was causing the webpacker error.

So we rolled back one of the changes, we removed the staging environment from the webpacker.yml so it looks like a vanilla Rails 6 app. We kept the config/environments/staging.rb file.

We deployed again. We noticed the cache didn't clean up, so we forced a rebuild by modifying one of the files under app/javascripts. That didn't work, which made me create https://github.com/paketo-buildpacks/rails-assets/issues/38. So we modified a stylesheet in app/assets. That caused a precompile again. Now the app was working but we switched back to RAILS_ENV=production.

However, every time we deploy again and the cache is used, the manifest is missing, causing the error I mentioned above.

We are still investigating this issue, but we noticed that the buildpack forces two environment variables

https://github.com/paketo-buildpacks/rails-assets/blob/main/build.go#L95-L96

        assetsLayer.LaunchEnv.Default("RAILS_ENV", "production")
        assetsLayer.LaunchEnv.Default("RAILS_SERVE_STATIC_FILES", "true")

Which makes us think that when forcing RAILS_ENV=staging it goes crazy.

I will investigate the issue nonetheless and come back to you with a log when deploying using a cache and non cached.

matiasgarcia commented 3 years ago

Tagging @bonzofenix to help debug this issue.

ryanmoran commented 3 years ago

After doing some investigation, I'm convinced that we've missed the inclusion of public/packs in the assets layer. Since the manifest.json exists in that subdirectory, it makes sense why we would be seeing this error. It also explains why we would only see it on a rebuild, where the cache isn't restoring the manifest. I'll pick this up with a PR at some point tomorrow as its the end of my day here, but I wanted to update here with what I've been able to find so that we reported that progress is being made.

sophiewigmore commented 3 years ago

Hey @matiasgarcia we've cut release v0.2.0 of the rails-assets buildpack, and released Ruby buildpack v0.5.0 with the changes @ryanmoran put in for this issue and #38. If you try out the new version, let us know if this problem is resolved for you. Feel free to reopen this issue if you run into anything