How to use this gem from within rails engines?

fiedl commented 7 years ago

I'm not sure if this is the right approach and maybe I've not understood this new part of the pipeline, yet, but:

Suppose, the major part of the app code lives inside a rails engine including all javascripts. And there are several main apps using that engine. The main apps only contain some layout changes and some minor patches.

Therefore, in order to integrate the webpacker gem into this setup, I'm trying to do the heavy lifting inside the engine, i.e. keep the work that has to be done inside all the main apps as little as possible.

How would I do that? Any pointers or suggestions are appreciated.

I'm not sure if this is a duplicate of https://github.com/rails/webpacker/issues/21. But in any case, I'd like to conclude this issue with a step-by-step guide how to approach this, for others facing the same use case.

What to do in the engine

Include webpacker in the *.gemspec file.
require 'webpacker' in the lib/foo/engine.rb.
...

What to do in each main app

bundle install
Include ./bin/webpack-dev-server in the Procfile if using Foreman.
...

gauravtiwari commented 7 years ago

@fiedl Out-of-the-box webpacker doesn't support engines yet. There are quite a few moving pieces that needs to be considered for this setup to work properly and usually people will have different use cases. I started doing some work on this but it's going to take time.

Feel free to leave any ideas/suggestions 👍

fiedl commented 7 years ago

@gauravtiwari Yes, after diving into this, I see your point. One would have to decide for each component where it belongs---to the engine, the main app, or both---and tie the pieces together in the right manner.

Some thoughts on this:

The bin/webpack-dev-server is needed in the main app. One could add it to the Procfile of the main app to start the dev server with Foreman. Is there already a way to have the webpack dev server started by Pow?
But one would need the ability to add a package from the engine.
The app/javascript files have to be loaded from both, the engine and the main app. The main app files need to have the final say. I guess if there are several engines, the files from all engines need to be loaded.
There are some files like .babelrc, .postcssrc.yml that would seem right in the main app root folder; but I'm not sure what these files are doing, yet.
The config/webpack files appear to be tricky. In my use case, I would like to minimise the work to be done in the main app. But for other use cases, I imagine, the main app would need more control over the configuration. I really like how initializers work. I have most of them in the engine, but those of the main app have the last say in case of a conflict. Maybe we could utilise this mechanism: Would it make sense to tell webpacker in an initializer where to look for the webpack config? If no initializer is present, just use config/webpack from the rails root folder.
For package.json and yarn.lock, one would need a mechanism similar to what is used with the *.gemspec, Gemfile and Gemfile.lock. If the engine decides that a certain package version is required, the package will be updated together with the engine when running bundle update my_engine from the main app. But the last say regarding the package versions has the main app as they are locked in the lock file. After all, the yarn.lock looks like a Gemfile.lock.
This leads to the question how the upgrade process would look like from the view of the main app. I guess, just running bundle install from the main app would not cover any javascript package updates from the engine.

himdel commented 7 years ago

@gauravtiwari I started adding webpacker support to ManageIQ, where the UI is just an engine (so, app root != UI root), and we may need to support reading assets from multiple engines, and we need to output them to the rails root folder, not engine root.

To achieve that, I needed to override most of the webpacker methods to use the engine root instead of hardcoding Rails.root, I also needed to add a rake task that outputs Rails.root and call that from webpack config (didn't find another way of providing that path to webpack). (This also means that we need to include the initializer that overrides those paths from our Rakefile, which is a bit unfortunate :).)

.. And I still haven't figured out what would be needed to be able to call webpacker:compile, so far it it works if you manually call Webpacker.bootstrap but only when called from the root app, not when called from the engine.

I still haven't finished work on the "read assets from multiple engines" bit, but so far, I have a rake task that outputs a json of all the engines and their root paths (and filters those engines by existence of /app/javascript, until there's a proper way to register this), and intending to read that in the webpack config.

So.. I think at least these changes would be needed:

don't assume all paths are relative to the current directory
don't hardcode Rails.root anywhere
come up with a sane way of passing those engine paths all the way to JS

If you're interested in the changes that we needed (and feel free to criticize and/or make suggestions), https://github.com/ManageIQ/manageiq-ui-classic/pull/1132 .

Furthermore (and these are only my personal opinion):

we should default to expecting the assets compiled, UI devels can enable webpack-dev-server, and for the rest, it should Just Work as long as they don't make UI changes.
we should never deal with yarn, yarn and webpack are separate tools, and webpacker's job starts when you have all the assets, not before
all those safety checks (compile depends on verify_install, verify_install depends on check_node and check_yarn, ..) are out of scope and needlessly complicate these tasks. All we need is to see the real error when it fails.

EDIT: Oh, and if there was a nice way of disabling the assets:precompile hook, it would be lovely, when wrapping that task, you may want the other taks to run on precompile :).

EDIT2: Aand making all the compile-time yarn deps as --dev, so that non-dev is reserved for actual UI dependencies, but I guess that's a separate issue :)

zacksiri commented 7 years ago

Right now i'm doing this, but obviously the solution is quite incomplete. Here is what I'm doing, what I see as the potential solution

by default wepbacker will always look for the manifest in the /public/packs folder of the Root application, that's because the default webpacker.yml is configured as such. However when coming from an engine environment it should respect the setting of the engine's webpacker.yml as well. which means webpacker needs to be aware of multiple namespaces.

For webpacker to be successful in an engine environment we need things to be isolated like every other part.

Which means, when we install the engine into an app, it needs to have an 'install step' that basically installs the pre-built modules into the rails app /public/packs/engine/name/manifest.json

We will probably need to modify the stylesheet_pack_tag and javascript_pack_tag helper to be aware of namespaces. So basically if you want to use the engine's assets you should be able to do something like javascript_pack_tag 'engine/name/vendor' etc... or javascript_pack_tag 'vendor', module: 'Engine::Name' which resolves to /public/packs/engine/name/vendor.js

I will have more clarity on this as I develop the solution. right now for me developing the engine with JS code I am just using a symlink from the dummy app's public/packs folder to the engine's public/packs folder, which works for development purposes.

A question for @dhh is also that should the Root app using the engine have to worry about compiling an engine's assets? If not it should make thing simpler since we would expect the JS code from the engine to already be compiled and usable which would mean we just need to make the javascript_pack_tag and stylesheet_pack_tag aware of those assets based on namespaces. or the module: 'Engine::Name' option that gets passed in.

ohadlevy commented 7 years ago

I've made a POC using webpack and rails engines that I believe that can be reused to solve this request: see https://github.com/ohadlevy/foreman/commit/2afa796552c3331a1f22df7143ef6f9def733b67 for a reference.

soundasleep commented 7 years ago

I would love to see some progress on this feature - this is preventing me from using webpacker in my newest project.

dhh commented 7 years ago

Please Do Investigate 👍

On Jul 19, 2017, at 23:03, Jevon Wright notifications@github.com wrote:

I would love to see some progress on this feature - this is preventing me from using webpacker in my newest project.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

ohadlevy commented 7 years ago

we have been using webpack successfully (without webpacker) with rails engines on theforeman/foreman project. they key was to create a simple ruby script that loads all engines paths, and pass that output as json to webpack, which in turn simply add those path to its lookup paths

dhh commented 7 years ago

Sounds like something we should be able to automate 👍

On Thu, Jul 20, 2017 at 7:01 AM, Ohad Levy notifications@github.com wrote:

we have been using webpack successfully (without webpacker) with rails engines on theforeman/foreman https://github.com/theforeman/foreman project. they key was to create a simple ruby script https://github.com/theforeman/foreman/blob/develop/script/plugin_webpack_directories.rb that loads all engines paths, and pass that output as json to webpack, which in turn simply add those path to its lookup paths https://github.com/theforeman/foreman/blob/develop/config/webpack.config.js#L30

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rails/webpacker/issues/348#issuecomment-316682127, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAKtcL8JHDO8sv-eFyRzPOOtBxPNcmoks5sP0GAgaJpZM4NUgTi .

mltsy commented 7 years ago

Should the Root app using the engine have to worry about compiling an engine's assets?

On the surface, it's a fair argument that an asset engine (which is essentially what a webpack app gem is) could be expected to have all the assets pre-compiled, because... once it's released the assets aren't going to change... but looking at how assets in engines currently work, that is not the case, probably to allow including uncompiled assets within other uncompiled assets in the Root app (e.g. application.scss could use mixins from an engine's scss files) if that's the intent of the engine. I'm not totally sure whether that would ever be the case with a webpack engine I guess... is it feasible that there might be a use case for an engine that provides pieces to be used within a larger webpack app? Perhaps the engine would expose some Elm module or React component, and the Root app's UI would utilize that component from its own Elm/React app?

In Elm at least, that actually doesn't make sense as a use case, because all modules have to be explicitly included in the elm-package.json, and will then be added to the dependency graph independently of any Rails Engine. But I'm not sure the same can be said of every potential webpacked app - for instance, what if your Engine's app is just straight ES6 that exposes some modules? Could the Root app import those modules from the engine to be used as potential internal components of it's reactive UI?

If that kind of integration is possible (and useful, which seems likely to me if it's possible), then in order to facilitate it, a Root app obviously needs to be able to compile the engine's assets (after they are included and used however they are going to be used).

My main argument against allowing this type of integration, through a Rails Engine specifically, would be that it's mixing what are becoming ever more the separate responsibilities of the server-side app and the client-side app. That is, if your reactive UI requires some component, it should import it using a javascript dependency manager like npm, whereas the Gemfile should manage dependencies for your server-side Rails app (which may include some compiled reactive UI app to be used on the front-end, like any other asset to be sent to the client)

My question is: is that a good enough argument to make it impossible using webpacker (by assuming there will be no precompiled integration between the UI components of an engine and its root app, and hence not providing any considerations to facilitate such integration)?

tomprats commented 7 years ago

I was originally patiently waiting for this kind of support so I could upgrade a gem I manage to be used with webpacker, but @mltsy's argument resonated pretty well with me.

Javascript provided by an engine to be used through webpacker could just be put in (an) npm package(s). This would also allow the javascript to become part of the npm ecosystem, becoming a dependency or depending on other packages.

The engine could still provide CSS and JS when needed through sprockets. One downside is the potential overlap of an engine providing a page that depends on javascript that is also provided in its npm package. It could lead to a few use-cases of an end user having to download the same javascript code from multiple locations

tomasc commented 7 years ago

@mltsy @tomprats I partially agree with this view.

However, let's say the engine is a pluggable CMS that defines its own JS dependencies (similarly to gems in .gemspec). I would hope there would be a way for the main Rails app to pickup these dependencies as well.

As of now, in my case, most of the JS dependencies defined in a Rails engine are facilitated via https://rails-assets.org. It would be fantastic to be able to define them via webpacker.

mltsy commented 7 years ago

Yeah - this is a fairly unique situation where we have essentially two dependency managers in a single application (bundler and npm), and specifically the situation you're describing, where we have a Rails app with a bundler dependency on a Rails Engine, and an npm dependency on the "webpack app" (for lack of a better term, assuming it's not published as an npm package) that is defined in the Rails Engine. And that's the sticking point - the Rails engine "depends on" the webpack app that is implemented in its javascript/packs directory, but that dependency isn't really part of bundler or npm. So it's hard to say how/if it should be exposed in the contexts of npm and bundler. If we expose that somehow, we would essentially be adding javascript dependencies via the ruby dependency manager, which confuses things a bit when it comes to the javascript dependency manager... (there are some things included via an external channel, which creates somewhat of a mess when it comes to dependency management).

The best solution I can think of to handle that would be to somehow create/expose a kind of meta-package (or legitimate package?) out of the webpack app(s) contained within the Rails Engine, and automatically add that package to the packages.json of the Rails app before compiling with webpack (or maybe when the Engine is installed?). But this is all very theoretical, so maybe I should just shut my mouth and let someone who has attempted something like this share their wisdom ;)

justin808 commented 7 years ago

@mltsy @tomasc @tomprats @ohadlevy @soundasleep @fiedl:

The beta version of React on Rails is built on top of Webpacker and here are the docs for using it.

I just released v9 beta.1, built on top of:

chimame commented 7 years ago

How about specifying more than one in source_path of webpacker.yml?

default: &default
  source_path:
    - app/javascript
    - engine/javascript
  source_entry_path: packs
  public_output_path: packs
  cache_path: tmp/cache/webpacker

Specify the entry point of the application and the entry point of the engine, and build it with the application. If it is dynamic, it will be glad even if it can be done dynamically with ERB etc.

tomasc commented 7 years ago

Thanks @chimame, looks like a good idea, I will give it a try and report back.

lazylester commented 6 years ago

@chimame I have successfully done something similar to your suggestion. Since all the engines in my app are in vendor/gems, my source_path definition in webpacker.yml is:

source_path: '**/app/javascript'

And this pulls in the modules from all the engines as well as the main app.

This is working very well for me, except I had to hack a single line in the rails webpacker npm module, as the manifest.json keys were not correct. I would happily create a PR for this, but I think there may be other use cases I'm not thinking of. Also, although my hack doesn't break the tests, I don't think this line of code is covered by tests and it's not clear to me why it was written that way.

I have a small example app with this implemented at https://github.com/lazylester/webpack_example

mltsy commented 6 years ago

Hmm... You're right! That's cool! At least partly... I mean that example app isn't quite the proof of concept that is necessary, because it doesn't actually require a module defined in the engine, but I bet it could!

I was slightly misunderstanding the role of npm and webpack. npm defines and downloads the sources (dependencies) from which webpack can choose what to include in any given pack. An engine is just another source for webpack to pull from, so it supplements the sources supplied by npm. Then webpack can choose from either npm's dependencies or the additional sources provided by the engine(s)...

Now... that is still a little bit disconcerting, just because those sources are outside of npm's dependency graph, meaning... the other thing we still need to solve is how to install the dependencies of the engine's pack(s). I see your "inventory" engine has no package.json (no JS dependencies) - that's not too likely in the real world, I imagine. Would you want to try adding and using one dependency and see if that works? I can't imagine running npm install on the main app would also install the engine's dependencies. But maybe there's a way to get around that... ? (I'm no expert in npm, so someone else here might have a better idea than I do about how to solve that issue)

lazylester commented 6 years ago

@mitsy you're right, I don't currently require any modules in the engine. I have done it elsewhere but forgot to include it in the example app. I'll update the example and post here. However my engines do not have any of their own npm dependencies, just local modules, within the engine, and a global (ractivejs) module that is in the main app's node modules.

So, yes, my approach has limitations. and doesn't cover many use cases. But it gets me to production!

mltsy commented 6 years ago

@justin808 - now that I understand what I'm looking for more I looked through the docs for using react_on_rails to see how you're handling this issue too. I see you're using a rails task to run a command provided by the engine (configuration.build_production_command) to compile the engine's pack, but (and this may be my ignorance about webpack) I see two issues in using this as a general solution:

1) It looks like that just runs webpack... when do the dependencies get installed? Does webpack install them? Does the user need to install them before running the task? (I see a yarn build:production version of the command in the test suite, but it looks like that script is not actually implemented or used)

2) If that command runs webpack with a totally separate webpack.yml, that wouldn't allow the parent application to include any of the engine's modules, etc. right? It's just compiling a totally independent pack? I see there's a comment just above the command saying if you don't want the engine to build the pack for you, that command can be set to nil, but I don't see any documentation on how to compile your pack in that case. Have you designed a workflow for compiling a webpack in the parent app that uses components/modules from your engine?

mltsy commented 6 years ago

Aha! Maybe we could use npm's local dependency feature? https://docs.npmjs.com/files/package.json#local-paths

Yarn also respects this syntax... so we could tell the user to add this local dependency (file:vendor/gems/engine/? I'm not sure where the "package" root actually is - I would assume whatever directory the engine's package.json ends up in) to package.json in the parent app if they want to use any of the locally provided modules in their parent app's pack, and if not, provide a task for compiling the engine's pack independently! (could even make the installation part of an engine:install task)

I have to try this... when I get a chance... (if anybody else does, I'd love to hear the result)

chimame commented 6 years ago

@mltsy I think it's a good idea.

If both the idea to add to my webpacker.yml and the plan to add to yourpackage.json come true, you can do the following.

Write javascript_pack_tag 'engine_ javascript' in View (if it exists in webpacker.yml)
import engine from 'engine_ javascript' can be defined in JavaScript (if it exists in package.json)

lazylester commented 6 years ago

@mitsy I updated my example app https://github.com/lazylester/webpack_example.

The engine now has a local module of its own and npm modules. You'd probably have to build the packs for the engines in addition to the packs for the main app in a build script, but I assume that's not a big deal, since there has to be a build script anyway, right?

I'll generate a PR for the webpacker change that permitted this to work. It may or may not be accepted, depending on whether it breaks something else. As I mentioned before, the tests still pass.

lazylester commented 6 years ago

Pull request: https://github.com/rails/webpacker/pull/875

mltsy commented 6 years ago

Nice! So... I guess there are two use-cases for engines here which have different solutions:

I want to use an Engine to provide uncompiled modules for use in a parent app's pack(s)
I want to use an Engine to provide one or more compilable packs to a parent app

In case 2, that's what react_on_rails seems to be doing - it provides a rails task for compiling the engine's pack, and then you can use it like any other asset. @lazylester and @chimame's solutions are nicer ways to be able to do that without having to run a separate task, by including the engine's entry points in the webpacker sources of the main app. :star:

The use case I'm looking to solve for my personal use is case 1 though. And now that I think about it, we don't even need to include the engine's entry points in the webpacker sources to enable that, we just need to depend on it as a local dependency in package.json right? If that's true, it makes this all very clean, since there are orthogonal/independent solutions to each use case, and each can be employed depending on the purpose of the engine. (I still have to try the local dependency thing though for case 1, and the specifics of automating case 2 still need to be ironed out, since neither suggestion currently works with webpacker)

Since they seem to be orthogonal concerns, I suggest we open a new ticket for one of these use cases, and clarify the title of this ticket to represent the other - but I'm not sure which one corresponds best to the original post here... @fiedl ?

chimame commented 6 years ago

As a result of various thinking, I implemented it like this.

We are considering the following.

Packages managed by yarn can be managed on the Engine side
We assume that JavaScript on the Engine side is to be built in the upper application

1 means that it is not necessary to add the package on the upper side. This became possible by correspondence with @mltsy's comment. There is also the idea that 2 was wanted to create on the Engine side while keeping the image of the JavaScript constructed by the upper application.

Please refer to this commit for details.

etewiah commented 6 years ago

My usecase is super simple. I just want webpacker to compile everything into one file in the standard rails javascripts location. In my usecase that will be:

property_web_scraper/app/assets/javascripts/property_web_scraper/my_webpack.js

I will then reference that js file as per normal from my application layout.

When I setup webpacker I was obliged to install it in the dummy app so the config file for example is here:

property_web_scraper/spec/dummy/config/webpacker.yml

I imagine I can set up a config to do this and invoke it somehow but I just don't know how :(

tleish commented 6 years ago

We are testing a solution for using webpacker within an engine using the engines dummy app and the asset pipeline. The general idea is:

Remove the webpack fingerprint hashing from generated files as it will be fingerprinted in the asset pipeline.
Export compiled packs to a namespaced folder inside the packs folder using the webpacker.yml (test/dummy/packs/my_namespace/file.js)
Include the test/dummy/packs folder within the engines asset pipeline (config.assets.paths << config.root.join('test/dummy/public/packs')
Include the packs file config.assets.precompile << %r{my_namespace/.*\.(?:js|js.map|css|png|jpg)\z}
Since the default webpacker precompile behavior is to automatically compile before after asset precompile, we disabled this default behavior ( ENV['WEBPACKER_PRECOMPILE'] = 'false' ), and instead have webpacker auto-compile with rake assets:precompile before the asset precompile is performed. That updated packs are automatically included with the assets.
Last, we monkeypatched the pack helper methods to use rails asset pipeline helper methods instead the engine was being used used by the parent application, but still use default pack helper methods when developing using the dummy app.

With the above changes, we can develop the engine in isolation using the standard webpacker functionality, but then in it's hosting parent, it can just use the generated assets when it performs rake assets:precompile. Monkeypatching the include pack helper will then allow us to use the pack include tags in the event that someday a solution is created for allowing parent and engines to both use webpacker, at which point the monkeypatch can be removed.

We made all the above configurable. For example, step 1 is controlled by fingerprint: true in the webpacker.yml file.

Questions:

Is there any reason we couldn't change step 5 to be the default behavior? The less we override changes to deviate from the main webpacker gem the better.
Is there any interest in including any of our changes as part of the main webpacker gem?

gregogalante commented 6 years ago

@dhh

What about a copy of source code as a node module on a temporary directory when assets are compiled?

Take this example:

I have an engine with apackage.json file on root directory and assets on app/javascript/engine_name path.
When i need to compile assets the webpacker:compile task takes all gem with webpacker dependency and a package.json file on the root (or with a configuration that tells rails to consider the gem as a webpack dependency) and copies them on a temporary directory (it copies only the package.json file and the app/javascript/engine_name directory).
The webpacker resolved_pathsoption automatically contains also the temporary directory used.
After the compilation of assets the temporary directory can be deleted.
If I run rails server with RAILS_ENV=development the application should copy dependencies to the temporary directory on startup (so i can develop the application using the gem dependency as a normal npm dependency).

This method permits also to mantain the possibility to compile source code of the engine on app/assets/javascript (to be used with Sprockets) and to publish the engine javascript code to npm (similar to what is done on rails activestorage).

I have created a generator that should be used inside an engine to develop it with webpacker (https://ideonetwork.github.io/dakarai/#/pages/EngineGenerators). The generator should also update the dummy app to import engine assets and watch them on change.

radeno commented 6 years ago

This is quick hack when you use only one engine with webpacker. For example separated CMS UI from base rails. Whole idea is about using symbolic files.

Just make Rails Engine with Webpacker (add into gemspec). All Webpacker required files will be in engine directory. Then symlink files or directories from engine directory to main directory:

config/webpack
config/webpacker.yml
.babelrc
.postcssrc.yml
node_modules
package.json
yarn.lock
public/packs

Javascript and styles are separated to engine.

jrochkind commented 6 years ago

I'm confused where this issue stands right now.

The webpacker README says "If you are adding Webpacker to an existing app that has most of the assets inside app/assets or inside an engine...", which sort of implies this is a supported use case. But the example following that suggestion actually isn't an example of "assets inside an engine" at all.

Is it actually a supported use case to have "assets inside an engine"? If so, can an example be provided? If not, should that suggestion be removed from the README?

mltsy commented 6 years ago

@jrochkind - It looks like the README is showing specifically how to include uncompiled assets from any directory (including an engine) - I haven't tried this, but it sounds like it may be possible just by adding the engine's path to the resolved_paths array. The trick is figuring out what that path is (if it's not part of your project's working tree). I would also like to see an example of what that directory reference should look like... if you figure it out, please post!

Actually here is an interesting example of someone including assets from a gem by vendoring the gem (unpacking it into /vendor): https://github.com/rails/webpacker/issues/57#issuecomment-330457616

From the rest of that discussion though, it sounds like the crowd is moving toward publishing a separate NPM package for any gem/engine that is supposed to work with webpacker...

jrochkind commented 6 years ago

I would suggest that if the README is going to mention engines, it probably should provide an example.

I do not vendor my gems into ./vendor, that isn't really a general purpose solution.

inkhrt commented 6 years ago

what exactly do you need help with?

On Sat, Apr 14, 2018 at 8:40 AM, Gregorio Galante notifications@github.com wrote:

What about a temporary copy of source code as a node module on a temporary directory when assets are compiled?

Take this example:

I have an engine with apackage.json file on root directory and assets on app/javascript/engine_name path.

During development the dummy app package.json file define the engine dependency as a local dependency.

"dependencies": { "engine_name": "file:../../" }

When i need to compile assets with the webpacker:compile it takes all gem with webpacker dependency and a package.jsonfile on the root and copy them on a tmp directory (it copy only the package.json file and the app/javascript/engine_name directory).

The webpacker resolved_pathsoption automatically contains also the temporary directory used.

After the compilation of assets the temporary directory can be deleted.

This method permits also to mantain the possibility to compile source code of the engine on ``àpp/assets/javascript``` (to be used with Sprockets) and to publish the engine javascript code to npm.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/rails/webpacker/issues/348#issuecomment-381310855, or mute the thread https://github.com/notifications/unsubscribe-auth/Ah9VyP7bSPc-cgN7B5B1vpmGjoNGdheLks5toaf6gaJpZM4NUgTi .

jrochkind commented 6 years ago

An engine is providing javascript assets.

In sprockets world, they are on the sprockets load path, and including them in the sprockets-compiled JS is as simple as require 'on/load/path/file.js' in your sprockets file.

In webpacker world, where the host app is using webpacker instead of sprockets (and the engine-provided js assets may be ES6 etc)... what does one do?

gregogalante commented 6 years ago

@inkhrt

In my company we have already created a generator used to create the described type of engine (https://ideonetwork.github.io/dakarai/#/pages/EngineGenerators?id=webpacker). The only thing that should be included inside the webpacker gem is a task that copies all gems javascript modules on the temporary directory when "rails assets:precompile" is executed.

The code used now is this:

# copy package.json to vendor directory
src = "#{MyEngine::Engine.root}/package.json"
dst = "#{Rails.root}/vendor/modules/my-engine/package.json"
FileUtils.mkdir_p(File.dirname(dst))
FileUtils.cp(src, dst)
# copy javascript directory to vendor directory
src = "#{MyEngine::Engine.root}/app/javascript/my-engine"
dst = "#{Rails.root}/vendor/modules/my-engine/app/javascript/my-engine"
FileUtils.mkdir_p(File.dirname(dst))
FileUtils.cp_r(src, dst)

It should be executed automatically on assets compilation and we need to find a way used by gems to tells webpacker that they have javascript dependencies that needs to be compiled (the first idea was to consider all gems with a package.json file on the root path).

jejacks0n commented 5 years ago

I would like to add to this a bit. Maybe it's been discussed already, but what I've seen is just pathing issues when used inside a dummy app or in a non-straightforward way. I've written a fairly widely used (maybe less so these days) javascript testing library for rails and so understand some of the nuances of javascripts and engines.

I'm doing this in a non-open source engine for what it's worth, but the same thing applies if I were to upgrade my teaspoon project to use webpacker. Ok, so typical setup, an engine, inside the project is a spec/dummy app that rails more or less generates when generating a new engine project. There are also tests that are sitting next to the dummy app, and that use the dummy app to utilize the engine in a fully functioning rails app for testing behaviors etc.

Now, worth mentioning that this dummy app is in fact fully functional at this point, everything actually works when you cd into spec/dummy and run it in the development environment utilizing the webpack-dev-server. Everything also works when you fire it up in the test environment (e.g. rails s -e test) and the assets are compiled into public/packs-test as the default configuration dictates.

However, when running the tests, which are run from the engine root and not the dummy app root, webpacker will fail to compile when capybara fires up the dummy app, and it appears the reason is because pathing is assumed to be based on where you're running a thing from, and not where your rails application lives. I tend to believe this is a bug.

I tracked it down and I've monkey patched the compiler class to resolve this issue, and if anybody cares to make a PR that would be awesome -- I'm mostly sharing this for the dev team to help identify potentially how better to integrate with existing rails engine core concepts because it's a fairly simple adjustment to gain a lot of potential benefit. The thing to note here is that I'm only chdiring into rails root to run the command found at compiler.rb#L59, and when I do so everything begins to work as it should.

require "webpacker/compiler"
class Webpacker::Compiler
  private

    def run_webpack
      logger.info "Compiling…"

      stdout, sterr, status = Dir.chdir(Rails.root) do
        Open3.capture3(webpack_env, "#{RbConfig.ruby} ./bin/webpack")
      end

      if status.success?
        logger.info "Compiled all packs in #{config.public_output_path}"
      else
        logger.error "Compilation failed:\n#{sterr}\n#{stdout}"
      end

      status.success?
    end
end

I guess I should add that I'm not trying to load javascripts or assets from my engine directly, that's done through npm or yarn and is simply bundled in the same library as the gem by specifying a package.json -- anyhow, this is more dealing with how to test your implementation in an engine. If anybody wants to pick my brain on this more I'd be open to walking someone through the whole structure that I've setup and what I think is working nicely as a distribution platform for an engine with complex logic and javascript assets.

markets commented 5 years ago

Hi 👋 #1836 has been recently (~ 8 hours ago) merged, hope this helps...

Specially this file: https://github.com/rails/webpacker/blob/master/docs/engines.md

mltsy commented 5 years ago

Awesome! Thanks @markets I'm no longer developing with webpacks in an engine currently, so I'm not in a good place to evaluate this, but just to help resolve the conversation (and possibly move toward closing this ticket), could you identify whether this PR solves one or both of these use cases discussed in this ticket (mentioned in a previous post above):

I want to use an Engine to provide uncompiled modules for use in a parent app's pack(s)
I want to use an Engine to provide one or more compilable packs to a parent app

adamwgriffin commented 5 years ago

I tried following the docs that @markets linked to at https://github.com/rails/webpacker/blob/master/docs/engines.md and I'm having a really hard time getting it to work. Has anyone else tried following these docs? If so, do you have a repo with a working example that I could look at?

jejacks0n commented 5 years ago

It’s in an early stage, but the concepts are in place at:

https://github.com/legworkstudio/protosite/tree/master/spec/dummy

I’m not at a machine, otherwise I’d provide more insight. Note the node_modules symlink, and the place I patch webpacker:

https://github.com/legworkstudio/protosite/blob/master/spec/dummy/config/application.rb

The patch in https://github.com/legworkstudio/protosite/blob/master/spec/dummy/config/webpack/compiler_patch.rb is what I suggest as a fix here but haven’t been bothered to submit a PR until I hear more feedback for core devs.

On Jan 9, 2019, at 5:49 PM, Adam Griffin notifications@github.com wrote:

I tried following the docs that @markets linked to at https://github.com/rails/webpacker/blob/master/docs/engines.md and I'm having a really hard time getting it to work. Has anyone else tried following these docs? If so, do you have a repo with a working example that I could look at?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

adamwgriffin commented 5 years ago

Thanks @jejacks0n

jejacks0n commented 5 years ago

@adamwgriffin it's not exactly what you were looking for, because I did it without having a walkthrough to follow, so I'm pretty sure it doesn't follow that exactly.

adamwgriffin commented 5 years ago

@jejacks0n it's not exactly what I was looking for but maybe this will work better 👍

ghaydarov commented 5 years ago

Has anyone else tried following these docs? If so, do you have a repo with a working example that I could look at?

@adamwgriffin I was able to get it working on a sample app below.

https://github.com/ghaydarov/webpacker-with-engines

adamwgriffin commented 5 years ago

Thanks @ghaydarov that's just what I was looking for!

emanuel-campos commented 5 years ago

@ghaydarov

Webpacker can't find c1-application.js in <root_path>/webpacker_engine_app/public/packs/u-packs/manifest.json. Possible causes:
1. You want to set webpacker.yml value of compile to true for your environment
   unless you are using the `webpack -w` or the webpack-dev-server.
2. webpack has not yet re-run to reflect updates.
3. You have misconfigured Webpacker's config/webpacker.yml file.
4. Your webpack configuration is not creating a manifest.
Your manifest contains:
{
}

zedtux commented 5 years ago

Thank you @ghaydarov very much for the nice repo, but can you give some instructions about how to redo the same please?

In my engine's repo I don't have the webpacker:install task, therefore when I do app:webpacker:install:react it's complaining :

webpack binstubs not found.
Have you run rails webpacker:install ?
Make sure the bin directory or binstubs are not included in .gitignore
Exiting!

zedtux commented 5 years ago

I got Webpacker working well within my Rails engine following instructions from https://github.com/rails/webpacker/blob/master/docs/engines.md and then fixing few issues.

The gem is here : https://gitlab.com/pharmony/active_record_migration_ui

angarc commented 5 years ago

@zedtux

Quick question, I'm a little new to webpacker and rails 6 in general. I followed your guide, and now I'm just wondering what the next step is? Would I run bundle exec rails app:webpacker:install? When I run that, I get another error about not knowing how to build a rake task app:template. I appreciate you putting up the step by step guide, I'm just not too sure what to do afterward.

Update I figured out what the problem was. I was supposed to run the command from the test/dummy/ directory, and not the directory of the rails engine. However, no my only question is what exactly I should put on my <%= javascript_pack_tag 'application' %> because when just putting application doesn't work. I saw that output path for the packs was in the public folder as specified by your guide. However, I can't seem to point my javascript_pack_tag there. Should it be something like <%= javascript_pack_tag 'public/my-engine-packs/application' %>? Sorry if this is a stupid question.

Another update Okay I see where I went wrong... sort of. I moved my javascript folder to the app directory like any other normal rails 6 application. I want my rails engine to have bootstrap on it. I've already installed it using webpacker successfully. Now, the only problem is that the bootstrap styling is not displaying. I just get the regular default browser styling. When I checkout my logs I get this:

ActionController::RoutingError (No route matches [GET] "/my-engine-packs/js/application-0c695baf97e03e4d45ba.js"):

I remember setting my-engine-packs as the public_output_path in my webpacker.yml. Do you know what I could go about fixing this?

rails / webpacker

How to use this gem from within rails engines? #348

Questions: