search

broccolijs / broccoli

Browser compilation library – an asset pipeline for applications that run in the browser
https://broccoli.build
MIT License
3.33k stars 217 forks source link

How to do incremental rebuilds if previous tmp folder is gone? #206

Closed rickharrison closed 6 years ago

rickharrison commented 9 years ago

I am working on a plugin to do incremental rebuilds with browserify. I'm having trouble figuring out how to implement the rebuilds.

I have been hacking around with the internals used by watchify to accomplish faster rebuilds. However, I have run into a problem where the browserify cache points to the file in the old /tmp/ folder, which no longer exists. I then get an error like this:

Error: ENOENT, open '/Users/rickharrison/projectname/tmp/replace_filter-tmp_dest_dir-k8h7XRHN.tmp/routes/sign-out.js'
  at Error (native)

Do you have any suggestions on how to get around this or a better approach to incremental rebuilds?

joliss commented 9 years ago

In general, the expectation is that you use caches only as long the server (broccoli serve) is running, that is until the final .cleanup call. Between Broccoli runs, the code and configuration can change, so when I run broccoli serve or broccoli build, I want there to be a fresh initial build at first.

I hope this helps - if not, could you explain more about what you're trying to do?

rickharrison commented 9 years ago

Currently, my build process does not use broccoli serve and instead uses gulp to watch my source. Within one of my gulp tasks, I am using Builder.build() to build my assets. This involves browserify, stylus, etc. The problem is that browserify takes from 2000-3000ms per build. I would like to get this down to only a few hundred milliseconds by utilizing caching/incremental rebuilds.

I have been able to hack together a solution where I am caching the node_modules directory between rebuilds for browserify. However, I am having trouble implementing a cache layer for my own source inbetween rebuilds.

stefanpenner commented 9 years ago

let broccoli manage the build rather then gulp, you are unfortunately going away from the primary architecture goals of broccoli. (where it manages the build life-cycle).

@joliss may have some ideas, but it really does sound like you are causing yourself lots of extra work.

rickharrison commented 9 years ago

There is no such thing as broccoli watch so I'm not sure how I can do that. I can't use broccoli serve because my app also involves a node server which I need to serve the assets. Whenever I change a file, I need the build to be in /dist, hence why I am using gulp to watch the files and then call Builder.build()

I think my main problem is figuring out how to add caching. With the current broccoli-browserify plugin, it just re-compiles everything every single time, which causes the multi second build times. Browserify caching involves using the full path to the file because of how it parses the require tree. Once the tmp folder changes, that browserify full path no longer exists breaking the cache. I'm trying to figure out if there is a way around that.

recipher commented 9 years ago

Try broccoli-timepiece to watch and build without running a server?

— Johnny H 07971 880871

On Sun, Oct 26, 2014 at 5:33 PM, Rick Harrison notifications@github.com wrote:

There is no such thing as broccoli watch so I'm not sure how I can do that. I can't use broccoli serve because my app also involves a node server which I need to serve the assets. Whenever I change a file, I need the build to be in /dist, hence why I am using gulp to watch the files and then call Builder.build()

I think my main problem is figuring out how to add caching. With the current broccoli-browserify plugin, it just re-compiles everything every single time, which causes the multi second build times. Browserify caching involves using the full path to the file because of how it parses the require tree. Once the tmp folder changes, that browserify full path no longer exists breaking the cache. I'm trying to figure out if there is a way around that.

Reply to this email directly or view it on GitHub: https://github.com/broccolijs/broccoli/issues/206#issuecomment-60524866

rickharrison commented 9 years ago

I did try that and I've had a lot of trouble with it so I switched to gulp. The gulp part is working perfectly fine for my whole build process. Css, javascript, etc. I really don't think that is part of the problem. I am just looking for advice on how to build a plugin to implement some sort of caching layer so the entire tree does not need to be re-compiled.

stefanpenner commented 6 years ago

mixing two different pipelines isn't going to be much fun. I have some ideas to make it better, but likely not a short-term thing.

For anyone looking for broccoli + browserify integration then: https://github.com/eploko/broccoli-watchify may help. I need to fix it for 1.0, but that should happen soonish. (ping me if it doesn't)