0.5.4 urequire0.6.9 edition

[anodynos]

As described in #5

Initial grunt tasks, building as UMD & combined, runs basic Bob test on browser

Initial tests on web / phantomjs run as UMD

enhancements in package.json & bower.json deps

see urequire_notes.md

[anodynos]

Its now using urequire 0.6.10 - the branch name is misnomer :-( It adds clean and dependencies.replace to translate hardcoded '../lib/XXX' path in tests to '../UMD/XXX'

[kmalakoff]

Thank you for all the hard work on this and sorry for taking so long to get back to you.

I have to admit that I'm having some difficulties understanding all of the changes. Can I confirm some things to see if I understand:

1) How do I run your grunt changes? eg. what is the new test and release workflow? I'm used to just running "npm run test" and having everything built and tested. Is there a main command that runs and tests all your changes?

2) Where is the uRequire universal build placed? I used to output backbone-orm.js and backbone-orm.min.js to the root directory for browser consumption, but they do not appear to be being built anymore and so I cannot check where your uRequire library wrapping is injected.

3) I've noticed that you added some extra dependencies to bower. I believe neither should be browser dependencies - why have they been added?

    "mocha": "*",
    "requirejs": "*"

4) In urequirenotes.md, you mention "underscore' is exported as '' to all modules - as a bundle global, as an example. It could be removed from the code and it will eventually save some bytes." What does this mean? Are you suggesting that I publish underscore globally for the browser AND Node.js? If so, this isn't something that I'm prepared to do.

5) The grunt file is now quite large. Is there any way to turn any of your changes into libraries so that the grunt file is more like a minimal configuration file? (eg. put your registerTask functionality into an npm module). I'm not a big fan of Grunt because it tends to sprawl too much (I wrote easy-bake to make a non-coding build system) and prefer to treat it like a configuration file.

6) I am not sure why you moved the browserified node modules into src (and how they even get referenced?). I definitely would prefer to release a version of backbone-orm after this is supported.

7) In urequire_notes.md, you mention "All web-looking stuff & tests should use bower_components instead of node_modules." I'm really not convinced to make the tests depend on yet another module system and have to define anther automation path (eg. bower install). Are there really any significant differences in library versions released via npm vs bower?

8) For tests on web, it is already quite complicated so I haven't been doing any automated tests (only testing the Node.js versions and then opening the tests in test/web). Do you have recommendations for how to re-use the tests with the window.BackboneORM symbol? I just don't see a simple path to reuse tests.

It is really incredible how much time and effort you put into this!

[anodynos]

Hi @kmalakoff and happy holidays :-)

First of all, keep in mind this grunt/urequire config is a 'hybrid' and 'semi-complete', as I left you existing build flow & all outputs almost as it was and just showed some examples of how a urequire based build workflow would work.

To this end, all urequire outputs are in _build_urequire, which is of course as an intermediate example and not meant to be published as it is.

1) "npm run test" should work just like before (as all your previous commands & outputs should, like npm run prepublish. The main difference is that I am redirecting the plain coffeescript compilation (i.e npm build) to grunt build, which in turns uses urequire so that it can take advantage of the automated VERSION injection. Its just an example, but I think its a good practice to have all build-related stuff in one place (i.e gruntfile.coffee).

2) All urequire outputs are in _build_urequire:

Specifically

• grunt webCombined (a shortcut to grunt urequire:webCombined) uses the combined template and outputs to _build_urequire/backbone-orm-combined.js,
• grunt webMin does similarly to _build_urequire/backbone-orm-combined.min.js.

These are the combined .js files, which can't unit test individual files. So I 've also convert the individual modules into UMD:

• grunt webUMD uses the UMD template, converting each nodejs module to UMD (a target for your unit tests), outputing to _build_urequire/UMD

and finally

• grunt specsWeb converts your test/* with UMD into _build_urequire/test (translating paths starting with ../lib to ../UMD and also copying all non-converted files there:-) so you can unit test on the browser!

So if you run grunt webSpecs (a shortcut to grunt webUMD specsWeb mocha) you see the tests run with phantomjs on the console - or just open /_build_urequire/test/web/SpecRunner_unoptimized_AMD.html with your browser and enjoy (some) tests run with mocha/chai on the browser. Bare in mind its using suiteWeb that excludes nodejs-only tests.

Also, you can check the simplistic 'Bob' test on /_build_urequire/test/web/tests_urequire_combined.html, which loads the backbone-orm-combined.js file.

There are also some other UMD builds onto /lib, so these are tested on nodejs as is. Remember that all urequire builds derive (i.e inherit) from _defaults. Again these are just some examples of how you can build & test on both node & web with urequire. Feel free to amend as you see fit.

3 & 7) Your right - these should be just devDependencies - I 've pushed a commit that fixes it. In an ideal world we should have one module system, one package system etc. Unfortunately this is not the real world :-( In any case, its just one more .json file, which you already have anyway. As far as differences, I am not sure, but I guess its a better practice to load web stuff from bower_components rather than node_modules in web spec runners, tests etc. But that's again up to you.

4) That is just an optional urequire feature where you can save your self from typing _ = require('underscore') in each module, when you know you need it in (almost) all modules. Its added automatically to all your converted modules with a simple urequire declaration. There's no other obligation or side effect on your side. It can save some typing and build size/speed on the combined template, since these dependencies are loaded only once and available throughout the bundle's closure. On UMD/nodejs templates they are required each time, as if you had typed them :-) Again, completely optional - you can always enable the feature in the future (and remove the _ = require 'underscore' from your modules, not that it harms if you dont) when you feel more comfortable.

5) Well this is an example build that's both your current config and the new urequire version. In a urequire-only version you would save a lot of config stuff like uglify, clean, banner, coffee compiles etc and also when you take advantage of derive you can create new targeted builds with two liners:-). The registerTask could be an npm module indeed, but its an overkill maybe ?

6) The files need to be there cause they are to be included in the build. They are referenced from your code when you require('util'), and since they are in the bundle's root (.ie src/) they are picked up as belonging to the bundle. Currently urequire reads only from only one directory, which is a bummer. I plan to change this in 0.7, but I am not sure when this is going to be developed.

7) see 3)

8) That path is entirely up to you. Right now you can run your unit tests on the browser as individual UMD files. What you need is to refactor the tests to require the 'backbone-orm' dependency (instead of hardcoded '../../../lib/xxx' paths) and test the library from the user's perspective. See the trick I used in uberscore so I can test only against the combined.js library from within the specs, without ever requiring a specific module.

Perhaps this is not suitable for all your unit tests - after all that's why they are called 'unit' tests! But when the test can be performed from 'outside', then its best to do so I think.

Still, for the unit-tests-as-UMD, since your librabry & spec modules can be used on the browser as AMD-loaded modules (thus requiring each module you wish) you need to remove nodejs specific dependencies & features or shim them somwhow. Again that is up to you cause you know best what this task needs - I'll be glad to help wherever I can :-)

Regards, happy holidays and a Happy New Year!

[anodynos]

Forgot to mention the watching in the workflow, i.e grunt tasks watch: compile, watch: webUMD, watch: webCombined etc - they are very useful while developing :-)

Especially when building modules with uRequire, since its a quite lengthy operation, uRequire makes sure that only really needed conversions are carried out.

BTW, you gave me a nice idea about a 'urequire-grunt' package, that would generalize what urequire does to backbone-orm, as a generic-pick-and-override-what-you-like-ready-for-multiple-builds-config-with-standard-commands - I hope I'll find the time to implement it :-)

Also I have some general suggestions for your build/workflow/conventions:

• keep file names more close to what they export - eg name file with module.exports = class NodeUtils as NodeUtils.coffee.
• Why do you need to keep track of a version header in your .coffee files - leave git to do the source versioning and urequire to inject it in your builds :-)

FYI: Both your knockback & backbone-orm projects are awesome - keep up with the good work :-)

[anodynos]

Check out https://github.com/anodynos/urequire-grunt/ - still v0.0.0 :-)

[kmalakoff]

Wow. That's awesome! I'll try it out and submit any patches.

I'm really sorry about not being more quick on responding. I've been working on a crunch period and you are so prolific, it is hard to keep up! I will make some time tomorrow to work through this.

[anodynos]

You're welcome, Happy New Year :-)

[kmalakoff]

Happy New Year. I'm getting there, but didn't get everything done. I'll finish up over the weekend.

Quick question: I only recently heard about gulp, but find the concept and implementation of scripts cleaner with it. What do you think about gulp vs grunt?

I was thinking about, as a second step, taking your work in urequire-grunt and using it to make a gulp module. The config should select the destination type for the conversion (I'm assuming the default stream output would be 'combined') and no need for minification; also, because gulp tries to limit plugins to single operations, maybe the specs would be handled in a separate gulp plugin?

[anodynos]

I just looked at gulp, I can't quickly grasp what it's trying to solve, apart from being in-memory (which is the urequire mantra also). I need to see more real world examples, but I can see that modules aren't 1st class citizens. I do think grunt tends to be too verbose, and uses the file system for intermediate results which is too slow. urequire-grunt aims become an extendible boilerplate around grunt, using require to build mainly around modules. I guess the main class could be used standalone or become the basis for other builders like gulp. I will take a look and get back to you.

[kmalakoff]

I think the real benefit of Gulp is simplicity: each plugin does one thing, takes a stream in, and writes a stream out - that's all. It makes connecting all the transformations easy to follow and customize.

When I look at Grunt files, they look like disconnected sections of configuration spew; whereas, Gulp files focus on the connections between transformations making understanding and maintaining a build script easier.

[kmalakoff]

I've been playing around with Gulp and uRequire to try to define a desired pipeline. I think I've got it!

Here is what I am thinking in terms of how a Gulp plugin should be written for uRequire.

And here is a sample gulpfile to consume it:

gulp = require 'gulp'
gutil = require 'gulp-util'
coffee = require 'gulp-coffee'
urequire = require 'gulp-urequire'

gulp.task 'build', ->
  gulp.src('./src/**/*.coffee').pipe(coffee({bare: true}).on('error', gutil.log))
   .pipe(urequire({template: 'combined', project_name: 'backbone-orm', root: './src'}))
   .pipe(gulp.dest('./dist/'))

The idea is that uRequire Builder should have a super-simple interface that you add files to.

builder = new urequire.BundleBuilder(options)
builder.compile @files, (err, result) =>

Also, root should be optional in case someone wants to combine file streams (like my client/node-dependencies). For example:

es = require 'event-stream'
gulp = require 'gulp'
gutil = require 'gulp-util'
coffee = require 'gulp-coffee'
urequire = require 'gulp-urequire'

  es.merge(
    gulp.src('./src/**/*.coffee').pipe(coffee({bare: true}).on('error', gutil.log))
    gulp.src('./client/node-dependencies/**/*.js')
  )
    .pipe(es.map (file, callback) ->
      file.path = file.path.replace("#{path.resolve('./src')}/", '')
      file.path = file.path.replace("#{path.resolve('./client/node-dependencies')}/", '')
      callback(null, file))
    .pipe(urequire({template: 'combined', project_name: 'backbone-orm'}))
    .pipe(gulp.dest('./public/'))

Here's some WIP on converting the backbone-orm build over to Gulp. Let me know what you think!

[kmalakoff]

@anodynos: after playing with Gulp for a while, I can't help but get the feeling like uRequire could be written as a series of Gulp plugins and then having a consumer API designed to interface with it that hides the implementation details.

Let me explain, I studied the uRequire code over the last week and it almost feels like uRequire has a little too much custom code to do non-core things (eg. watching file, getting file contents, etc) so that its core purpose, API, and settings become obscured. I feel like there should be a uRequire core library that does the transformation between input files and compiles an output string and that is all that core does. Then there could be a series of modules around core that expose it in different ways for different build paradigms: 1) urequire-cli, 2) grunt-urequire, 3) gulp-urequire, 4) urequire-node, etc that allow people to interface with uRequire depending on their build system needs.

Take a look at the way I re-wrote the backbone-orm build in Gulp to build the client. This sort of functionality could be wrapped in uRequire core where it uses streams of files, checks if they need compilation to coffeescript, concatenates then, and allows consumption of them through promises, callbacks, or streams. The options to the uRequire core builder would only specify the compilation options and the build-system-specific modules would provide additional options to, for example, write the result to an output file.

I know this would be a major refactoring effort and I'm not sure how compatibile it would be with almond (but you could always write your own wrapper and require shim quite trivially).

I have appreciated so much all of your feedback on making backbone-orm's build system awesome and fixing problems (AMD compatibles, removal of redundant underscore code, automating browser tests, etc) - I want to implement them all! The main issue I have had in moving over to uRequire so far has been the learning curve and how it has made the grunt build file so huge to the point that I don't understand how it all works so it is difficult for me to move over. This is why I am proposing that uRequire core does less and provides a simpler API so I can focus on fixing all of those problems rather than trying to understand what is going on.

I think backbone-orm is a perfect use case for uRequire, but I need to feel that I'm in full understanding and control of the build system to move over. I don't have a lot of free time (I'm a co-founder at a startup), but I'd be happy to allocate a few days to help with a refactor if you agree with my approach.

[anodynos]

Dear @kmalakoff - it seems to me that the gulp approach largely deviates from uRequire 0.6.x approach.

Let me briefly explain:

• uRequire is mainly about building bundles of Modules & dependencies - hence it directly contradicts with gulp's strict guidelines and specifically the "do one thing, and do it well" mantra.
• uRequire philosophy DOES use many "config options that make your plugin do completely different tasks", in order to streamline the building of resources that happens to be Modules (AMD or commonJS) into uglified-UMD-or-part-of-a-combined-js with injected/maniuplated code, deps, banners and other arbitrary one-liner tranformations. uRequire's derivable configs aim to alleviate 90% of grunt's config with 10% of the code/config - i.e in a _defaults or as a single setting like 'optimize: true'. With urequire-grunt, this will go down to 3% of the code :-)
• uRequire has its own in-memory pipeline (ResourceConverter). With its current mechanics, it controls over each file's content for a number of good reasons (dependencies, selective watch/changed rebuilding, injections, user/generic transformations etc). Note for example that npm run watch using your gulp config recompiles all .coffee files when you have the slightest change in one. uRequire compiles only the changed file and only if its JavaScript compiled code has changed, it rebuilds it as a module. I can't see how this can be replicated in gulp's approach.
• uRequire is treating a whole bundle of source files instead of single files. Effectively each file/resource/module goes through various states & transformations but I can't easily see how this functionality could be passed down to gulp's approach of (optionless?) pipes.

Gulp might have many virtues, but I can't see how resources/modules & their bundle/build config options would be piped in and how uRequire in overall would be refactored to fit in gulp's approach. Sadly right now I have very little time available for such a huge undertaking - but greatly thanks for you offering to work towards it! Anyway, when urequire-grunt hits v0.0.1 I will PR backbone-orm with a urequire-only config and then you can make a clearer choice :-)

[kmalakoff]

Understood. Optimizing a build pipeline for incremental changes is extremely important on large projects. I am looking forward to checking out what you do with urequire-grunt! And I'll implement your improvements to be ready...

PS. Here's my current implementation for the build task in gulp:

  gulp.src(['src/**/*.coffee', '!src/node/*.coffee', 'client/node-dependencies/**/*.js'])
    .pipe(es.map (file, callback) -> file.path = file.path.replace("#{path.resolve(dir)}/", '') for dir in ['./src', './client/node-dependencies']; callback(null, file))
    .pipe(compile({coffee: {bare: true}}))
    .pipe(modules({type: 'local-shim', file_name: 'backbone-orm.js', umd: {symbol: 'BackboneORM', dependencies: ['underscore', 'backbone', 'moment', 'inflection']}}))
    .pipe(gulp.dest('./'))

Part of the complexity (in the second line) is that I need to merge in a coffeescript and javascript stream and renormalize the module spaces (would be messier if used options). Also, it doesn't yet properly support a non-manual shimming approach (I would need to move to AMD-style defines and dependency tracking), the module system should be separate from the UMD-wrapping, and it is inefficient from an incremental building perspective.

The real interesting thing is how it provides points/methods of customization for the end user just like the combining of input streams, choice of compilation into JavaScript, UMD wrapper library, etc.

[anodynos]

The real interesting thing is how it provides points/methods of customization for the end user just like the combining of input streams, choice of compilation into JavaScript, UMD wrapper library, etc.

Have you checked ResourceConverters and Module Manipulation ? I find it much more clearer that pipes and streams.

[kmalakoff]

Interesting. If I understand correctly, it looks like they define an extensible, modular DSL for smart transformations.

Putting my Node module/API hat on, it looks to me like this could be an independent module from uRequire (or multiple modules - perhaps one for ResourceConverter and the other for ModuleManipulation and maybe another for watching). This is assuming that uRequire's purpose is knowing about the combinations of inputs to generate each module system variant and combinations of them (eg. specialized knowledge).

The module documentation/example could provide sample DSL files that generate outputs, provide documentation about how to publish/require your own ResourceConverter (they look like they are reusable building blocks), the module would provide a CLI, and more importantly, the module would provide a native API that takes the in-memory DSL and outputs the result to a string perhaps deltas and full result and shows how to start a watch loop and subscribe for changes (watch).

The idea then would be that this framework compares with grunt and gulp based on its principles of: 1) providing a DSL for specifying transformations, 2) providing a framework for ultra-efficient, in-memory, smart/incremental transformations, 3) does not provide a generate purpose task framework (this makes it more like gulp which is focussed on pipelining conversions compared to grunt which seems to be a general purpose task framework) but more of a transformed file aggregation framework.

By modularizing these modules outside of uRequire, the value proposition and usage is more clear plus people can leverage the value of them in new and different ways, eg. for example in testing workflows, file-based data aggregation (eg. logging), etc.

[anodynos]

Overall, I feel that streams are great when you have potentially slow connections and then you definitely need I/O buffering, but they are an overkill for a build system.

uRequire RCs are also plugged/connected in a flow, but without the cluttering 'coding' pipe() syntax and the inherent lack of versatility (eg rich options, passing a Resource / Module instance around that allows rich manipulation, having states of conversion etc).

Allow me to assert that the lack of buffering is not really adding any benefits to a other build system (eg no real speed improvements even for lack of I/O asynchronicity) apart from defining a flow / path.

Furthermore since RCs and XXXResource instances remain in memory, partial rebuilds are really rapid when watching (even for combined -min.js builds), something that pipes can't possibly do.

[anodynos]

Oops, we posted almost simultenously!

The previous points are all valid, I would like to greatly modularize uRequire, but I am afraid I dont have the resources(!) to do it right now.

The module documentation/example could provide sample DSL files that generate outputs,

Check http://urequire.org/resourceconverters.coffee#default-resource-converters

[kmalakoff]

Makes sense. I think the benefit of using streams and pipes is less about the performance, but the modularity and way to customize a modular, conversion pipeline using a forward-propagating network graph.

You should compare gulp (stream-based modules and pipe-based aggregation) with grunt (task importing with multiple option block configuration and tasks aggregation) and RCs (DSL with the notion of watching) from an API and code-sharing standpoint, to see what the benefits and differences are and where each is best since they are all specialized to solve different problems or their specialization makes them stronger at solving different problems.

One thing that you should bear in mind is that today gulp may not have state in its nodes for incremental updates, but there is no reason why someone wouldn't write caching extensions for the files that each node in the stream could register and check before recalculation during watch cycles or even across runs.

[kmalakoff]

I guess we posted at the same time again.

We all have too many ideas and not enough resource to implement them! It's unfortunately too easy to come up with good ideas!

Thanks for pointing me to the examples. Maybe putting them in a git repository with a cli for them would help people experiment and contribute.

[anodynos]

someone wouldn't write caching extensions for the files that each node in the stream could

Still, you are leaving a lot outside of what the core should handle, having incompatibilities and the such.

I use RCs extensively (check an O/S example and although they are not (yet) a complete-plugin-community-driven-infrastracture, they are awesome in expressive transformations with only a few lines of call back code, with a rich transformation/manipulation API (mainly for Modules, but can do anything else).

The main problem with RCs at the moment is a) they aren't asynchronous b) they aren't published / reused

[kmalakoff]

Understood. I've looked at the example and the DSL is very verbose. I think that means for new people coming to it that it can be a little overwhelming (it is probably easier for you because you wrote it!).

It would definitely could be improved by a simple modular approach on npm with little effort. For example, you could publish your default-resource-converters in a separate npm module (they look too small for a module per conversion? if so, I'm not sure how each one should be named and referenced - maybe named properties) so that the DSL learning curve is only for experts.

I guess my main thought is that one of the benefits of gulp over grunt is that there is less to remember (fewer options due to pipe-based composition), and the DSL approach seems to push into even a deeper learning or referencing curve which feels "expert" so publishing on npm for something like 90% of use cases where people can just reuse existing DSL work would be needed to unleash the benefits of a DSL approach.

[anodynos]

Hi @kmalakoff

the example and the DSL is very verbose

I would greatly appreciate it if you could elaborate on which parts you 've had problems understanding or think they could become less verbose and improved.

Please do note that uBerscore is used as a uRequire testbed & the Gruntfile.coffee is deliberately long cause it serves as (multiple) examples / testbed. It could be easily reduced to a couple of essential builds with only ~50 lines.

It would definitely could be improved by a simple modular approach on npm with little effort. For example, you could publish your default-resource-converters

Yes, I would love to split+publish npm packages and make uRequire more modular / platform-y but I have little time currently :-(

one of the benefits of gulp over grunt is that there is less to remember

I agree, I dont like grunt configs either, cause they tend to be huge and quite repetitive.

Note that uBerscore's Gruntfile.coffee uses (almost) solely urequire facilities (instead of grunt-*plugins) but performs many different tasks, such as coffee/coco/ls compile without any setting or as easy as optimize: true or copy: /./ etc.

I personally think gulp is a nice example of pipes :-) but compared to uRequire which is firstly a Module manipulator (but can also manipulate resources) combined with a task runner like Grunt is far richer build system for a library like Backbone-ORM. Simplicity is a welcomed attribute, if no significant feature lack is imposed :-\

With your current gulp build you're missing out many different features, multiple builds, the mocha/phantomjs & browser tests, automatic versioning, bannerin one place, cleaning, proper watching etc - so its unfair to compare verbosity :-( Also if you wanted anything more advanced (eg manipulate modules like inject/remove/replace deps or code) you simply can't do it:-)

Anyway, check out this commit which has a uRequire only set of builds.. quite small and clear I think. Just hit $ grunt test_web and open build/test/web/SpecRunner_unoptimized_AMD.html (and the other HTMLs there) and enjoy mocha on a browser (also runs on phantomjs ;-)

I think that for a project like Backbone-ORM, browser testing is a must-have !!!

...publishing on npm for something like 90% of use cases where people can just reuse existing DSL work would be needed to unleash the benefits of a DSL approach.

I have started working on urequire-grunt and it looks v.good so far - it's becoming a DSL-like grunt/urequire 'driver' - it can hugely reduce boilerplate (I've even considered generating the mocha/chai AMD/script HTML for free) - I will let you know when it hits v0.1.0 :-)