Use async file operations

[englercj]

Right now all IO is done synchronously. The speed of this library could be greatly increased by doing IO operations asynchronously, and even operating on multiple files at once.

I would be more than happy to write it for you if this library wasn't written in CoffeeScript. I do think it is very important to look at making this library async however, since that is almost certainly why it is so slow.

[anodynos]

Thank you for the interest. Becoming async would be great, as mentioned in #30 & #33.

It would be nice to know the magnitude of speed increase for sync VS async file IO. I was looking for some I/O benchmarks and I found http://afshinm.name/async-vs-sync-io-benchmark-in-nodejs, which unfortunately isn't dealing with file IO. But still it looks like there's a small magnitude of speed increase (~0.1) for single pipeline scenarios, which is good but not amazing.

I m sure async shows it teeth on UI, or multi user/request scenarios, or multiple parallel DB queries etc. But still, I see people using sync for nodejs file IO all the time - for example grunt 4.x uses some sync internally. Perhaps its not such a demise.

That is not to say I dont wish for async for uRequire - there are many benefits to it. I hope uRequire v2.x can become the core of an on-the-fly-compiled-modules-server scenario, where async would be paramount.

To be on the right path, it would be great if someone could provide a small file IO benchmark for sync & async, eg:

• read 100 files from a dir
• Do some work with their contents (eg compile 'em or uglfy 'em etc)
• save them on another dir

PS: dont be afraid of coffee-script - "its just javascript", and it rocks even if you just use it a sugar syntax and avoding ))})}))}; hell!

[englercj]

I think the benefit for uRequire is that you operate on a lot of files in sequence, when those same files could be operated on in parallel. That is where you will get your speed increase.

[anodynos]

I just executed the simplest benchmark scenario that validates that uRequire's slowness has nothing to do with IO (and not being async).

The test is to convert uBerscore v0.0.15 specs, in 4 ways:

Using uRequire 0.6.8beta5 with

    debugLevel: 0
    copy: undefined
    files: ['**/*.coffee']

1) The normal build is to treat all ~40 modules, well as Modules.

• The .coffee source is read as text/utf with fs (I/O: SYNC)
• converted from .coffee to .js (in memory: parsing/converting)
• Their js code is parsed with 'esprima' (in memory: parsing/AST building)
• Their dependencies analyzed with uRequire's Module (in memory: AST traversing)
• Their factory is regenated from AST with 'escodegen' (in memory: traverse/concating)
• They are converted via UMD template (in memory: string concat)
• finally they are written to their destination, as text/utf with fs (I/O: SYNC).

>> grunt-urequire: task 'spec' #1 is done in 5.712 secs (:-)

2) Then we treat all modules ['#asTextResource', ['**/*.js']], (this is actually a ResourceConverter).

• The .coffee source is read as text/utf with fs (I/O: SYNC)
• converted from .coffee to .js (in memory: parsing/converting)
• Their js code is parsed with 'esprima' (in memory: parsing/AST building)
• Their dependencies analyzed with uRequire's Module (in memory: AST traversing)
• Their factory is regenated from AST with 'escodegen' (in memory: traverse/concating)
• They are converted via UMD template (in memory: string concat)
• finally they are written to their destination, as text/utf with fs (I/O: SYNC).

>> grunt-urequire: task 'spec' #1 is done in 1.218 secs (:-)

3) Run with ['&asBundleFile', ['*/.js'] ] with copy: [/./]

• uses copyFileSync http://procbits.com/2011/11/15/synchronous-file-copy-in-node-js/)

>> grunt-urequire: task 'spec' #1 is done in 0.044 secs (:-)

4) Run with ResourceConverter

   (rcFind)->
     (cs = rcFind('#coffee-script')).convert = (f)-> f.source
     cs

which effectivelly disables module treatement AND coffeescript compilation

• The .coffee source is read as text/utf with fs (I/O: SYNC)
• .... nothing really...
• finally they are written to their destination, as text/utf with fs (I/O: SYNC).

>> grunt-urequire: task 'spec' #1 is done in 0.041 secs (:-) 

I think its conclusive: SYNC IO is not the real speed problem, its the parsing/genarating etc.