Asynchronous code in pre-hooks.

[andrewdeandrade]

Looking at the prehooks, how exactly does one perform an asynchronous operation before calling add(). From what I can tell add takes an object as it's first and only argument, so I can't figure out how I should handle any errors in my asynchronous code in the hook. e.g.

db.pre(function(record, add) {
  asyncOp(record.key, function(err, otherValue) {
    if (err) { return /* what do I do here? */ }
    add({
      prefix: subDb, 
      type: 'put', 
      key: otherValue,
      value: record.key
    });
  });
});

db.put(key, value, myCallback);

Looking at the docs for level-hooks, it appears that you can do add(false) to cancel the operation. It's not clear from the level-hooks documentation what happens with the original callback (myCallback in the example above) when an operation is cancelled. Does cancelling the operation return an error to the callback? If so, it seems like information about the error would be lost, making things harder to debug.

(in hindsight, I probably should have filed this in level-hooks, however it would be nice to add a link from the sublevel readme to the level-hooks docs)

[dominictarr]

async code is not supported in (pre) hooks. I have looked into that, and tried a few things (see the async2 branch) but I decided that it added a lot of complexity, without really presenting something useful.

can you describe what you are trying to do? maybe there is a different approach that would work better.

[andrewdeandrade]

I have a manifest file I want to read (e.g. a package.json or bower.json) and there are a lot of values I want to calculate based on the contents on that manifest file and I want to store those in the database.

If the manifest file content changes (I check this via an fs-stat check that checks the file shasum in a content-addressable store if the mtime changed), I want to either invalidate all the the cached values based on the previous copy of the file or even better pull all the values from the DB for those dependent values, check if they've changed and update all those cached values accordingly.

Basically there are a bunch of cached values and there is a directed acyclic graph of dependencies in the cached values and I want to update all the cached values when a value upstream has changed. I was hoping to use pre-hooks to clean out all the cached values (and suffer a cache miss when they are needed (lazy evaluation)) or possibly actively recalc the new values and store them in the db so I don't suffer a cache miss next time.

[dominictarr]

what if you just used fs.watch to keep detect changes to the manifest, and keep that data in memory (since it will only be a very small amount of data) then you could have a sync pre hook.

[dominictarr]

hey, what sort of thing are you gonna build with this? One thing that I want to add to npmd is a thing that detects when some one updates a module B that your module A depends on, and then runs A.tests with the new version of B installed.

It kinda sounds like you might be thinking in this direction...

[andrewdeandrade]

So the interface I built for my virtual filesystem that detects changes already accepts watching data from chokidar or mikeal's watch. I didn't want to use watch as the only interface because I want all the caching to be persistent instead of only in memory. What I do upon startup, is perform a quick scan of all folders watched and check which ones have changed. This is just like what git does. This virtual filesystem already does something like your content-addressable-store module, which is why I asked about levelup support.

Basically I'm building tup for javascript right now, but the basis of everything is just like npm. The core of what I'm making is a generalized package manager that supports npm, bower, component and some other ideas we're exploring at famous. All the pieces that are different between those package managers are fully pluggable, so you can explore different strategies for all the parts where ideas differ. Furthermore, it's all designed from the bottom up to be as fast as possible.

Long term, the Javascript community is really big. Like open web big, and we would all benefit if we ended up with a situation like what happens with the browsers (user agents) and the w3c. The way I see it, everything that has been done with npm is like Mosaic and HTML 1.0 and it would be valuable to have a package manager 100% compatible with npm and the npm ecosystem, but which was more easily modifiable to explore different directions.

The testing idea you described is already on the roadmap. I want to be able to run all the tests of everything my module is dependent on.

I've been meaning to catch up with you, but I haven't been to a beer.js in ages.

[dominictarr]

Oh Wow! This sounds very interesting. I'm also working on making npm more hackable, although I'm not so interested in bower or component... I feel like making something that generalizes over those 3 will be very very general indeed. But I'm keen to hear what you have to say.

Another thing that you should look at is my npmd-cache module. This is the heart of my new npmd rewrite, that uses content addressable store to make a more precise cache.

There is a resolve (npmd-resolve) stage, which figures out the tree of modules that need to be installed, then that is passed to the install (npmd-install) stage, which actually puts them into node_modules directories. npmd-cache is in the background of both of these stages, if any url is requested the tarball it returned is saved in the CAS, and the url -> {hash, ts} mapping is saved in a leveldb. Since these urls only change sometimes - this means you can always get an old value at least.

I'll be in California again for a bit in july - otherwise come and hang out in #stackvm on irc. I'm currently in GMT+10 (Melbourne)

[andrewdeandrade]

Personally, I'd like to make npm modules the gold standard, but our target market is largely front end engineers, many of which are used to bower and some with component. The reason for bridging the three is to have the greatest reach in the front end software engineering community and eventually make it possible to wrap our tool in an sdk for other language backends.

There's still a lot of work still to do, but it's overly general at the end of the day as one might imagine. There are basically a few pieces that need to be pluggable:

• resolution strategy for require() calls
• module folder organization strategy to maintain compatibility with the original package manager (only necessary for placing modules, not retrieving)
• manifest file interpretation (tons of similarities between all the manifest formats so this is less of a problem than you'd think, especially since the approach I'm taking is based on json-schema with key-to-key mapping and optional post json-schema async validations that can be shared among manifest types)

Beyond that everything else is basically the exact same across package managers (or would be if the all the package managers had all the same features of each other).

I'm sure you've discovered in your work as I have with regards to the power of making the package manager more hackable and extensible. We have tons of ideas we're building towards long term, and its really only possible to explore these ideas by starting from scratch and building the base infrastructure we want from day one. However, the general approach we're taking is not at all like what meteor took. Keeping compatibility with npm is a core goal. We're trying to get to the point where you can install npm with famous and famous with npm, i.e. a package manager "quine" :)

I've been looking a lot through the npmd source code and all your level modules, and except for limited comments, I really dig the techniques and ideas and it's prompting me to make a greater effort to break things up into smaller modules and eliminate the use of prototypes and this as much as possible or even entirely. The ideas you've implemented in npmd-cache are pretty similar to what I have planned.

I'm currently rewriting a bunch of core pieces to make things more functional and bake in very aggressive caching from the bottom up. Since we're making performance critical browser code, this is key to being about to quickly benchmark how changes in our core libraries impact modules and apps written by our community.

It will probably be a few months before we have something ready for public use, but when you come in July, I'll show you what we're building.

Are you working on StackVM full-time from Melbourne or are you doing something else full-time?

[dominictarr]

haha, stackvm is a irc channel - not a company. I do freelancing - and then hacking on other stuff in spare time.

hmm, you may have a point here - brower and component resolve modules slightly differently, but they still resolve them. they have a slightly different way of specifying deps - like using github directly... although, npm can install from github too.

so, I have npmd-resolve and npmd install. npmd-resolve generates a json tree that is compatible with npm shrinkwrap would it be possible to represent bower and component modules in that structure somehow? could you just the same structure, but then tell npmd-install to install it bower or component style?