version 2

[dominictarr]

I'm working on v2,

here is the basic idea:

• all parts will come from separate modules (sha.js for hashing, whatever @jducanator or @jdeblese publish!)
• what test frameworks a submodule uses is up to it's maintainer.
• all (non-dev) deps will be fixed to an exact version. maintainers should post an issue when you push a new version, so code can be reviewed.
• to test, you should generate expected results with node (i.e. openssl, a validated crypto implementation), save that as JSON (use base 64) and then require('./data.json') this will make it easy to run exactly the same tests in node and the client. If there is a standard set of vectors, with matching output, you should use those, if not generate some from node.

I minimum set of operations I'd like to support:

• createHash (sha1, sha256, md5) (done)
• createHmac (as above) (done)
• randomBytes (done)
• pseudoRandomBytes
• createDiffieHelman (use @jdeblese or @jdcanator's stuff, when they publish module)
• createCipher (aes) (probably port from forge or crypto-js)
• createDecipher (aes)
• createSign (rsa)
• createVerify (rsa)

this will give a minimal useable subset of the node api, although, the set of algorithms is lessened.

@jducanator @jdeblese @chrisdickinson @brianloveswords @tonistiigi @juliangruber

[dominictarr]

work has started here: https://github.com/dominictarr/crypto-browserify/tree/v2

which I'll merge once I have a binary string to buffer module (it's easy but there isn't one on npm yet!)

[jdeblese]

Sounds good. I'll see if I can set aside some time to make a lightweight DH module. I expect @jduncanator's version will be more complete, however.

[jduncanator]

I'd almost say you were better off submoduling others Git repos into yours.

[dominictarr]

@jduncanator that is a separate discussion. it's they should still be published to npm as a normal module, and should be installable on their own. You could even check in node_modules and/or use npm's bundle deps. Publish a diffie-helman module, and then we can discuss the best ways to include them into crypto-browserify.

[jduncanator]

I'll publish it as soon as my colleague is finished looking over the code to make sure its all done correctly :smile:

[dominictarr]

okay, great

[jprichardson]

I started https://github.com/cryptocoinjs and use Browserify for just about everything on the client. Maybe we might have some overlapping goals?

[dominictarr]

@jprichardson yes, definately! A large part of what you need crypto is dealing with encodings and binary correctly, and unfortunately, most of the JS crypto doesn't do this very well, which is forgivable, because JS hasn't had a good way of doing binary until recently. Node.js on the other hand, has always been about doing that stuff well - so, the approach here, and my approach in general, is to aim to have the browser be as much like node as possible. If you feel that too, then we can definitely work together!

The more we can share and reuse stuff, the easier it is to optimize it!

[dominictarr]

@jprichardson oh by the way, @kyledrake is also doing overlapping stuff in the https://github.com/kyledrake/coinpunk project!

[jprichardson]

@dominictarr Yes, I think that makes a lot sense. I was leaning in that direction before I discovered your work with crypto-browserify. It seems to me that a browserify organization should be used to group and organize this effort. I created one and added you and @substack as owners. You can delete me if you don't feel that I should be apart, you have full control as well :) But, we could shuffle all of these modules there and start aligning efforts.

Thoughts?

[jduncanator]

I'd like to maintain this repo as the main repo, considering its already used and integrated with Substack and Browserify, however I suggest submitting pull requests if you want some or all of your repo integrated :) Thoughts @dominictarr?

[dominictarr]

@jprichardson so I know that @substack prefers to use his own account, and dislikes github organizations. I could go either way, but I think a browserify org wouldn't float unless @substack was into that. An org for crypto-browserify is a different matter, I'm not sure it's really necessary, but we should certainly discuss it. Of course, crypto stuff is much more serious than most modules, if an org helps with code reviews I think that would be the best argument for it.

[dominictarr]

okay I have published version 2, since sha.js now supports binary encoding, and passes the same tests as before. I have also deleted some tests, including the vector tests, since those are now in sha.js

closing.

@jduncanator how is diffie-helman coming?

[jduncanator]

@dominictarr Should have a modules published tommoz :smile:

[dominictarr]

@jduncanator can you publish what you have?

[jduncanator]

Sure can. Sorry, been busy with another Node project that has had deadlines. I'll publish it without tests asap.

[jduncanator]

@dominictarr Library can be found here: https://github.com/jduncanator/dh-browserify and under dh-browserify on the npm registry.

[kyledrake]

Just to throw this out there, it would be amazing to have a pbkdf2 implementation with this. Browser password hashing is a very important component for doing client-side encryption, and we can't really go over 1-3k iterations right now due to general slowness. Also, just being able to do pbkdj2 in node and the browser would be awesome.

There's an implementation of this in sjcl right now (the cryptojs version is much, much slower) that could be useful.

[jduncanator]

@kyledrake Take a look in my fork, there is a pbdfk2 implementation in there :smile:

[dominictarr]

that still depends on the old sha1 implementation, so that will be slow.

[jduncanator]

I did benchmarking and it hit just below the speed of PHP? I'm not sure whats going on for you...

Although that was run in the latest git version of chrome...

[dominictarr]

javascript needs to be more secure than php, so we gotta get that faster.

[jduncanator]

@dominictarr That statement is 100% incorrect. Platform has nothing to do with security when you are talking about things like speed. Also, more iterations does not make an implementation more secure, it makes the overall output more secure. You will never get JavaScript as fast as PHP as it has direct bindings to OpenSSL and runs all the functions natively. The best we can do is try and make things faster. At the end of the day, just because JavaScript takes 1 second to compute a PBKDF with 1000 iterations compared to PHP which takes 500ms does not make it less secure, it makes it less efficient and efficiency has nothing to do with security.

[dominictarr]

If something with openssl bindings takes 500 ms to do 1000 iterations that is insanely slow. node's openssl bindings can do 10k iterations in 11 ms, and forge can do 10k in 416 ms. (on my laptop)

https://github.com/dominictarr/crypto-bench

where you including the time to start and exit the process?

Efficiency is related to security because the more efficency you have the more security you can afford. Crypto is only "secure" because it's computationally infeasible to reverse, if you have a slow crypto implementation, then you are on the backfoot. Someone with a faster implementation (and/or faster hardware) will be able to bruteforce your crypto before they would have if you had faster (i.e. more) crypto. A 10x improvement probably means several years more privacy. (assuming that everything else about the implementation is correct)

[jduncanator]

@dominictarr, That was a purely hypothetical example to get my point across.

Whilst your argument about efficiency being related to security is correct for the offensive side, it is not correct for the defensive. Your argument is that if we implement something slow, that it then becomes faster for my attacker to "crack" my passwords than if I had have implemented a faster algorithm. The whole point to KDF's is that the amount of time to do ONE set of 10,000 iterations is a lot less than someone who needs to do 10000000x 10,000 iterations. Even if our implementation took 10 seconds to do 10,000 iterations of PBKDF and an attacker could do 10,000 iterations in 1ms with appropriate hardware, it will still take them thousands of times longer to crack your password than it would for the one time (or less) per day that a user will need to run the KDF to login. That is the whole point to KDFs, a little time trade off for the user that reaps massive consequences to an attacker. Speed is not going to be our enemy here.

If I were developing implementations for an attacker to use (which need to be fast to be of any use), I wouldn't be using JavaScript...

Anyway, PBKDF2 is old and if you were really thinking about doing crypto, you'd be using scrypt or at least bcrypt by now.

[dominictarr]

Ah, okay. that makes sense.

[dominictarr]

right. so the biggest direct benefit of performance will be user experience.

[jduncanator]

@dominictarr Correct. At the moment our MD5 implementation is about 4x slower than OpenSSL's which is extremely good for an implementation that isn't completely unrolled. A little optimization for a JavaScript runtime engine and we should be able to double that.

[dominictarr]

Yes. Although I think the think the real difference will probably be closer to 40x slower. If javascript was only 4x native that would be amazing.

I was able to improve on the old sha1, sha256 implementations by 10x (for large inputs) but it's still 20x slower than node's openssl.

[jduncanator]

@dominictarr I just ran benchmarks and MD5 can do about 100k iterations per second. Node.js does 400k. Thats 4x slower which is pretty decent.

Edit: Just reran it, got some different numbers, still roughly 4x slower:

MD5

83455.20838487164 iterations/s

Node MD5

345902.231169681 iterations/s

[jduncanator]

Here is what I got running my benchmark script:

MD4

114939.0 iterations/s

Node MD4

443292.9 iterations/s

MD5

95883.6 iterations/s

Node MD5

408225.7 iterations/s

SHA1

76253.8 iterations/s

Node SHA1

365100.7 iterations/s

SHA224

44646.5 iterations/s

Node SHA224

324781.8 iterations/s

SHA256

48309.7 iterations/s

Node SHA256

346193.4 iterations/s

[dominictarr]

@jduncanator can you post the benchmark script so we are both looking at the same thing? A table of numbers isn't very useful compared to a benchmark that I can run my self.

Hmm, I realize now (after doing crypto-bench stuff) that iterating a hash is one thing, and hashing a large object quite another thing. for example, sjcl is quite fast at iteration, but not fast at a large file. sha.js is fast at a large file, but not so fast at iterating (although, that is improving)

[jduncanator]

@dominictarr Benchmark script is here: https://github.com/jduncanator/crypto-browserify/blob/master/support/benchmark.js

To run it, run npm run benchmark in the folder with the package.json :smiley:

[dominictarr]

thanks

[jduncanator]

No problem

[jduncanator]

@dominictarr After a bit of full on work, I've finally had some time to "relax" a little and pump out a new and revised MD5 implementation. This one is completely hand written and uses Node Buffers for maximum speed. Its not implemented in the actual library yet but you can see the new code in this (3501f33) commit. It all works and I've tested it against the actual NodeJS API, against OpenSSL and against the MD5Sum program. Everything seems to work nicely!

[dominictarr]

great, you should make it it's own module so I can add it as a dependency to crypto-browserify

[jduncanator]

Does everything look alright to you? Is it in the direction you wanted to head in?

[dominictarr]

The external API is fine!

you should put more helpful error messages: https://github.com/jduncanator/crypto-browserify/commit/3501f33f62a844475346c7e7e500d23356408468#diff-aec7cba554e3206904eb422d66d87976R184

and also I wrote a thing that maybe you can use here, it implements copying the update(data, enc) into the working block so that the hash function can do one iteration. I did some optimization and got my hash functions much faster.

https://github.com/dominicstarr/sha.js/blob/master/hash.js

for benchmarking use https://github.com/dominictarr/crypto-bench

[jduncanator]

@dominictarr Yes the plan was to implement "subclasses" and this implementation was only a play around with the "Streaming" API idea. The way the current MD5 class is designed is that you simply swap out the transformBlock function for another digest's function and it still works (as most if not all hashes work on 32bit integers).

[dominictarr]

okay cool, you should publish it as it's own module and then I'll add it into crypto-browserify

[jduncanator]

@dominictarr Before I do that I want to port all the other hashes too the same model.

[dominictarr]

my sha library already uses that model

[jduncanator]

@dominictarr I know, but my implementation is already significantly faster than yours (based on your own benchmark script).

[dominictarr]

faster than my sha1? show me the results!

[jduncanator]

@dominictarr The reason its faster is I'm using templates to unroll everything. Instead of using a loop, it actually inlines every single thing. So you get really ugly, but really FAST javascript code. Doesn't matter how ugly it is because its going to get minified anyway :smile:

[dominictarr]

ah, cool i was thinking about doing that!

[jprichardson]

You have any interest in moving this work over to an org? If so, may I propose: https://github.com/crypto-browserify? A lot of these components make sense as separate modules, a Github organization would help keep this work organized. I'd like to bring over https://github.com/cryptocoinjs/pbkdf2-sha256 (relevant for Node v0.12) and https://github.com/cryptocoinjs/aes (still need to implement ebc and cbc on top).

Any interest?

[jduncanator]

I'd be for a Org, however I suggest that when doing so we actually split, tidy up and audit all our existing code. Keeps things maintainable and bug free.

[jduncanator]

@dominictarr Just FYI, I ported MD5 to be inlined and doubled its performance! Its now only around 3x slower than Native!

MD5

66344.9 iterations/s

MD5 Inlined

138452.5 iterations/s

Node MD5

351730.1 iterations/s

I'm pushing all the fancy "Inlined" ones into a subfolder for now until I port ALL the hash algorithms to the new streaming and inlined format. If you wish to modify it, you'll need to run the source code through a C compiler with the "-E" flag to preprocess the Javascript which will give you the inlined source code. *.c.js == "compiled" JS code, *.pp.js == preprocessor JS code.