C speedup extension

[GoogleCodeExporter]

It would be useful to add an (optional) C extension to passlib which could 
speed up some of the core routines, ala how simplejson does things.

C backends for mdes_encrypt_block() and pbkdf2() would speed up a number of 
hashes all by themselves.

Additionally, hash-specific backends could be written for: bcrypt, sha1_crypt, 
(apr_)?md5_crypt, sun_md5_crypt, sha(256|512)_crypt, phpass.

However, this would make passlib no longer pure-python, and introduce a number 
of build-related issues, so would need to be done right if done at all.

Original issue reported on code.google.com by elic%astllc.org@gtempaccount.com on 29 Apr 2011 at 7:22

[GoogleCodeExporter]

Original comment by elic%astllc.org@gtempaccount.com on 29 Apr 2011 at 7:23

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Original comment by elic%astllc.org@gtempaccount.com on 29 Apr 2011 at 7:32

• Changed state: Chatting

[GoogleCodeExporter]

Initial work begun on this issue in "speedup" branch. 

Extension currently contains pbkdf2-hmac implementation (required openssl-dev), 
and self-contained mdes_encrypt_block implementation. Only tested under py26 
linux x64

Original comment by elic%astllc.org@gtempaccount.com on 29 Apr 2011 at 8:22

[GoogleCodeExporter]

Original comment by elic%astllc.org@gtempaccount.com on 11 May 2011 at 7:00

• Changed state: Discussing

[GoogleCodeExporter]

I'm the author of https://bitbucket.org/dholth/cryptacular . It has a simple C 
extension for the conveniently public-domain openwall bcrypt implementation.

https://bitbucket.org/dholth/cryptacular/src/8619799362c0/cryptacular/bcrypt/_bc
rypt.c

Original comment by dho...@gmail.com on 20 Jun 2011 at 2:49

[GoogleCodeExporter]

Nice to meet you!

I didn't realize Cryptacular had a builtin bcrypt extension. With your 
permission, I'd love to integrate a copy of that .c file into Passlib - I don't 
have much experience writing cross-platform Python C-extensions, and that would 
definitely help bootstrap the process a little :)

Original comment by elic%astllc.org@gtempaccount.com on 21 Jun 2011 at 4:05

• Added labels: Milestone-Release1.6

[GoogleCodeExporter]

Original comment by elic@astllc.org on 9 Aug 2011 at 5:28

[GoogleCodeExporter]

Original comment by elic@astllc.org on 1 Sep 2011 at 6:29

• Removed labels: Milestone-Release1.6

[GoogleCodeExporter]

Original comment by elic@astllc.org on 12 Sep 2011 at 2:26

• Added labels: Milestone-Wishlist

[GoogleCodeExporter]

Just wanted to say that I like very much how easy passlib is to use and deploy 
an many platforms due to its pure python nature (except the extra packages 
required for bcrypt, for good and documented reasons, which one is not required 
to use).

So, please don't make it a pain (like lxml or pil or ... is) just for gaining 
more speed. Keep in mind that stuff needs to be easily installable be the users 
of the software (not just somehow installable by everyday developers).

For MoinMoin, we could just bundle passlib and it works, no compiler, no binary 
package needed. On AppEngine [or on Windows], one can also just use it (no 
compiler there anyway [usually]).

Original comment by Thomas.J...@gmail.com on 19 Jan 2013 at 1:45

[GoogleCodeExporter]

Thomas -

No worries on that front. The primary reason this project got started was 
because I needed an easy-to-install password hashing library. I end up cursing 
way too much at those same "required for basic performance" C extensions... 
particularly during the part of my day job that involves compiling on Windows.

Towards that end, I can guarantee the pure-python part of the library will 
always support the full functionality, with as much speed as I can squeeze out 
of the hashes (Heck, the pure-python bcrypt backend is almost fast enough to be 
usuable under PyPy 1.9 :). I also plan to keep the ability to just copy the 
passlib package dir wherever it's needed, without requiring setup.py. 

So if I ever do add a C extension, it's going to be for those people who want 
the last little bit of speed; and I plan to have it work like sqlalchemy's C 
extension: that everything works just fine even if the extension fails to 
build. That said, the headache of getting cryptographic C code to compile 
correctly under the cornucopia of environments out there has left me 
unmotivated to pursue this feature so far.

- Eli

Original comment by elic@astllc.org on 19 Jan 2013 at 9:45

[GoogleCodeExporter]

Things have changed a little bit since I last actively pursued this issue back 
in 2011. Since then....

1. It's worked out quite well for passlib to rely on external bcrypt C 
extensions. The two main ones, bcrypt & py-bcrypt, are seeing active 
development, and I don't want to duplicate all the effort of these other 
projects. 

2. When adding support for scrypt under issue 8, a similar use of the 3rd part 
'scrypt' package is probably in order, removing the need for a C extension for 
it.

3. Passlib's builtin PBKDF2, sha256-crypt & sha512-crypt implementations are 
reasonably competitive in speed (passlib's pbkdf2 implementation even seems to 
be one of the faster ones). 

4. The remaining hashes (md5_crypt, sha1_crypt, etc) are deprecated anyways, 
accelerating them would be of little benefit compared to encouraging users to 
upgrade.

Because of all of this, I'm (for now) closing this issue as WONTFIX, closing 
the bitrotten 'speedup' branch, and plan to pursue an explicit pure-python 
policy until a truly persuasive reason crops up.

Original comment by elic@astllc.org on 27 Dec 2013 at 9:00

• Changed state: WontFix