Building sha256c.so and f32-Linux-*.so with critcl 3.x in a modern Linux environment

[eddiegb]

I've been using PG for several years and would like to thank the devs for a fantastic piece of software.

However, for me usage of PG is impeded under Linux by what I see as a big problem, which I'd like to somehow help improve if I can. The problem relates to the included sha256c.so and f32-Linux-*.so libraries. First of all, let me make clear that the inclusion of these libraries in my view is a great thing, and I'm NOT arguing that they shouldn't be. It's a great convenience that gives all platform users, including Linux users, the ability to use PG out of the box with great performance. The problem is that these days it is very difficult for a Linux user to build their own copies of them from the provided sources, because of Critcl. It is also very difficult for at least some Linux distro maintainers to build their own copies for the same reason. At least the Debian package of Password Gorilla excludes the .so files, so Debian users do not get a performant experience out of the box.

Why should users & packagers be able to build their own copies? I think it is incredibly important, especially and particularly for a password manager. For me, I use PG to protect very sensitive data, and for any such purpose I only really trust binaries that I can build myself. (I'm a Gentoo user so I have the luxury of being able to do that to the nth degree, but all Linux and indeed other OS users have the option to easily build tcl/tk themselves from sources if they want to.) If I have no option but to run a binary file (e.g. source code not available) on my personal systems I usually think very hard about whether I trust the person/organisation who provided it.

It's fantastic that you provide the sources for the Critcl binaries as well as the other pure C binaries. In the last couple of days I've been extremely pleased to discover that I can easily build my own copies of twofish.so and stretchkey.so just using GCC and the provided .c files, and the resultant binaries work fine with my Gorilla db AFAICT.

However, I've spent 2 days now trying to build my own copies of sha256c.so and f32-Linux-x86_64.so using Critcl and sadly though I can build them they don't work.

Now, from what I can tell, Critcl these days no longer uses produces .kit files for use with Tclkit (I certainly could not find any on Critcl's github ( /andreas-kupries/critcl ), the current Critcl documentation, or other places like wiki.tcl-lang.org. I cannot find any confirmation anywhere here that it should be even possible to build these 2 libraries using Critcl 3.x. So I'm aware from the outset that this is unknown territory. But I'm willing to put the effort in so this can be improved in PG, hopefully at some point down the line would love to be able to produce a pull request even if just documentation improvements.

Gentoo does not have Critcl in its main repository. So I tried to install it from the latest release on GH. But it has an extremely peculiar installation process which makes it almost impossible to integrate into Gentoo, at least not without lots of modifications.

So instead I've tried to build the 2 libraries on a Debian 11 installation I already have.

I have these packages installed:

critcl 3.1.18.1+dfsg-3 (compiled runtime in Tcl) itcl3 3.4.3-3.1 (incr Tcl] OOP extension for Tcl - run-time files) itcl3-dev 3.4.3-3.1 ([incr Tcl] OOP extension for Tcl - development files) libtcl8.6 8.6.11+dfsg-1 (Tcl (the Tool Command Language) v8.6 - run-time library files) tcl 8.6.11+1 (Tool Command Language (default version) - shell) tcl-dev 8.6.11+1 (Tool Command Language (default version) - development files) tcl-trf 2.1.4-dfsg3-2.1 (Tcl data transformations - runtime library) tcllib 1.20+dfsg-1 (Standard Tcl Library)

So I download the pre160 branch of PG and run the following inside their respective directories:

/usr/bin/critcl -keep -cache B -pkg f32-critcl.tcl

/usr/bin/critcl -keep -cache B -pkg sha256c.tcl

Each successfully builds the library for me. The first produces f32-critcl.so so I replace the bundled f32-Linux-x86_64.so with it. The second produces a "sha256c" directory containing several files including linux-x86_64/sha256c.so. So I remove the bundled tcllib/sha256c (preserving stretchkey.so) and replace it with the newly produced "sha256c" directory, copying the original stretchkey.so back into it.

But as I say, the new binaries built this way do not work, I'm unable to open my PG database, with PG complaining that the password is incorrect (it definitely isn't). Reverting to the bundled libraries from here works again.

There is an unfortunate complication, in that I cannot test PG on the Debian installation as it has no desktop it is just a server. So I'm copying the new libraries to my Gentoo installation where I can test them. The Gentoo installation also has tcl 8.6 and tcllib 1.20, same as Debian 11, similar versions of toolchain packages, and is also x86_64, so the newly built libraries should work. I can also see using strace that PG appears to load them successfully. They look perfectly fine running ldd on them on the Gentoo system.

I suppose it would just be great as a start to know if anyone has been able to build these libraries with recent versions of Critcl? Also, do you guys use any particular GCC flags/options when you build the ones bundled here?

I wonder if I you might share what your thoughts are on how I'm going about trying to achieve this? Can you suggest anything else I can try? I guess you might suggest turning up debugging in PG, which I'm not sure how to do at this stage. I did use strace but nothing interesting was output when it fails to open the db. gorilla.tcl also does not output anything to stderr at any time.

Thanks!

[rich123]

I can offer a few initial comments now, but more than this will need some investigation first.

1) the sha256.c lib fails with critcl -- the sha256.c in Gorilla is a direct copy of the same from Tcllib (https://www.tcl.tk/software/tcllib/). And by "copy" I do mean "copy". I changed nothing from the version copied out of Tcllib. I forget which exact Tcllib version now (lilkely circa 1.12 to 1.14). The critcl I used to build was a of similar age, but it was a critcl.kit, not a standalone, so maybe that makes some difference.

So for sha256.c, there is one thing to try. Try swapping in the sha256.c (and other necessary files) from the newest Tcllib (under the assumption that if critcl changed, maybe Tclllib was appropriately patched to track the change). If this works, then that is a solution.

If it still fails to build, then report as a bug to Tcllib that it no longer builds sha256.c under critcl (and maybe get some attention to have a patch built).

f32-Linux- is mine, and it was built as a critcl extension because at the time critcl was working (for me at least) and building it as critcl was a straightforward way to speed up that inner loop within PWG (and, as well, at the time I had zero experience with creating Tcl C extensions against the native Tcl libraries). I have since created a few small C extensions for my own use using the native Tcl API's directly, and so recoding f32-Linux- as a C code file targeting the Tcl API directly should not be too difficult (and doing so would remove the 'critcl' dependency). As to when I might find time to do so, well that is a question with a not-so-good answer, it may take a while before I can find an open chunk of free time to sit (preferably without being interrupted every 8.72 minutes) and code up the translation. So I can't make any "timing" promises at this moment.

Now, thoughts on "debugging". Most of PWG is Tcl script, so there's no default "debug" mode to switch on. As well, I've been creating programs for myself for going on 40 years now (ugh, that answer make me feel old now) and back when I started (Atari 800 - Atari BASIC) there were no "debug modes" so I'm rather very much a "printf()" debugger (as in I insert "print" statements where needed to observe what's going on, then remove them when I track down the bug). So you can insert Tcl's "print" statements (the "puts" command) where you would like to observe what is going on.

Also, the fact that the compiled .so files load implies the "compile" worked. The fact that something goes wrong when using them implies a mismatch somewhere in the call in or the return from the C library. So first step is to work out which one. Disable them all (rename them something like *.so.off so they are not found) and then enable one at a time and try decrypting a DB. I suspect you may find that only one is causing trouble, and if that is the case, then you've now cut the detective work down by 2/3 vs not knowing which is the cause.

Once narrowed down, you can dump ("print" statements) data before and after calls on both sides (Tcl and C) and see if there is a miss-match. I suspect it will likely turn out to be a subtle type difference somewhere (int vs. unsigned, 32bit vs 64bit default sizes, some change in libtcl's assumptions, etc.) that is resulting in a data change that makes the hashes fail (and if the hashes fail, you get the "can't open database" error).

[eddiegb]

This is super helpful, so thanks very much :-)

I've just checked the contents of tclib/sha1 in PWG versus what's in the current release of tcllib and a diff of each shows small changes to each of sha256.c, sha256.h, sha256.tcl and sha256c.tcl. So I'll try using the newer files to build sha256c.so and stretchkey.so and will report back.

Really appreciate the pointers toward "debugging" the TCL code also, that will help a lot if I need to start doing that later.

A burning question that comes to my mind is; generally speaking, very roughly, how much of the PWG TCL code relies on calling these external crypto libraries?

Reason I ask because in recent years, especially following the Heartbleed vulnerability, I've seen a general consensus arise amongst crypto people against "rolling your own", for good reason I think. Now I know that PWG has not done that here, you've taken the crypto libraries from tclib, which I think was a very good approach years ago, given I've seen examples of other projects who tried to literally write their own crypto entirely within a scripting language. And the sha256.c is from Freebsd so a very good choice. I can also see how the solution was arrived at by trying to remain as much within TCL as possible and so using tcllib. However, the problem with lifting these libraries from tclib into PWG is you then create a maintenance burden for yourself to keep them uptodate with upstream tclib. I also notice that tclib has a good deal of their own TCL code written to interface with the C code which has to be maintained too, as well keeping the C code uptodate with upstream developments, not to mention the critcl layer. All in all it seems to me not that ideal.

So, back to my question, if you said that the TCL code calling these external libraries isn't all that much, relatively speakking (e.g. maybe it's only happening on opening/saving the db?), I wonder if it's worth considering simply replacing that with calls to e.g. openssl executables on the user's system? Wouldn't that in fact simplify your life somewhat, especially in taking away the burden of worrying about some unknown bug in tclib or critcl defeating the crypto one day?

Such an approach might be much easier on some platforms than others. But maybe I could look into how that might look like under Linux depending whether you think the general idea is good?

[rich123]

generally speaking, very roughly, how much of the PWG TCL code relies on calling these external crypto libraries?

Relies? None. The shared object/dll modules are purely "speed" enhancements. Password Gorilla's original author (Frank P.) created Tcl versions of the encryption algorithms in order to have a "script only" password manager. I and the late Zdia added the C extensions much later, after Zdia took over from Frank as maintainer, as performance enhancements (as performing low level encryption routines in pure Tcl script is much slower than the equivalent coded in C).

So all of the shared objects can be disabled, and PWG should continue to open safes and function, just much more slowly than with the shared objects loaded.

I've seen a general consensus arise amongst crypto people against "rolling your own", for good reason I think.

This is true, and it was true long before all the "bleed" and other issues arose in the last few years. However, at the same time, this is the "simple, 10,000 ft, understandable to the general public" statement. Given that most developers who might try will not have sufficient background to understand the details, the reasoning is simplified to "don't, use this or that instead". If one is to have a "script only" password safe compatible tool, one has to create the required cryptographic algorithms in that scripting language in order to accomplish the goal of a script only tool.

However, the problem with lifting these libraries from tclib into PWG is you then create a maintenance burden for yourself to keep them uptodate with upstream tclib.

True, however tcllib is relatively stable, and so the burden is low. So far your issue here is the first report of some "divergence" somewhere causing a build issue. Note that I can not comment on any possible unreported divergences.

I also notice that tclib has a good deal of their own TCL code written to interface with the C code which has to be maintained too, as well keeping the C code uptodate with upstream developments, not to mention the critcl layer. All in all it seems to me not that ideal.

Tcllib has the same goal as Frank P.'s goal for Password Gorilla. Tcl script only implementations of all the modules, including the cryptographic ones. The critcl modules in Tcllib for some of the cryptographic primitives exist for the identical reason that PWG has the modules you are trying to rebuild - performance.

As to Tcllib's maintenance burden, while true they do have one as well, that is one that we here working in whatever way on PWG can generally not concern ourselves with (unless we go about fixing them and submitting patches).

So, back to my question, if you said that the TCL code calling these external libraries isn't all that much, relatively speakking (e.g. maybe it's only happening on opening/saving the db?),

The heaviest usage is, obviously, during open or save of a safe file. But the crypto routines are also used to obscure the data within the safe while it is in memory within the running process. Granted, a determined attacker will eventually be able to bypass that protection, but doing so is at least some small impediment to non-determined attackers.

I wonder if it's worth considering simply replacing that with calls to e.g. openssl executables on the user's system?

Most likely, no, for at least the following reasons:

1) not all systems have openssl executables installed (i.e., windows does not ship with the "openssl executables" in its default from MS, as far as I am aware, as to MacOS, it 'might' have them, I have no Mac with which to find out)

2) openssl typically does not include a Twofish algorithm (which is required to access Password Safe version 3 format files). I just checked the output of "openssl list-cipher-algorithms" on my system and I did not see any listing that led me to believe it contains a Twofish implementation. Now, it is possible that Twofish is a "compile time option", and if so, my distribution's (Slackware) default was to not enable that option. Therefore it would be a reasonable assumption that other distributions may also not enable it, which would likely mean no Twofish being available from the openssl executables on multiple Linux distributions.

Such an approach might be much easier on some platforms than others.

Yes, and that is an understatement. Using openssl executables would, likely, only be reasonable under Linux and/or FreeBSD. It would be a non-starter for Windows and I have to say "I don't know" for MacOS. Which would mean that for PWG to maintain "windows compatibility" it would have to carry along its existing pure Tcl and C performance libraries anyway, in which case they may as well also be used under Linux and FreeBSD. I.e., doing so would simply be adding additional complexity unless PWG also, at least, dropped windows support (and I'm sure those who do use PWG on windows would say "no, don't drop windows support" here).

But maybe I could look into how that might look like under Linux depending whether you think the general idea is good?

I think you'd be adding additional complexity for little gain. For a Password Safe that is simply sitting on disk unopened, the weak point is most likely the master password. So an attacker specifically targeting your 'safe' would prefer to simply retrieve it to their own system and then run a "hashcat" attack using something similar to this: https://gist.github.com/epixoip/a83d38f412b4737e99bbef804a270c40 or install a keylogger to try to capture your master password, whereby they don't have to run a 'hashcat' attack.

For worries about the various 'bleed' attacks, if the system is running some code performing a 'bleed' attack, and the system has not had any 'bleed' mitigations installed, then the bleed can occur from the openssl executables just the same as it might occur from the PWG process. In fact, as most attackers would focus on openssl (because of its common usage) the 'bleed' code is likely to be focused on openssl, and not ever consider Tcl script code, so it might not even "look at" tcl script code, or it might just see "noise" because the script code is not storing/processing the data in the same manner as the openssl routines. So it is possible that doing so might convert a "security by obscurity" (which is, arguably, not real security) situation into a directly successful attack.

[eddiegb]

OK, I admit I glossed over somewhat the difficulties of producing something that runs on 3 platforms :-) Yes, that's a monumental task. The combined efforts of PWG developers have produced, technically, a very, very good solution to the cross platform challenges by using TCL, and the chosen C code for acceleration, leveraging tcllib and critcl.

Just 5 years ago I would not have taken any issue with the approach and would have agreed with almost everything you've said. Today, right at this very moment, PWG meets (what I personally consider) the main requirement of storing secrets on disk using strong encryption and providing a great interface for managing them.

However, the more I've thought about it since discussing it with you, I think it's almost certain that changes in the world of cryptography are going to make the current approach impossible for you to sustain for more than only a few more years. Purely because, right now, you don't have the flexibility available to adapt to such changes.

Let me use another famous application to illustrate what I mean. OpenSSH 9.0 was released 3 months ago with one significant change; it now uses the "hybrid Streamlined NTRU Prime + x25519 key exchange method by default" in order to be able to resist quantum computer attacks in the future. Unfortunately all the indications are that Quantum computers are going to become a huge headache for all of us not at some point in the distant future as was commonly thought but in the next 5 or 10 years. The impact on PWG is simply going to be that a Quantum computer will be able to brute force our sha256 encrypted safe files in minutes not hours. Who knows what will eventually become the best choice of algorithm but almost certainly in just a few years time sha256 will be considered as weak as md5 is today. Anyway, following OpenSSH's change I'm planning over the coming months to start regenerating all SSH host keys on all servers I manage using the new defaults, which is going to be a major undertaking which I'm not looking forward to.

I'm not trying to compare PWG to OpenSSH, very different applications I know. But imagine if OpenSSH required a major development effort to support a new key exchange method because it had relied and coded around one in particular? My point is by adopting an approach now that's not strongly tied to one algorithm, when the time comes to retire sha256 you will have much less of a headache.

I did use the wrong words by asking you how much the code "relied" on the external libraries. Yes the external libraries aren't a requirement with the current codebase. However, from my own personal point of view, without the acceleration libraries PWG would probably become intolerably slow, so in practise I see them as effectively a requirement. In fact, I'd go as far as to say PWG without acceleration is just not possible in the long term when you think about how computationally expensive cryptography is by nature.

Anyway, sorry to have shifted the topic somewhat on this issue report!

[rich123]

My point is by adopting an approach now that's not strongly tied to one algorithm, when the time comes to retire sha256 you will have much less of a headache.

I suspect based upon your paragraphs above this quote that you may have inadvertently glossed over another detail. The Password Safe file format that PasswordGorilla utilizes is also not our (PWG's) creation.

It is the native file format of PasswordSafe (https://www.pwsafe.org/) and so we use Twofish and sha256 because that is what is used to encrypt/hash the PasswordSafe file format. And to the best I understand, another of Frank P.'s goals when he created PWG was a cross platform, script only, PasswordSafe compatible tool. Now, as time has moved on, PasswordSafe the application is also now cross platform (I have no idea if it was cross platform when Frank began PWG) so some of Frank's reasons may no longer exist. But the fact that we (PWG) are using PasswordSafe's file format means it is not that we are "tied to one encryption algorithm" but rather we are "using the algorithms that are compatible with that (PasswordSafe's) file format". And, one of the huge advantages of using that format is we don't make those mistakes that are why so many in the crypto world assert: "don't roll your own cryptography". The PasswordSafe file format was originally designed by Bruce Schneier (yes, that Bruce Schneier) so it at least can be seen as having been designed by someone who clearly knows what they are doing cryptography wise. Granted, I would expect Mr. Schneier would make different choices now, were he designing the file format today, than he did then but at least it was designed properly, for the time it was designed.

As to quantum computers, yes, if the quantum apocalypse occurs in five years, then lots of change will have to happen, but also if it does, everyone, us, PasswordSafe, the rest of the world using conventional crypto, will all be in the same, sinking boat. But as we are compatible with PasswordSafe files, we somewhat have to follow PasswordSafe's lead in that regard.

However, from my own personal point of view, without the acceleration libraries PWG would probably become intolerably slow, so in practice I see them as effectively a requirement

Agreed, that is why the binary extensions are there, so that the tool is not intolerably slow. Granted the advance in CPU horsepower over the years means that today's Tcl, running on an i7 or i9 class CPU, is much faster than Tcl, running on a Pentium2, 3 or 4 around the time that Frank P. first created PWG, but it is still slow vs. using the binary extensions for the encrypting and hashing work. So yes, they are all but a requirement for a reasonable user experience.

[eddiegb]

Gosh, yes I'd completely forgotten all about the long history of the project! And now I'm reminded how I came to be using PWG today, which I'd also completely forgotten. I first started using Password Safe in the 00's, by running the then Password Safe GUI application under Wine on Linux (the Linux version was very early beta back then). And I remember well now that Bruce Schneier's involvement was one of the things that persuaded me to choose it. Then about 10 years ago fed up of running the beast which is Wine on my personal machine so finally decided to look for a non-emulated way to manage my PasswordSafe db, which I'd come to rely upon a lot by then. I did consider the Password Safe Linux version again at that juncture, decided against it for reasons I cannot remember, and my search for something else led me to PWG which I've been using a number of years now. But it's funny how in my mind I've come to see PWG as the whole integrated package.

So, yes, I see now you do have somewhat of a constraint there if PWG wants to only stick to the PasswordSafe db format. And maybe I should have addressed my comments to the PasswordSafe project. However, I would say there's nothing stopping PWG becoming an application which supports both the original PasswordSafe db format as well as perhaps developing it's own alongside it, a new PWG native format. And this does not have to be in any way a "roll your own crypto" type endeavour. I'm sure you would agree there can be a very clear separation between a db format, the GUI that manages it, and the crypto used to protect it, which can be entirely taken care of by a 3rd party solution such as openssl which is something designed primarily to be used in that way. But of course that's something for someone to step up and develop whoever has the time, resources and desire to do so, assuming that person even thinks it is a good/worthwhile idea, whether as a pull request here or fork or whatever. It's just a thought I'm leaving at that without any expectation about anything from anyone.

Thank you for being very willing to engage with my comments rich123, I really appreciate it and it is very impressive to see in an oss project. In all honesty my concerns about the algorithms and implementation of them have led me to question if PWG/PasswordSafe is something I'll continue to use, which I need to have a good think about now. If I decide to continue, I'll certainly try what you originally suggested about building the .so libraries - the original technical issue I have strayed from - and update further here how that worked out.

[rich123]

as well as perhaps developing it's own alongside it, a new PWG native format.

Ah, but, you see, that idea right there is sitting smack in the very center of what the real actual underlying meaning of "don't roll your own crypto". As I said earlier, the common phrasing "don't roll your own crypto" is really the simplified, easy for the lay-person to understand statement. But what it really means is not (as it sounds): "don't write your own implementation of AES in language Y". The real actual, more complicated, would confuse the layperson, meaning is, don't combine these crypto primitives in novel ways that are also likely insecure. I've dabbled in the area enough to know enough to be dangerous, and as well to know enough that I don't know enough. :) More cryptographic breaks occur due to insecure combinations of the various primitives (ciphers, hashes, hmac's, etc.) than to actual insecure implementations of each primitive itself. I.e., the cracks occur when the basic building blocks are snapped together into a new system design, and the cracks appear at the connections between the blocks. One example, there are several ways to work out the key to an encrypted message that uses a block cipher if the padding for the final block is not done correctly. There is a bit more required than just "bad padding", but there are techniques that can allow one to work out the key bits by manipulation of the padding used to fill out the final block. As well, each of the different block cipher modes has their own pros and cons, and for some, the cons can open cracks here or there if used in the wrong way. I known I am in no way properly knowledgeable enough to even be aware of most of the "bad ideas" in order to layout a system of connecting the primitives together to create a final "new PWG file format" to be in any way confident I did not, by lack of knowledge, leave a gaping hole somewhere in the design. That's the real meaning of the "don't roll your own crypto" phrase. Don't design your own system of connecting the primitives, because you'll likely leave a gaping hole somewhere.

Alternatively, I am confident that Bruce Schneier was more than fully qualified to design a file format that at least left no easy gaping holes at the boundaries between the crypto primitives and how they were utilized. So I'm reasonably comfortable that the PasswordSafe file format does not have any cracks around the edges. I would not at all be comfortable making that same statement about a new, PWG specific, format, because I know enough to know I don't know enough.

In all honesty my concerns about the algorithms and implementation of them have led me to question if PWG/PasswordSafe is something I'll continue to use,

Indeed, that is a decision that you will have to make for yourself.

Thank you for being very willing to engage with my comments rich123

You are welcome.

[eddiegb]

I think what you are saying is absolutely correct if you are trying to do something relatively ambitious in the first place, and here you and I are probably picturing quite different things when thinking about a new PWG file format. I dare say what I am picturing is probably much "dumber"/simpler than perhaps what you are picturing. Let me try and explain my picture: it would begin by taking the current PWG codebase and stripping out everything and anything related to cryptography in the code, so that what you end up with is a version of PWG which works exactly as it does today but which saves a file with no encryption or authentication whatsoever. The task of then making safe this resultant database file using crypto would be entirely farmed out to a 3rd party library such as openssl, or whatever may be the best solution which can simply be handed a single file and saves it encrypted to disk. This would leave PWG to concentrate on what it has become very good at, which is being a GUI written in TCL for managing and storing the type of data we want it to. Of course the user experience would be exactly the same; they would open an encrypted file with PWG, PWG would ask for the master secret, and then manage the calls to the external crypto solution.

However, I suspect one problem with this which you might point out is that, as you have alluded to already in one of your earlier messages, it seems PWG relies on crypto to some extent to frustrate the attempts of a bad actor on the same machine trying to read data PWG is handling in memory? Of course that's a whole other kettle of fish which would not have anything like such a simple solution.

But the task of simply taking an unencrypted file and using existing tools and well known methods to encrypt it on disk is definitely something that is within the reach of us mere mortals and not just people with extensive crypto knowledge.

You also might argue that such an approach would require much too invasive changes to PWG which would take it too far away from supporting the PasswordSafe. I would hope that supporting both the existing PasswordSafe format as well as what I am picturing (or some version of it) might be possible but being completely unfamiliar with the PWG codebase I'm probably being completely naive in thinking this would be possible because, as always, the devil is in the detail :-)

[kunwarpart]

How can I install hashcat in windows 7 32 bit?

[rich123]

@kunwarpart I have no idea, as I have nothing to do with hashcat.

Perhaps you might try asking the hashcat team at https://hashcat.net/hashcat/