search

elasticdog / transcrypt

transparently encrypt files within a git repository
MIT License
1.43k stars 102 forks source link

Version 2.2.0 is not compatible with MacOS Ventura's `/usr/bin/openssl` #147

Open jmurty opened 1 year ago

jmurty commented 1 year ago

In release 2.2.0 we included a fix for #133 to work around a compatibility problem with OpenSSL versions 3+, which changed how salt values are included in encrypted output (they aren't)

Unfortunately our work-around relied on a check of the openssl version's major version: if the major version is 3 or greater it forcibly includes the salt values in the encrypted output.

The recent release of MacOS Ventura includes the LibreSSL project's openssl command which also reports a major version of 3, but our work around is not necessary for the LibreSSL project's openssl because that project didn't make the breaking change to remove salt values from encrypted output.

The upshot of all this is transcrypt release 2.2.0 is currently broken on MacOS Ventura when using the default built-in /usr/bin/openssl binary, because our work-around will end up prefixing the salt value unnecessarily.

jmurty commented 1 year ago

This should be fixed on the main branch as of commit ca6a4675fbd8693c0f214382a5651a405a1c6843

stherold commented 1 year ago

Same issue here. We installed the main branch instead which contains the merged branch.

alb777 commented 1 year ago

Still failing even doing the workaround above.

jmurty commented 1 year ago

Hi @alb777 can you describe how the latest main branch version of Transcrypt is failing for you?

And are you sure you have installed/upgraded to the in-progress version? E.g:

repo$ ./.git/crypt/transcrypt --version
transcrypt 2.3.0-pre

I'd like to release an official 2.2.1 version soon with the MacOS 13 Ventura fix, but I'd also like to be sure the latest version on main actually fixes the problem(s).

alb777 commented 1 year ago

Hi @jmurty

Sure, follows my current transcrypt state that fails during decrypt/git push to stash:

transcrypt 2.3.0-pre ca6a4675fbd8693c0f214382a5651a405a1c6843
LibreSSL 3.3.6

My workaround to fix were change LibreSSL 3.3.6 to OpenSSL 1.1.1s on git default configurations forcing transcrypt to work with the version above instead of the default LibreSSL that comes with Ventura, just added the following line to my local .git config file:

[transcrypt] openssl-path = /usr/local/Cellar/openssl@1.1/1.1.1s/bin/openssl

Guess that shouldn't be the definitive solution since default packaging combination may cause the same issue.

ronangaillard commented 1 year ago

Same problem here on Ventura :(

I tried the version on main branch andI have the same issue. I tried the solution by @alb777 but even if file is decrypted I have weird characters added to the file.

Makes team work nearly impossible.

Let me know if you need more info or help.

jmurty commented 1 year ago

Hi @ronangaillard sorry you're having trouble.

I've done some more testing of the latest version from main with encrypted files between MacOS 13 Ventura (openssl version : LibreSSL 3.3.6) and Ubuntu (openssl version : OpenSSL 1.1.1f 31 Mar 2020) but I haven't been able to reproduce any problems.

I'd appreciate your help in testing this and pinning down the issue.

Can you try each of the following things and let me know for each whether or not it worked properly?

Finding out which of these steps helps, if any, might help narrow down the issue.

FWIW I am now running and testing against Ventura version 13.0.1 (22A400) though I don't think the included openssl command version has changed since the initial 13 MacOS release.

Minishlink commented 1 year ago

FYI, to downgrade brew installation of transcrypt, use the following:

brew tap-new $USER/local-transcrypt
brew extract --version=2.1.0 transcrypt $USER/local-transcrypt
brew uninstall transcrypt
brew install transcrypt@2.1.0

Make sure to have a clean repo without transcrypt v2.2.0 configured, otherwise uninstall transcrypt configuration with transcrypt -u beforehand.

joaooliveiramc commented 1 year ago

Hello,

Me and my coworker are having the exact same issue. I'm using Ventura and he is using an older version of MacOS. When I push things into the repo and he pulls he has weird characters at the start. We tried different approaches. Including updating to the latest 2.2.0 transcript, updating and defining the default openssl to 1.1.1s or LibreSSL.

When doing openssl version it gives OpenSSL 1.1.1s 1 Nov 2022 doing Which openssl returns /opt/homebrew/opt/openssl@1.1/bin/openssl

Although the only way to make it work is doing @alb777 said and manually setting the git config file to /opt/homebrew/opt/openssl@1.1/bin/openssl

jmurty commented 1 year ago

Thanks for the feedback @joaooliveiramc I have been having trouble isolating the exact problem so this info is useful.

Did you try using the version of Transcrypt on the main branch (commit https://github.com/elasticdog/transcrypt/commit/ca6a4675fbd8693c0f214382a5651a405a1c6843)?

I ask because that branch has a change which I think fixes compatibility with the LibreSSL 3.3.6 openssl binary on MacOS Ventura but I haven't released it officially yet because the fix doesn't seem to work for everyone. If it does work for you, you wouldn't need to set the path to the Homebrew OpenSSL binary.

By the way, the reason you need to override the path to the openssl binary using transcrypt --set-openssl-path=PATH_TO_OPENSSL, or setting it in the git config yourself, is because Transcrypt may not inherit your PATH environment variable when run in the context of Git's clean & smudge operations.

joaooliveiramc commented 1 year ago

Hey,

So I just installed the main version by cloning and following the commands stated there on my MacOS Ventura. I also set the OpenSSL to LibreSSL and kept the git config openssl-path = openssl.

After cloning our work and configuring it with transcript now I'm having all sort of a weird string looking like a random generated password, which before was not happening.

jmurty commented 1 year ago

Ugh, so it's definitely not working for you. Thanks for trying, and sorry about that.

I will need to get to the bottom of this and find a proper fix. In the meantime your best options are probably to go back to using the Homebrew openssl path, or downgrading from 2.2.0

jmurty commented 1 year ago

Hi, could those of you experiencing incompatibility with MacOS Venture please help me to isolate the ongoing bug by running some tests on your Mac, ideally on both MacOS 13 (Ventura) and prior versions like 12?

I have pushed a new version of the transcrypt script in bea18476 with nothing but the released version 2.2.0 plus my prior attempted fix for MacOS / LibreSSL compatibility.

I'd very much appreciate feedback on your results of running some low-level encrypt and decrypt commands to test what results you get using this version of the script with the different versions of OpenSSL and LibreSSL you have on your system.

To get ready to run the tests:

  1. Create and cd into a new directory for testing, separate from any existing code directory, e.g: mkdir test-transcypt && cd test-transcypt
  2. Download and save this transcrypt version to test: wget https://raw.githubusercontent.com/elasticdog/transcrypt/bea184767608d0c24d20dce01381d3ea3ff50a62/transcrypt
  3. Make the transcrypt script executable: chmod u+x transcrypt
  4. Initialise git in the test directory: git init
  5. Set the cipher to use: git config --local transcrypt.cipher aes-256-cbc
  6. Set a fake password, use this exact command: git config --local transcrypt.password abc123
  7. Create this plaintext file to test against, use this exact command: echo "Plaintext for testing" > test-secret-file

Once the testing directory is set up, run these commands to test encrypting and decrypting the example file:

  1. Configure transcrypt to use your system default openssl: git config --local transcrypt.openssl-path /usr/bin/openssl
    • Check the version reported by that openssl path for cross-checking, e.g: /usr/bin/openssl version
  2. Output the encrypted version of the plaintext file: cat test-secret-file | ./transcrypt clean test-secret-file
    • The output should be exactly U2FsdGVkX18HgDaYdwMdMtRxglvIjDqKGpAP66vzqy4QHyqEzouDb0YweqfVm5+/
    • You can ignore output lines like *** WARNING : deprecated key derivation used. and Using -iter or -pbkdf2 would be better.
  3. Do a roundtrip encrypt-then-decrypt: cat test-secret-file | ./transcrypt clean test-secret-file | ./transcrypt smudge
    • The output should be exactly Plaintext for testing

Repeat the above commands, but substitute the path to openssl for step 1. to use different versions you have from the OpenSSL or LibreSSL projects, for example:

If the script is working correctly you should get exactly the same outputs each time for the 2. encrypt (clean) command and the 3. encrypt-then-decrypt (clean and smudge) commands.

If you see anything different, such as a different encrypted text blob or garbage characters after the round-trip, that could give me a vital clue to figure out what problems remain. Any anomalies you can report would be much appreciated, I need the help to figure this out!

For what it's worth, when I run the test commands on my system (Ventura 13.1) I get the same results for all versions of OpenSSL and LibreSSL openssl paths. So it "works on my machine" which is making it very difficult to debug.

jmurty commented 1 year ago

For advanced testing, you can run the low-level encrypt (clean) command in an existing repository with decrypted files, to check whether the encrypted output exactly matches the encrypted file stored in git.

If there is any difference between the encrypted data stored in git versus the output of your test clean command, this will cause git to report phantom changes.

As an example, if you're in a repo with transcrypt installed (2.1.0 or later) and your file my-secrets is already decrypted you can generate the encrypted version with:

cat my-secrets | .git/crypt/transcrypt clean my-secrets 2> /dev/null

# NOTE: If you're using transcrypt from the `main` branch you will need to run this instead
cat my-secrets | .git/crypt/transcrypt clean context=default my-secrets 2> /dev/null

The encrypted version generated by this command should exactly match the encrypted data stored in git, which you can print with:

.git/crypt/transcrypt --show-raw=my-secrets 2> /dev/null
Fausto95 commented 1 year ago

Hi @jmurty, thanks to the above instructions I was able to fix the issue. All of them worked for me.

Z7oussem commented 1 year ago

Hello @jmurty, thanks a lot you saved my day :)

emcniece commented 1 year ago

Using these two packages on OSX 13.1:

$ openssl version
LibreSSL 3.3.6

$ /opt/homebrew/opt/openssl@1.1/bin/openssl version
OpenSSL 1.1.1t  7 Feb 2023

Test steps execute exactly as described: encrypted and decrypted content match the examples. Thank you!!

jmurty commented 1 year ago

Thanks very much for your feedback @Fausto95 @Z7oussem @emcniece, it gives me more confidence I've finally fixed the bug using the LibreSSL 3+ flavour of the openssl binary.

I have released a new version 2.2.1 of transcrypt with a fix for the original bug.

I also added a work-around which should properly decrypt files committed on systems that were affected by the bug, so these should now give you the original content instead of showing binary junk at the top of the file. See details in b380f2c9 Note that incorrectly encrypted files will remain in your repository even with this work-around – manifesting as phantom changed files that look no different in diffs – but if you commit the seemingly unchanged file transcrypt will fix the encrypted version stored in Git.

Please grab the 2.2.1 version of transcrypt, make sure it's updated on all systems by running transcrypt --upgrade within every repository, and check whether it finally resolves this problem.

joaooliveiramc commented 1 year ago

How do I catch the 2.2.1? doing upgrade is only trying to upgrade to 2.2.0

jmurty commented 1 year ago

Hi @joaooliveiramc it may take a while for the latest version to be made available by package managers. In the meantime, transcrypt itself is just a bash shell script so you can download version 2.2.1 and run it directly from within your repository like so:

# Download transcrypt script at version/tag 2.2.1
wget https://raw.githubusercontent.com/elasticdog/transcrypt/v2.2.1/transcrypt

# Make it executable
chmod u+x transcrypt

# Confirm the version, and upgrade it in the repository
./transcrypt --version
./transcrypt --upgrade

FWIW, I like to keep a copy of the transcrypt script in shared repositories that use it, at a path like bin/transcrypt. That way anyone using the repo can always run the latest version without relying on package managers to be up-to-date. It's easier to tell colleagues to run ./bin/transcrypt --upgrade in the repo, than to make sure they have the latest version available on their system.