OpenSSL update breaks encryption

ubmarco commented 6 years ago

My colleague I'm sharing files with Sparkleshare. We both are using Sparkleshare 1.5.0, he is on Ubuntu 16.04 LTS and I upgraded from Ubuntu 17.10 to Ubuntu 18.04 LTS the last days.

After the update he and I get Sparkleshare errors like:

bad decrypt
140557227241920:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:../crypto/evp/evp_enc.c:541:
error: external filter 'openssl enc -d -aes-256-cbc -base64 -S <salt> -pass file:.git/info/encryption_password' failed 1
error: external filter 'openssl enc -d -aes-256-cbc -base64 -S <salt> -pass file:.git/info/encryption_password' failed

I assume this has something to do with the different openssl versions we're using now.

My new openssl version is (Ubuntu 18.04 LTS):

$ openssl
OpenSSL> version
OpenSSL 1.1.0g  2 Nov 2017

His openssl version is (Ubuntu 16.04 LTS):

openssl version 
OpenSSL 1.0.2g  1 Mar 2016

This Q&A https://stackoverflow.com/questions/39637388/encryption-decryption-doesnt-work-well-between-two-different-openssl-versions/39641378#39641378 tells the default digest was changed from MD5 to SHA256 in Openssl 1.1. The solution is to add -md md5 to the openssl enc command. However, in .git/config it is not present under filters. We added it both and it works again.

Now our history is totally messed up because any changes could not be decrypted on the other machine but were put to the sync folder. Then, without decryption, it was a different file compared to the Sparkleshare server. This leads to changes getting back and forth all the time.

How can we fix the history?

Shouldn't Sparkleshare setup the config filter for encrypted repos with -md md5?

ubmarco commented 6 years ago

I was going to revert to an old version of the relevant files so we can work again. When clicking on the tiny dropdown button in the history, I get the following message

screenshot_2018-07-04_11-37-40

ubmarco commented 6 years ago

I've just checked the openssl version history for Ubuntu: Ubuntu 17.10 Artiful: https://launchpad.net/ubuntu/artful/+source/openssl -> openssl 1.0.2g Ubuntu 18.04 Bionic: https://launchpad.net/ubuntu/bionic/+source/openssl -> openssl 1.1.0g

So that should be the issue. By the way, thanks for this great collaboration, it's been working over several years now without any issue.

hbons commented 6 years ago

Yea, looks like you got caught out by the updates. :(

In v2, sha256 is explicitly enforced. See b2b5295e6. I could make a 1.5.1 release with just a small fix to use md5, but this should really be fixed in by Ubuntu as they are responsible for integrating everything in the distro working well together (and also not being a major version behind :).

In any case, I recommend to using v2 from Flathub. It breaks encryption compatibility but has a lot more fixes like this one. You will have to recreate your repositories however...

ubmarco commented 6 years ago

Thanks for the quick feedback. I already assumed that version 2 fixes this :) However, I cannot use v2 because:

I can't get it to build locally because on bionic some pre-built dependencies are not available (the mentioned ppa for gtk-sharp3 is for Ubuntu saucy release) and I could not manage to compile the dependencies on my own in 60 minutes...

I can't use the flatpak because if the issue (already reported)


$ /usr/bin/flatpak run --branch=stable --arch=x86_64 --command=sparkleshare org.sparkleshare.SparkleShare
15:24:53 Environment | SparkleShare 2.0.1 (Flatpak)
15:24:54 Environment | Git LFS 2.3.0
15:24:54 Environment | Git 2.14.1
15:24:54 Environment | Ubuntu (Unix 4.15.0.24)
15:24:54 Cmd |  | gvfs-set-attribute "/home/marco/SparkleShare" metadata::custom-icon-name org.sparkleshare.SparkleShare

(process:4): Gtk-WARNING **: Locale not supported by C library. Using the fallback 'C' locale.

hbons commented 6 years ago

oh it looks like there's a v2 package in the next Ubuntu: https://packages.ubuntu.com/cosmic/sparkleshare

hbons commented 6 years ago

Hopefully I can get those flatpak issues resolved as I'm attending the GNOME conference tomorrow and they'll know more. Stay tuned.

ubmarco commented 6 years ago

Flatpak is a good way to distribute software - you can't wait for all the distro maintainers if you're pushing out a new version. Users stick with old version because of their aging distros and therefore miss fixes for important bugs; and you have to maintain a huge list of issues which are already fixed in app new versions.

I just tried downloading the .deb file from the Cosmic release and succeeded installing it on Bionic. So to get the correctly encrypted files on my Sparkleshare server, I'll have to recreate the repository on the server, register and connect each client again and then upload all the files from one client - is that the way to go?

Thanks a lot for your efforts on this project and the quick help.

ubmarco commented 6 years ago

I tried to setup a local Sparkleshare server to test the migration using dazzle:

$  dazzle setup
 1/4 | Installing the Git package...
  -> The Git package has already been installed (version 2.17.1).
 2/4 | Creating account "storage"...
  -> Account already exists.
 3/4 | Configuring account "storage"...
  -> mkdir --parents /home/storage/.ssh
  -> touch /home/storage/.ssh/authorized_keys
  -> chmod 700 /home/storage/.ssh
  -> chmod 600 /home/storage/.ssh/authorized_keys
 4/4 | Reloading the SSH config...
  -> /etc/init.d/ssh reload
/usr/bin/dazzle: line 116: /etc/init.d/ssh: No such file or directory

Setup complete!
To create a new project, run "dazzle create PROJECT_NAME".

I assume I need the ssh daemon (which commonly runs on servers). Shouldn't that be considered by the dazzle script? It also considers the right version of Git. After install ssh, I get

$  dazzle setup
 1/4 | Installing the Git package...
  -> The Git package has already been installed (version 2.17.1).
 2/4 | Creating account "storage"...
  -> Account already exists.
 3/4 | Configuring account "storage"...
  -> mkdir --parents /home/storage/.ssh
  -> touch /home/storage/.ssh/authorized_keys
  -> chmod 700 /home/storage/.ssh
  -> chmod 600 /home/storage/.ssh/authorized_keys
 4/4 | Reloading the SSH config...
  -> /etc/init.d/ssh reload
Enter Private Key Password:

He's waiting for a password - why would restarting the SSH daemon need a private key password? (I know this is a bit off-topic...)

ubmarco commented 6 years ago

To be able to work with version 1.5.0 again, I downgraded openssl to 1.0.2g using the 17.10 .deb package.

Then I reinstalled Sparkleshare 1.5.0, deleted the .config/sparkleshare folder, and re-added our hosted project again. Cloning was successful, entering the password and decryption too. Then, when stopping and starting Sparkleshare, I receive this error:

13:14:02 | Controller | Failed to load backend 'Git' for 'sync': : Exception has been thrown by the target of an invocation.   at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00019] in <8f2c484307284b51944a1a13a14c0266>:0 
  at System.Reflection.MonoCMethod.DoInvoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00089] in <8f2c484307284b51944a1a13a14c0266>:0 
  at System.Reflection.MonoCMethod.Invoke (System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <8f2c484307284b51944a1a13a14c0266>:0 
  at System.RuntimeType.CreateInstanceImpl (System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes, System.Threading.StackCrawlMark& stackMark) [0x00243] in <8f2c484307284b51944a1a13a14c0266>:0

It's fully reproducible and won't go away. Do you have a clue?

hbons commented 6 years ago

I'll have to recreate the repository on the server, register and connect each client again and then upload all the files from one client - is that the way to go?

Yes. Also you no linger need to name the repository "-crypto". Now when SparkleShare detects an empty new repo it will ask you if you'd like it encrypted on the first clone.

hbons commented 6 years ago

Sorry I can't really help on the dazzle script or that error in the 1.5 version, it's just too old.

owenjm commented 6 years ago

I'm also wrestling with this issue:

I've built the latest Sparkleshare from git (and also release 2.0.1 from git) on linux to overcome the openssl changes, but when I try to add an already-established, encrypted repository (with a -crypto name), the new versions of Sparkleshare don't correctly detect that it's encrypted? They add the repository without any password prompt and include "-crypto" in the share name.

Am I missing something, or did this new functionality break connecting to any old, encrypted shares? Or even any newer shares with a Windows machine (given that the last Windows release was 1.5)?

ubmarco commented 6 years ago

@owenjm From my investigation, you'll have to setup your server repository again after the migration to version 2. The old Sparkleshare version does not constrain the openssl digest algorithm it uses to create a key from the passphrase, so it uses whatever the defaults of openssl are (presumably md5 in your case). When pulling that old repo with Sparkleshare 2, it specifies explicitly in the .git/config filters section, that the repo uses sha256 as digest algorithm. If that's wrong, the decryption will fail.

I set up a new repo and copied the content from one client using Sparkleshare 2.0.1.

@hbons I think we should do one of the following: either release a patched 1.5 version which explicitly sets -md md5 so users don't break their sync when updating the openssl library. However this version would have to reach the Distro maintainers... Or put a warning into the docs that Sparkleshare 1.5 repos will not work with openssl 1.1.x (and thus with Ubuntu 18.10) - this is really a big problem because the decryption fails silently with existing repos and starts corrupting the files (at least in my case). Or introduce a compatibility feature to sparkleshare 2 which chooses the 'old' algorithm, if that does not collide technically with other changes being introduced since 1.5.

What do you think?

owenjm commented 6 years ago

@ubmarco Yes, I notice that the changelog does actually mention that version 2.0+ was not backwards compatible with previous versions.

In the end I solved the problem by adding the explicit -md md5 flag to the 3 relevant lines of the 1.5.0 release code and recompiling, which works fine.

hbons commented 5 years ago

I've tagged a new 1.5.1 release with the fix: https://github.com/hbons/SparkleShare/tree/1.5.1 As nobody is going to find a a note in the source tree. :)

Could you let the maintainers of your distro know and point to this issue?

ubmarco commented 5 years ago

See here for the Ubuntu bug report pointing to the new release.

hbons / SparkleShare

OpenSSL update breaks encryption #1855