search

PhilippC / keepass2android

Password manager app for Android
https://play.google.com/store/apps/details?id=keepass2android.keepass2android
GNU General Public License v3.0
4.84k stars 390 forks source link

Problems with HTTP/2 (waiting for OkHttp update) #747

Closed jan844 closed 4 years ago

jan844 commented 5 years ago

I am trying to use keepass2android with a keepass database stored on a webdav server. If I change the database in the app and click on database sync ("Datenbank synchronisieren ..."), there is a message, that the database has changed by another application and the there is the possiblity to reload the database. It is not possible to store my local changes on the webdav server.

PhilippC commented 5 years ago

if you have opened the database through "Open database" -> "WebDav", the changes made in the should be uploaded immediately during the saving operation. Can you check if that happened?

FadeFx commented 5 years ago

Looks like a config issue, i use the same setup with nextcloud as a webdav server and it just works

Edit: after some more testing i have to take that back, its not syncing upwards

dbielz commented 5 years ago

I have the same issue since a couple of months, I think since updating on Nextcloud 15.0x. Keepass2android will always get the latest changes from the server, but fails to save its own changes. Here some lines from the server log (apache2.4):

212.201.19.220 - - [03/May/2019:14:43:11 +0200] "GET /remote.php/webdav/DBpwds.kdbx HTTP/2.0" 401 1202 "-" "okhttp/3.9.0" 212.201.19.220 - - [03/May/2019:14:43:11 +0200] "GET /remote.php/webdav/DBpwds.kdbx HTTP/2.0" 200 362145 "-" "okhttp/3.9.0" 212.201.19.220 - - [03/May/2019:14:43:22 +0200] "PUT /remote.php/webdav/DBpwds.kdbx HTTP/2.0" 401 1224 "-" "okhttp/3.9.0" 212.201.19.220 - - [03/May/2019:14:43:27 +0200] "GET /remote.php/webdav/DBpwds.kdbx HTTP/2.0" 401 1197 "-" "okhttp/3.9.0" 212.201.19.220 - - [03/May/2019:14:43:27 +0200] "GET /remote.php/webdav/DBpwds.kdbx HTTP/2.0" 200 362170 "-" "okhttp/3.9.0" 212.201.19.220 - - [03/May/2019:14:43:47 +0200] "GET /remote.php/webdav/DBpwds.kdbx HTTP/2.0" 401 1202 "-" "okhttp/3.9.0" 212.201.19.220 - - [03/May/2019:14:43:48 +0200] "GET /remote.php/webdav/DBpwds.kdbx HTTP/2.0" 200 362173 "-" "okhttp/3.9.0" 212.201.19.220 - - [03/May/2019:14:43:52 +0200] "PUT /remote.php/webdav/DBpwds.kdbx HTTP/2.0" 401 1227 "-" "okhttp/3.9.0" 212.201.19.220 - - [03/May/2019:14:44:42 +0200] "GET /remote.php/webdav/DBpwds.kdbx HTTP/2.0" 401 1195 "-" "okhttp/3.9.0"

If I compare this to successfull requests from the android client DAVx5, I notice the following:

212.201.19.220 - - [03/May/2019:11:09:40 +0200] "PROPFIND /remote.php/dav/calendars/xxxx/xxxx/ HTTP/2.0" 207 1569 "-" "DAVx5/2.2.3.1-ose (2019/01/27; dav4jvm; okhttp/3.12.1) Android/9" 212.201.19.220 - - [03/May/2019:11:09:41 +0200] "PROPFIND /remote.php/dav/calendars/xxxx/xxxx/ HTTP/2.0" 207 1574 "-" "DAVx5/2.2.3.1-ose (2019/01/27; dav4jvm; okhttp/3.12.1) Android/9"

Maybe it's just necessary to update some lib (okhttp)?

Thanks for your care and efforts, D

ttomasini commented 5 years ago

Same problem here. Downloading works for me but as soon as I change the database on my mobile device keepass2android is not able to sync it anymore because of a okhttp3 exception:

03.05.2019 17:04:28:583 -- stream was reset: NO_ERROR 03.05.2019 17:04:28:635 -- couldn't save to remote https://username:password@cloud.example.com/remote.php/dav/files/username/file.example 03.05.2019 17:04:28:638 -- System.Exception: stream was reset: NO_ERROR ---> Java.IO.IOException: stream was reset: NO_ERROR at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000a] in <1234deadbeef1234deadbeef1234dead>:0 at Java.Interop.JniEnvironment+InstanceMethods.CallVoidMethod (Java.Interop.JniObjectReference instance, Java.Interop.JniMethodInfo method, Java.Interop.JniArgumentValue args) [0x0000a] in <1234deadbeef1234deadbeef1234dead>:0 at Android.Runtime.JNIEnv.CallVoidMethod (System.IntPtr jobject, System.IntPtr jmethod, Android.Runtime.JValue parms) [0x0000a] in <1234deadbeef1234deadbeef1234dead>:0 at Keepass2android.Javafilestorage.WebDavStorage.UploadFile (System.String p0, System.Byte[] p1, System.Boolean p2) [0x0000a] in <1234deadbeef1234deadbeef1234dead>:0 at keepass2android.Io.JavaFileStorage+JavaFileStorageWriteTransaction.CommitWrite () [0x0000a] in <1234deadbeef1234deadbeef1234dead>:0 --- End of inner exception stack trace --- at keepass2android.Io.JavaFileStorage+JavaFileStorageWriteTransaction.CommitWrite () [0x0000a] in <1234deadbeef1234deadbeef1234dead>:0 at keepass2android.Io.CachingFileStorage.UpdateRemoteFile (System.IO.Stream cachedData, KeePassLib.Serialization.IOConnectionInfo ioc, System.Boolean useFileTransaction, System.String hash) [0x0000a] in <1234deadbeef1234deadbeef1234dead>:0 at keepass2android.Io.CachingFileStorage.TryUpdateRemoteFile (System.IO.Stream cachedData, KeePassLib.Serialization.IOConnectionInfo ioc, System.Boolean useFileTransaction, System.String hash) [0x0000a] in <1234deadbeef1234deadbeef1234dead>:0 03.05.2019 17:04:28:668 -- Pre-loading database file completed

I think the problem here is that okhttp3 tries to establish a HTTP2.0 connection. Can someone try to set okhttp3 to HTTP1.1 by default?

Ronkn commented 5 years ago

What other information can I provide to help resolve this issue? I hate using KeepassDX

chris42 commented 5 years ago

Works for me with Android 8.1, Nextcloud 16.0.1, Keepass2Android 1.07-r1 in both directions. Maybe some webserver config, or the non ASCII Issue in #478?

dbielz commented 5 years ago

What webserver config could be responsible? As you can see in my server logs, there is always the first GET responded with 401, and the second works. The PUT requests, however, only submit the first request and do not try a second time. Any idea? Everything worked for years, no server config change in the meantime, except for updates of NC to NC 15 at some time. But I think, the problem started with one of the last updates of Keepass2Android. Therefor I started looking here.

chris42 commented 5 years ago

I do not have an idea, but as with the linked Issues, this one seems to work for some and not for others. As Keepass2Android is quite static for everyone, your webserver config can be varied quite a lot. I would probably push up logging on the webserver, to see if there are errors in handshake, etc.

Luensche commented 5 years ago

I have the same issue and it is very annoying, because I cannot use the db on multiple devices anymore. In additon synchronising and merging doesn't work. It shows a overlay message with a timeout error.

dbielz commented 5 years ago

I also tried the app password workaround mentioned in #478 but it did not work. What does work is DAVx5 (aka davdroid), which also uses okhttp. Only difference is, that keepass2android uses okhttp 3.9.0, while DAVx5 uses okhttp 3.12.1. Could it be, that there is the crucial change, and that a newer okhttp also might cure #478 ? In Addition, the message in this issue sounds very much like the message I get, when I try to save a modification from android, although the issue refers probably to a newer version of okhttp, and the message I can on my device see is much shorter:

Exception: stream was reset: NO_ERROR

But I think this points into the right direction.

starsoccer commented 5 years ago

I have also tried the app password workaround but my issue is a bit different as on my server I cant even login. When I try I get the unsupported auth type. It may be related to this issue though as from my testing the error is independent of the correct password being used.

Also I have one server where I get the above error constantly then another server where it works fine.

cenur commented 5 years ago

I'm observing exactly the same issue as reported before, which shows up as

Exception: stream was reset: NO_ERROR

whenever trying to synchronize a kdbx database back to the server using WebDAV.

However, I'm not using Nextcloud for that at all but just the plain simple Apache mod_dav.

It used to work for years, but after migrating to a newer smartphone the problem exists. Maybe this can somehow help to track down the root cause of this? The Keepass2Android version is identical on both phones, nothing was changed on server side. So the only difference seems to be the Android version?

Old phone: Android 5.0.2 (Samsung Galaxy ALPHA) New phone: Android 9 PSA29.97-37 (Motorola One Visision, Android One)

Keepass2Android: 1.07-r1

On the server: Apache 2.4.25, HTTPS, HTTP/2 enabled, Digest authentication

Any idea for a workaround would be highly appreciated! ;-)

cenur commented 5 years ago

Just came across this issue: https://github.com/square/okhttp/issues/4964 It seems to be caused by HTTP/2. Latest version of okhttp has an issue with it...

I forced the usage of http/1.1 for my WebDAV virtualhost in the Apache config, and now saving kdbx files is working again! :-) Maybe this workaround is helpful to others here as well...

So probably nothing can be done on keepass2android side, except for waiting for a fixed okhttp version.

starsoccer commented 5 years ago

Weird, my http2 server is the one thats working fine(for logins atleast) but the http 1.1 server gives me the no auth scheme error. I will try forcing both to http 1.1 and see what happens

dbielz commented 5 years ago

Apparently it's the combination of server / okhttp / androidversion making the trouble. In my case turning off http2 made the sync work again. Woohoo! Thanks @cenur for that workaround.

starsoccer commented 5 years ago

Sadly this didnt work for me. Could anyone share there nginx or apache2 conf files so I can compare mine against theres?

Also is everyone using webdav directly(https webdav) from the options or selecting nextcloud?

dbielz commented 5 years ago

I did a quick test with apache 2.4 by doing "a2dismod http2" and restarting apache. After seeing that this works, I enabled the http2 module again and disabled it only for my Nextcloud instance:

\<VirtualHost default:443> ServerAdmin webmaster@localhost ServerName xxx.xxxxxx.de:443

owncloud security measure

Header add Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"

Header set Referrer-Policy "no-referrer"

Protocols http/1.1 ### only use http1 instead of (was before):

Protocols h2 http/ 1.1

DocumentRoot /var/www/nextcloud/ ....

Wether or not the STS-Header proposed by an owncloud instance long time ago has any influence is out of my scope. As well as any other default settings I am not aware of. The key difference between working or not is in my case the 'Protocols' line.

starsoccer commented 5 years ago

I gave that a try, disabling http2 and uncommenting it, but it didnt seem to matter. My config below:

<VirtualHost *:443>
        ServerName xx.xxx.xxxxxxx
        ServerAdmin xxxx@xxx.xxxxx
        DocumentRoot /var/www/nextcloud

        SSLEngine on
        SSLCertificateFile /path/to/certificate.crt
        SSLCertificateKeyFile /path/to/private.key
        SSLCertificateChainFile /path/to/ca_bundle.crt

        Protocols http/1.1
        <Directory /var/www/nextcloud/>
                Require all granted
                AllowOverride All
                Options FollowSymLinks MultiViews
                SetEnv HOME /var/www/nextcloud
                SetEnv HTTP_HOME /var/www/nextcloud
        </Directory>
</VirtualHost>
AmorbiaCacoa commented 5 years ago

I came across this problem as well. Here my observations:

I don't understand the last two points, if 'check for modifications' blocks upstream sync, switching in ON again should result in a failure. But maybe at least my first observation that switching OFF 'check for modifications' allows upstream sync (at least in my case) will help someone.

FadeFx commented 5 years ago

I have also done some tests again today, but only with one of my two phones, but to me sync worked flawless today without changing any settings. Only difference i am currently aware of, is that my nextcloud instance i sync to has changed from 15.x to 16.0.2. I have yet to test on my other device as well, but to me that looked good.

dbielz commented 5 years ago

So, meanwhile I updated my base system to Ubuntu 18.04 LTS and thus could update to nextcloud 16.0.3. Reenabled http2, but still no success; modifications from android still could not be saved via http2. Even not with @AmorbiaCacoa 's workaround to switch off 'check for modifications' before saving.

But on updating i stumbled across the fact, that at some time i had followed an advice on nextclouds settings page to switch my php to fpm-php (whatever that means), to make nextcloud faster (it ran really slow at that time). I followed all howtos and somehow managed; now all other web services (besides nextcloud I run piwigo, a foto gallery, and use the same instance of apache to provide the front-end of pihole) run with fpm-php. Could this be the difference?

PHP Version 7.2.20-1+ubuntu18.04.1+deb.sury.org+1 System | Linux 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 Build Date | Jul 10 2019 06:59:02 Server API | FPM/FastCGI Virtual Directory Support | disabled Configuration File (php.ini) Path | /etc/php/7.2/fpm Loaded Configuration File | /etc/php/7.2/fpm/php.ini Scan this dir for additional .ini files | /etc/php/7.2/fpm/conf.d ...

AmorbiaCacoa commented 5 years ago

But on updating i stumbled across the fact, that at some time i had followed an advice on nextclouds settings page to switch my php to fpm-php (whatever that means), to make nextcloud faster (it ran really slow at that time). I followed all howtos and somehow managed; now all other web services (besides nextcloud I run piwigo, a foto gallery, and use the same instance of apache to provide the front-end of pihole) run with fpm-php. Could this be the difference?

I'm running nextcloud 16.0.1 on a raspberry pi3 with raspian stretch. I also use php7.3-fpm, because it gives better performance compared to using mod_php with apache2.

FadeFx commented 5 years ago

My system is nc 16.0.2 on a raspberry pi2b and i use nextcloudpi without any modification to php whatsoever... and aa i say, for me webdav works perfect on two phones, changes made on one phone are loaded on the other phone as soon as db is synced. Both ways btw, webdav connection or selecting nextcloud and browsing to the file. But i guess it doesnt make any difference

dbielz commented 5 years ago

I think I found it: as @cenur mentioned, the library okhttp seems to have a problem with http2. So we disabled http2 on our webservers to make keepass2android work. But still I wondered, how the other apps using okhttp cope with this problem, so I reenabled http2 on my server and compared the lines in the server log (http2 enabled in all cases):

[16/Jul/2019:08:48:48 +0200] "GET /remote.php/webdav/Keepass.kdbx HTTP/2.0" 401 1201 "-" "okhttp/3.9.0" [16/Jul/2019:08:48:48 +0200] "GET /remote.php/webdav/Keepass.kdbx HTTP/2.0" 200 363507 "-" "okhttp/3.9.0" [16/Jul/2019:08:48:49 +0200] "PUT /remote.php/webdav/Keepass.kdbx HTTP/2.0" 401 1224 "-" "okhttp/3.9.0" [16/Jul/2019:08:48:52 +0200] "PROPFIND /remote.php/dav/files/username/ HTTP/1.1" 207 1395 "-" "Mozilla/5.0 (Windows) mirall/2.5.1final (build 20181204) (Nextcloud)" [16/Jul/2019:08:49:00 +0200] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 1497 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.7.0" [16/Jul/2019:08:49:00 +0200] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 23908 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.7.0" [16/Jul/2019:08:52:11 +0200] "PROPFIND /remote.php/dav/calendars/username/user1/ HTTP/1.1" 207 6018 "-" "DAVx5/2.5.1-ose (2019/06/13; dav4jvm; okhttp/3.12.3) Android/9" [16/Jul/2019:08:52:11 +0200] "PROPFIND /remote.php/dav/addressbooks/users/username/contacts/ HTTP/1.1" 207 6934 "-" "DAVx5/2.5.1-ose (2019/06/13; dav4jvm; okhttp/3.12.3) Android/9" [16/Jul/2019:08:52:11 +0200] "PROPFIND /remote.php/dav/calendars/username/default_shared_by_user2/ HTTP/1.1" 207 6936 "-" "DAVx5/2.5.1-ose (2019/06/13; dav4jvm; okhttp/3.12.3) Android/9" [16/Jul/2019:08:52:11 +0200] "REPORT /remote.php/dav/calendars/username/user1/ HTTP/1.1" 207 1668 "-" "DAVx5/2.5.1-ose (2019/06/13; dav4jvm; okhttp/3.12.3) Android/9"

I found two things remarkable:

  1. with http2, okhttp 3.9.0 makes two requests on GET: the first one with status code 401, the second with 200. On PUT, it makes only the first request, responded with 401. In earlier observations I already could see, that this is different from http1, where the PUT request is also performed twice, like the GET request: one with 401, one with 200. But I think I mentioned this one already above.
  2. The other clients visible in these log lines, Nextcloud for Android and DAVx5, both do all their requests only with http/1.1, although on server side http2 is still enabled.

So apparently the http-clients of Nextcloud and DAVx5 handle this http2 bug in okhttp by forcing their own okhttp-requests to be always in http/1.1, even if http2 is available. I would really appreciate if Keepass2android would handle things the same way, so that I could reenable http2 for browser requests on my Nextcloud instance without limiting the usage of Keepass2android by doing so.

dbielz commented 5 years ago

... at least as long the http2 issue in okhttp is not resolved.

Cye3s commented 5 years ago

I have problem too.

1.Synolog WebDav https on 5006 port 2.Openwrt router 3.Can access the kdbx file with other app(SE exploser) whit 4G network on same phone (not filewall or phone problem)

error: nothing, just 'error'(Simple Chinese translation), flow with nothing

daiaji commented 5 years ago

I used the caddy server to build the WebDAV server. When I enabled HTTP/2, Keepass2android could not connect to my WebDAV server. After I stopped HTTP/2, Keepass2android could connect to my WebDAV server.

PhilippC commented 5 years ago

can someone send me testing credentials to a server which has this issue? Please send to crocoapps@gmail.com.

tomich commented 5 years ago

Hi. I am also having this issue. Webdav is completely broken only with keepass2android (I use many other Webdav apps and they work fine). Can I be of any assitance? Do you already have credentials for a test site? I can give you access yo my Nextcloud server.

PhilippC commented 5 years ago

@tomich I just received some credentials but a second pair would certainly be nice to have!

rhatguy commented 5 years ago

Just to add an additional data point. Gentoo linux, apache 2.4.39, nextcloud 16.0.5. KP2A unable to save, getting similar errors. Updated virtual host definition to include "Protocols http/1.1". All issues with KP2A were immediately solved. Eagerly awaiting the overall fix PhilippC!

dbielz commented 5 years ago

Meanwhile I overthought my position to http2 and disabled the module for all instances for everyday operations. For testing, I reenabled the http2 module on the whole instance and switched in the virtual host definition: ... Protocols http/1.1 WORKS ... Protocols h2 h2c http/1.1 does NOT work

As already found, keepass tries to use http2 if available. As long as this is not resolved, it simply does not make sense for keepass to try to use http2, because it will not work.

dbielz commented 5 years ago

By the way, if anybody wonders, why I decided NOT to use http/2 any more, here are my reasons: https://blog.fefe.de/?ts=aa031bbb and https://blog.fefe.de/?ts=a3adeb1a

sserkez commented 4 years ago

same issue

daiaji commented 4 years ago

By the way, if anybody wonders, why I decided NOT to use http/2 any more, here are my reasons: https://blog.fefe.de/?ts=aa031bbb and https://blog.fefe.de/?ts=a3adeb1a

I have heard the relevant report, and the conclusion seems to be that due to TLS+TCP, the HTTP/2 Head of line blocking will be more serious. Fortunately, HTTP/3 no longer uses TCP.

ghost commented 4 years ago

I did face the exact same issue, failing to push any change from keepass2android to my server: getting "stream was reset: NO_ERROR". I did hit this issue just after my migration from Apache (w/o http2 configured) to nginx (w/ http2 configured).

Thanks to ttomasini, dbielz and PhilipC; disabling http2 does fix the issue. Will re-enable it once fixed directly in okhttp client.

Luensche commented 4 years ago

@PhilippC it seems to be fixed with https://github.com/square/okhttp/pull/5663. Can you update the references to check if it really solves the problem?

PhilippC commented 4 years ago

as far as I understand, this is just a test, not a fix. The issue on which I provided some demonstrating code is still open: https://github.com/square/okhttp/issues/4964 The current release candidate 1.08-r0 on https://play.google.com/apps/testing/keepass2android.keepass2android has an updated OkHttp version.

Luensche commented 4 years ago

@PhilippC Since we have to wait still some months(?) until it will be fixed in square/okhttp#4964, do you see any possibility to create a workaround in the app?

e-patrick commented 4 years ago

I just found this issue after creating #1229 - I can add the information that saving very small databases does work for me. It starts to fail when the become bigger.

e-patrick commented 4 years ago

After disabling http2 on the nginx-reverse-proxy container (did so by mapping a modified nginx.tmpl into the containers /app directory) Keepass2Android can save and synchronize even my larger databases. Whoohoo!

MEitelwein commented 4 years ago

Can Keepass2Android simply revert to http/1.1 as long as the http/2 issue is not resolved? It is a bit annoying to keep http/2 switched off for whole instances just to keep Keepass2Android running

yschimke commented 4 years ago

The fix in OkHttp isn't trivial, and I wouldn't wait on it if this is affecting usage.

The fix for your client should be 

builder.protocols(Arrays.asList(Protocol.HTTP_1_1))

https://github.com/PhilippC/keepass2android/blob/97fb35db90499fd9f5332d157b0648b4234c9490/src/java/JavaFileStorage/app/src/main/java/keepass2android/javafilestorage/WebDavStorage.java#L172

Also you should reuse the client instance if you can. See documentation in https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/

llucax commented 4 years ago

Also affected by this, I think I lost some passwords because of this issue, as the server returned a 401 response I thought my token was somehow wrong and created a new one, after editing the the DB entry in Keepass2Android with the new token, it said the DBs were in sync, so I think it downloaded the file from the server as if it were a new database instead of the old one.

On top of that, the workaround is to make the server stop using HTTP/2, affecting all other good clients performance, and this only if you even have the ability to configure your server, which is probably only true for users self-hosting the server.

This is a very bad issue, with potential of data loss and very hard to debug for users, so it would be really good to get a quick hotfix (just making K2A use HTTP 1.1 would be perfectly acceptable).

Luensche commented 4 years ago

The fix in OkHttp isn't trivial, and I wouldn't wait on it if this is affecting usage.

The fix for your client should be

builder.protocols(Arrays.asList(Protocol.HTTP_1_1))

https://github.com/PhilippC/keepass2android/blob/97fb35db90499fd9f5332d157b0648b4234c9490/src/java/JavaFileStorage/app/src/main/java/keepass2android/javafilestorage/WebDavStorage.java#L172

Also you should reuse the client instance if you can. See documentation in https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/

Can you just create an pr, if you know the places to fix? Maybe, we can have faster a hotfix if @PhilippC has just to review it.

yschimke commented 4 years ago

@PhilippC would preemptive Authentication help here? i.e. assuming the credentials for the API request are known ahead of time, you could set them to avoid the initial 401?

Or possibly use a HTTP/1.1 auth check, before switching to HTTP/2 with known working credentials.

Apologies that we don't have a quick fix for this in OkHttp.

PhilippC commented 4 years ago

thanks @yschimke for your input! I just changed to reusing the client and specifying the protocols explicitly. Do you think there might be any side-effects of setting the protocol - shouldn't it also contain HTTP/1.0? I know it's old but with a large user-base, it's hard to tell what is actually required.

Regarding preemptive authentication: To be honest, I don't know about enough HTTP to answer whether this would help. I don't know which authentication method might be used, but I do know the username/password.

yschimke commented 4 years ago

If it's working well. Why change?

drzony commented 4 years ago

@PhilippC HTTP/1.0 is mostly deprecated by all the servers, some memory constrained devices sometimes use it to conserve RAM/ROM size, but I don't think that will ever be case here.

PhilippC commented 4 years ago

@yschimke I can't test all setups of all users (many run their own servers) so I want to avoid that such a change is solving one issue but creating one or more new issues...