search

corona-warn-app / cwa-testresult-server

Backend implementation of the test result information.
Apache License 2.0
91 stars 49 forks source link

Android CwaWebSecurityException - Server Maintenance on 2021-06-18? #103

Closed vaubaehn closed 2 years ago

vaubaehn commented 3 years ago

Dear server devs,

from time to time in the past year some CWA Android users experience a Web Security Exception, which looks like the certificate chain of the backend servers looks invalid for the client. See: https://github.com/corona-warn-app/cwa-app-android/issues/968 While for many users this is only a temporary problem that recovers itself automatically after a while, other users were reporting, that in result some Android OS versions deactivate the root certificate T-Telesec Global Root Class 2 permanently, which makes it necessary to manually re-activate that certificate, what can be quite tricky for non-tech savvy users. I was suspecting, that server maintenance could cause the problem, when after the maintenance the server is re-initialising and for a (probably very short) time the certificate chain (root/intermediate/server) is still not (fully) available to be delivered to the client. (1 second downtime could affect > 100 Android users, given equal distribution of client requests over the day)

I would be happy if you could confirm whether there was any maintenance going on on Friday, 2021-06-18, between 18:50 CEST and 19:45 CEST ? That could help to find out where the issue originates.

Any idea from your side what could causethe problems described in https://github.com/corona-warn-app/cwa-app-android/issues/968 would be highly appreciated.

Thank you in advance, kind regards, v.

hilmarf commented 3 years ago

Hi @vaubaehn,

thanks for all your efforts and research!

I checked with the operations team and indeed there was a hotfix deployment for Test-Result and Verification-Service during that time.

🖖 live long and prosper Hilmar

vaubaehn commented 3 years ago

Hi @hilmarf ,

thanks a lot for your engagement and reply! I think this takes us a nice step further! And thanks @sapcoder123 to transfer the issue to where it belongs to.

Do you think it's possible to enhance the server configuration of the publicly exposed endpoint(s), that in future the clients can react better to exceptions due to server maintenance? If all endpoints could respond with a 503 Temporary unavailable (and maybe even a 'retry-after') over a still working and valid secure layer (with a still valid certificate chain: root/intermediate/server certificate) when there is a maintenance, clients could gracefully catch the downtime and display corresponding error messages. Last week's downtime led to Android error messages like "Your test QR code is invalid or has already been scanned", and later on to "Web Security" exceptions (due to problems with the secure layer/server certificate chain). It would be cool if CWA could show messages like "due to server maintenance no connection - we'll retry later" in similar situations in the future. A still valid certificate chain would also prevent web security exceptions that confuse many users, and in rare cases permanently disables Android's T-Telesec Global Root Class 2 root certificate on some OS versions (I know this sounds crazy, but according to user reports that had been the case in the past). But for this to work, the server config might be adapted.

I hope it is ok, if I ping @thomasaugsten here - maybe you want to get in touch and evluate how server-client-interaction may be improved in these cases.

Thanks a lot for your support!

🖖 live long and prosper

🪐 QaQjaj jajlIj ! (Don't talk Klingon? Try: https://www.bing.com/translator/)