search

cern-fts / davix

High-performance file management over WebDAV / HTTP
GNU Lesser General Public License v2.1
83 stars 34 forks source link

davix version 0.8.0 cannot authenticate using username and password #81

Open mrensan opened 2 years ago

mrensan commented 2 years ago

Hi, I faced with an issue with davix-* commands when trying to authenticate by username and password. It was working before but suddenly stopped and constantly returns: (Davix::HttpRequest) Error: HTTP 401 : Authentification Error

So after investigation I found out the davix client version in the machine was upgraded to version 0.8.0 so I tested it with older version successfully.

We connect to a server with valid certificate over https, but to reproduce the issue here; I use a docker image as a webdav server:

  1. Create a docker-compose.yml file with content: 
    version: '3'
services:
webdav:
image: bytemark/webdav
restart: always
ports:
  - "80:80"
environment:
  AUTH_TYPE: Digest
  USERNAME: alice
  PASSWORD: secret1234
volumes:
  - ./dav:/var/lib/dav
  2. Run docker compose up -d to start the webdav server
  3. Add some data files in ./dav/data path to have some thing to show
  4. I have two versions of davix client installed on my machine 0.7.6 and 0.8.0 let's first try the older version that works: 
    /usr/local/Cellar/davix/0.7.6/bin/davix-ls -l --userlogin alice --userpass secret1234 http://localhost
-rwxrwxrwx 0     10         2021-10-28 19:17:29 data1.txt
-rwxrwxrwx 0     16         2021-10-28 19:44:54 data2.txt
  5. Now try with new version that fails: 
    /usr/local/Cellar/davix/0.8.0/bin/davix-ls -l --userlogin alice --userpass secret1234 http://localhost
(Davix::HttpRequest) Error: HTTP 401 : Authentification Error
  6. For more details we can run it of course with debug version: 
    
/usr/local/Cellar/davix/0.8.0/bin/davix-ls -l --debug --userlogin alice --userpass secret1234 http://localhost
DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 1 locks.

DAVIX(core): HTTP/SSL Session caching ENABLED DAVIX(core): Redirection Session caching ENABLED DAVIX(core): libdavix path /usr/local/Cellar/davix/0.8.0/lib/libdavix.0.8.0.dylib, version: 0.8.0 DAVIX(posix): -> opendirpp DAVIX(http): Create HttpRequest for http://localhost DAVIX(http): -> negotiateRequest DAVIX(http): NEON start internal request DAVIX(http): no cached ne_session, create a new one HTTP session to http://localhost:80 begins. DAVIX(http): configure session... DAVIX(http): define connection timeout to 30 DAVIX(http): enable login/password authentication DAVIX(http): enable client cert authentication by callback DAVIX(ssl): ah_create, for WWW-Authenticate DAVIX(socket): Running pre_send hooks

PROPFIND / HTTP/1.1 User-Agent: libdavix/0.8.0 neon/0.0.29 Keep-Alive: Connection: Keep-Alive TE: trailers Host: localhost Depth: 1 Content-Length: 303

DAVIX(socket): Sending request-line and headers: DAVIX(socket): Doing DNS lookup on localhost... DAVIX(socket): Sending request body: DAVIX(socket): Request body sent successfully DAVIX(socket): Request sent; retry is 0. < HTTP/1.1 401 Unauthorized < Date: Thu, 28 Oct 2021 18:01:19 GMT < Server: Apache/2.4.37 (Unix) < WWW-Authenticate: Digest realm="WebDAV", nonce="kv9pf23PBQA=87dcf2978519b77f6a33ecf1023df29d2c0c5cd6", algorithm=MD5, qop="auth" < Content-Length: 381 < Keep-Alive: timeout=5, max=100 < Connection: Keep-Alive < Content-Type: text/html; charset=iso-8859-1 < DAVIX(socket): End of headers. DAVIX(socket): Running post_headers hooks DAVIX(ssl): Disable Session recycling DAVIX(http): <- negotiateRequest DAVIX(core): Destroy HttpRequest DAVIX(socket): sess: Closing connection. DAVIX(socket): sess: Connection closed. DAVIX(ssl): Disable Session recycling DAVIX(socket): Running destroy hooks. DAVIX(socket): Request ends. DAVIX(socket): sess: Destroying session. DAVIX(posix): <- opendirpp (Davix::HttpRequest) Error: HTTP 401 : Authentification Error


7. This scenario happens also for other commands like `davix-get`, etc.

Thanks a lot,
Shahram.
mpatrascoiu commented 2 years ago

Hello Shahram,

Great problem report! I will look into why it fails. From the logs, it seems the client login callback is never invoked.

Cheers, Mihai

kelson42 commented 2 years ago

Seems I'm impacted by the same bug

felix-egli commented 2 years ago

I'm affected by this bug as well.

nachter commented 1 year ago

I came across this error too. With davix version 0.8.2. When I compare the debug-output of davix-version 0.7.6: In 0.7.6 there is this line: creating session keys... httpsnextcloud.gbv.de This line is completly missing in 0.8.2

Just a hint, perhaps this helps.

nachter commented 1 year ago

So, the message about creating session keys is just missing in the output for 0.8.x, it's just not present in the code of 0.8.x. So this doesnt seem to be the problem.

But for 0.8.x in the debug-log, I get the message "DAVIX(ssl): Disable Session recycling"

Perhaps there is a case missing (so for username/passphrase authentication), when the session recycling should not be disabled?

nachter commented 1 year ago

Hello! I now compiled and tested the login_callback_0.8.0 branch. It seems to work fine with this branch. Thanks for having a look on this!

crstmkt commented 1 year ago

Hello, I'm affected by this bug aswell.

TGion commented 1 year ago

Hi there, I am not 100% sure this is the same issue / connected, but as far I could tell, there weren't any fixes about this issue yet. Sorry in advance if this is another issue / misconfiguration on my part. I am trying to access my Nextcloud instance with davix 0.8.4. on my FreeBSD system:

  [USERNAME@localhost.gion.io ~]$ davix-ls --debug --userlogin USERNAME --userpass PASSWORD davs://cloud.gion.io/remote.php/dav/files/USERNAME/
    DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 1 locks.

    DAVIX(core): HTTP/SSL Session caching ENABLED
    DAVIX(core): Redirection Session caching ENABLED
    DAVIX(core): libdavix path /usr/local/lib/libdavix.so.0, version: 0.8.4
    DAVIX(posix):  -> opendirpp
    DAVIX(http): Create HttpRequest for davs://cloud.gion.io/remote.php/dav/files/USERNAME/
    DAVIX(http):  -> negotiateRequest
    DAVIX(http): NEON start internal request
    DAVIX(http): no cached ne_session, create a new one 
    DAVIX(http): HTTP session to https://cloud.gion.io:443 begins.
    DAVIX(ssl): ssl: SNI enabled by default.
    DAVIX(http): configure session...
    DAVIX(http): define connection timeout to 30
    DAVIX(http): enable login/password authentication
    DAVIX(http): enable client cert authentication by callback 
    DAVIX(ssl): ah_create, for WWW-Authenticate
    DAVIX(socket): Running pre_send hooks
    > PROPFIND /remote.php/dav/files/USERNAME/ HTTP/1.1
    > User-Agent: libdavix/0.8.4 neon/0.0.29
    > Keep-Alive: 
    > Connection: Keep-Alive
    > TE: trailers
    > Host: cloud.gion.io
    > Depth: 1
    > Content-Length: 303
    > 

    DAVIX(socket): Sending request-line and headers:
    DAVIX(socket): Doing DNS lookup on cloud.gion.io...
    DAVIX(ssl): Doing SSL negotiation.
    DAVIX(ssl): Chain depth: 2
    DAVIX(ssl): ssl: Match common name '*.gion.io' against ''
    DAVIX(ssl): ssl: Match common name 'gion.io' against ''
    DAVIX(ssl): Identity match for '': bad
    DAVIX(ssl): Cert #0:
    DAVIX(ssl): Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number:
                04:b5:18:58:80:3d:d6:ec:2a:12:4d:db:84:25:11:78:0c:39
            Signature Algorithm: sha256WithRSAEncryption
            Issuer: C=US, O=Let's Encrypt, CN=R3
            Validity
                Not Before: May 22 10:48:47 2023 GMT
                Not After : Aug 20 10:48:46 2023 GMT
            Subject: CN=gion.io
            Subject Public Key Info:
                Public Key Algorithm: rsaEncryption
                    RSA Public-Key: (2048 bit)
                    Modulus:
                        00:f2:0a:8d:54:41:ce:9e:e9:83:a3:cd:6f:ae:08:
                        24:f0:b8:6f:b2:65:0e:53:e3:c3:3d:0f:1c:f2:fb:
                        ee:f6:cf:2c:eb:88:98:e9:14:10:83:34:00:9f:a9:
                        08:ee:88:e8:21:a2:71:cf:61:72:c5:d6:52:f5:6c:
                        d4:c1:27:d5:fe:f1:8d:d6:c3:b3:2f:1d:0f:f7:fc:
                        b8:a3:c0:53:6a:38:cd:0b:ed:c3:47:41:ed:08:50:
                        37:8a:f9:73:d2:76:a2:30:8f:4a:2e:b2:a5:b5:8e:
                        11:3b:4c:e0:1f:df:ad:67:23:1f:9a:79:f1:03:b1:
                        10:d4:1f:9b:7c:aa:1e:ba:97:06:78:8c:04:d4:4f:
                        56:3c:52:d7:f1:2e:0e:f4:61:c2:24:19:92:f0:0a:
                        de:0b:3a:5e:4a:6c:3c:d9:95:b5:7f:31:19:d4:cc:
                        82:bc:7b:0b:fe:76:03:34:42:dd:88:50:17:95:ba:
                        76:f5:66:95:6c:a8:e7:74:d5:8b:c0:72:ff:b3:94:
                        a8:eb:9a:ff:a3:2b:f9:58:1f:a9:62:0b:a1:35:3f:
                        a0:04:24:d0:42:7b:07:2a:9c:15:8b:ae:d5:0a:ea:
                        6c:d6:a4:68:86:b4:e6:ae:a2:0a:e3:f4:8d:be:63:
                        ac:5e:9c:2c:65:3b:16:b1:54:8a:3a:74:30:b4:50:
                        19:b1
                    Exponent: 65537 (0x10001)
            X509v3 extensions:
                X509v3 Key Usage: critical
                    Digital Signature, Key Encipherment
                X509v3 Extended Key Usage: 
                    TLS Web Server Authentication, TLS Web Client Authentication
                X509v3 Basic Constraints: critical
                    CA:FALSE
                X509v3 Subject Key Identifier: 
                    9F:AF:79:EA:A7:11:8C:28:EB:B2:5C:76:AF:97:E1:D3:2E:F4:8C:74
                X509v3 Authority Key Identifier: 
                    keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6

                Authority Information Access: 
                    OCSP - URI:http://r3.o.lencr.org
                    CA Issuers - URI:http://r3.i.lencr.org/

                X509v3 Subject Alternative Name: 
                    DNS:*.gion.io, DNS:gion.io
                X509v3 Certificate Policies: 
                    Policy: 2.23.140.1.2.1
                    Policy: 1.3.6.1.4.1.44947.1.1.1
                      CPS: http://cps.letsencrypt.org

                CT Precertificate SCTs: 
                    Signed Certificate Timestamp:
                        Version   : v1 (0x0)
                        Log ID    : 7A:32:8C:54:D8:B7:2D:B6:20:EA:38:E0:52:1E:E9:84:
                                    16:70:32:13:85:4D:3B:D2:2B:C1:3A:57:A3:52:EB:52
                        Timestamp : May 22 11:48:47.518 2023 GMT
                        Extensions: none
                        Signature : ecdsa-with-SHA256
                                    30:45:02:20:66:36:CF:29:95:A1:F7:54:E6:90:42:A1:
                                    F2:09:FD:50:A5:95:83:46:FA:64:18:23:61:69:4C:70:
                                    07:10:AE:9E:02:21:00:E2:3A:12:85:52:A5:6F:63:DE:
                                    EF:9E:EE:3B:46:DF:50:92:62:D2:9F:BC:DB:B1:5D:40:
                                    9C:DA:94:79:48:62:2D
                    Signed Certificate Timestamp:
                        Version   : v1 (0x0)
                        Log ID    : AD:F7:BE:FA:7C:FF:10:C8:8B:9D:3D:9C:1E:3E:18:6A:
                                    B4:67:29:5D:CF:B1:0C:24:CA:85:86:34:EB:DC:82:8A
                        Timestamp : May 22 11:48:47.541 2023 GMT
                        Extensions: none
                        Signature : ecdsa-with-SHA256
                                    30:45:02:20:27:1C:80:5D:5B:75:1B:87:7F:92:6A:57:
                                    DC:49:0C:54:CB:9B:50:20:66:84:56:41:52:A6:91:5D:
                                    B2:
    DAVIX(ssl): ssl: Match common name 'R3' against ''
    DAVIX(ssl): Identity match for '': bad
    DAVIX(ssl): Cert #1:
    DAVIX(ssl): Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number:
                91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a
            Signature Algorithm: sha256WithRSAEncryption
            Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
            Validity
                Not Before: Sep  4 00:00:00 2020 GMT
                Not After : Sep 15 16:00:00 2025 GMT
            Subject: C=US, O=Let's Encrypt, CN=R3
            Subject Public Key Info:
                Public Key Algorithm: rsaEncryption
                    RSA Public-Key: (2048 bit)
                    Modulus:
                        00:bb:02:15:28:cc:f6:a0:94:d3:0f:12:ec:8d:55:
                        92:c3:f8:82:f1:99:a6:7a:42:88:a7:5d:26:aa:b5:
                        2b:b9:c5:4c:b1:af:8e:6b:f9:75:c8:a3:d7:0f:47:
                        94:14:55:35:57:8c:9e:a8:a2:39:19:f5:82:3c:42:
                        a9:4e:6e:f5:3b:c3:2e:db:8d:c0:b0:5c:f3:59:38:
                        e7:ed:cf:69:f0:5a:0b:1b:be:c0:94:24:25:87:fa:
                        37:71:b3:13:e7:1c:ac:e1:9b:ef:db:e4:3b:45:52:
                        45:96:a9:c1:53:ce:34:c8:52:ee:b5:ae:ed:8f:de:
                        60:70:e2:a5:54:ab:b6:6d:0e:97:a5:40:34:6b:2b:
                        d3:bc:66:eb:66:34:7c:fa:6b:8b:8f:57:29:99:f8:
                        30:17:5d:ba:72:6f:fb:81:c5:ad:d2:86:58:3d:17:
                        c7:e7:09:bb:f1:2b:f7:86:dc:c1:da:71:5d:d4:46:
                        e3:cc:ad:25:c1:88:bc:60:67:75:66:b3:f1:18:f7:
                        a2:5c:e6:53:ff:3a:88:b6:47:a5:ff:13:18:ea:98:
                        09:77:3f:9d:53:f9:cf:01:e5:f5:a6:70:17:14:af:
                        63:a4:ff:99:b3:93:9d:dc:53:a7:06:fe:48:85:1d:
                        a1:69:ae:25:75:bb:13:cc:52:03:f5:ed:51:a1:8b:
                        db:15
                    Exponent: 65537 (0x10001)
            X509v3 extensions:
                X509v3 Key Usage: critical
                    Digital Signature, Certificate Sign, CRL Sign
                X509v3 Extended Key Usage: 
                    TLS Web Client Authentication, TLS Web Server Authentication
                X509v3 Basic Constraints: critical
                    CA:TRUE, pathlen:0
                X509v3 Subject Key Identifier: 
                    14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
                X509v3 Authority Key Identifier: 
                    keyid:79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E

                Authority Information Access: 
                    CA Issuers - URI:http://x1.i.lencr.org/

                X509v3 CRL Distribution Points: 

                    Full Name:
                      URI:http://x1.c.lencr.org/

                X509v3 Certificate Policies: 
                    Policy: 2.23.140.1.2.1
                    Policy: 1.3.6.1.4.1.44947.1.1.1

        Signature Algorithm: sha256WithRSAEncryption
             85:ca:4e:47:3e:a3:f7:85:44:85:bc:d5:67:78:b2:98:63:ad:
             75:4d:1e:96:3d:33:65:72:54:2d:81:a0:ea:c3:ed:f8:20:bf:
             5f:cc:b7:70:00:b7:6e:3b:f6:5e:94:de:e4:20:9f:a6:ef:8b:
             b2:03:e7:a2:b5:16:3c:91:ce:b4:ed:39:02:e7:7c:25:8a:47:
             e6:65:6e:3f:46:f4:d9:f0:ce:94:2b:ee:54:ce:12:bc:8c:27:
             4b:b8:c1:98:2f:a2:af:cd:71:91:4a:08:b7:c8:b8:23:7b:04:
             2d:08:f9:08:57:3e:83:d9:04:33:0a:47:21:78:09:82:27:c3:
             2a:c8:9b:b9:ce:5c:f2:64:c8:c0:be:79:c0:4f:8e:6d:44:0c:
             5e:92:bb:2e:f7:8b:10:e1:e8:1d:44:29:db:59:20:ed:63:b9:
             21:f8:12:26:94:93:57:a0:1d:65:04:c1:0a:22:ae:10:0d:43:
             97:a1:18:1f:7e:e0:e0:86:37:b5:5a:b1:bd:30:bf:87:6e:2b:
             2a:ff:21:4e:1b:05:c3:f5:18:97:f0:5e:ac:c3:a5:b8:6a:f0:
             2e:bc:3b:33:b9:ee:4b:de:cc:fc:e4:af:84:0b:86:3f:c0:55:
             43:36:f6:68:e1:36:17:6a:8e:99:d1:ff:a5:40:a7:34:b7:c0:
             d0:63:39:35:39:75:6e:f2:ba:76:c8:93:02:e9:a9:4b:6c:17:
             ce:0c:02:d9:bd:81:fb:9f:b7:68:d4:06:65:b3:82:3d:77:53:
             f8:8e:79:03:ad:0a:31:07:75:2a:43:d8:55:97:72:c4:29:0e:
             f7:c4:5d:4e:c8:ae:46:84:30:d7:f2:85:5f:18:a1:79:bb:e7:
             5e:70:8b:07:e1:86:93:c3:b9:8f:dc:61:71:25:2a:af:df:ed:
             25:50:52:68:8b:92:dc:e5:d6:b5:e3:da:7d:d0:87:6c:84:21:
             31:ae:82:f5:fb:b9:ab:c8:89:17:3d:e1:4c:e5:38:0e:f6:bd:
             2b:bd:96:
    DAVIX(ssl): ssl: Match common name '*.gion.io' against 'cloud.gion.io'
    DAVIX(ssl): Identity match for 'cloud.gion.io': good
    DAVIX(socket): Sending request body:
    DAVIX(socket): Request body sent successfully
    DAVIX(socket): Request sent; retry is 0.
    < HTTP/1.1 401 Unauthorized
    < Server: nginx
    < Date: Sun, 04 Jun 2023 09:50:03 GMT
    < Content-Type: application/xml; charset=utf-8
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < Keep-Alive: timeout=5
    < Set-Cookie: [removed]
    < Expires: Thu, 19 Nov 1981 08:52:00 GMT
    < Cache-Control: no-store, no-cache, must-revalidate
    < Pragma: no-cache
    < Content-Security-Policy: default-src 'none';
    < WWW-Authenticate: Basic realm="cloud | gion . io", charset="UTF-8"
    < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
    < Referrer-Policy: no-referrer
    < X-Content-Type-Options: nosniff
    < X-Download-Options: noopen
    < X-Permitted-Cross-Domain-Policies: none
    < X-XSS-Protection: 1; mode=block
    < Permissions-Policy: camera=(), microphone=(), geolocation=()
    < X-Frame-Options: SAMEORIGIN
    < X-Robots-Tag: noindex,nofollow
    < 
    DAVIX(socket): End of headers.
    DAVIX(socket): Running post_headers hooks
    DAVIX(ssl): Disable Session recycling
    DAVIX(http):  <- negotiateRequest
    DAVIX(core): Destroy HttpRequest
    DAVIX(socket): sess: Closing connection.
    DAVIX(socket): sess: Connection closed.
    DAVIX(ssl): Disable Session recycling
    DAVIX(socket): Running destroy hooks.
    DAVIX(socket): Request ends.
    DAVIX(socket): sess: Destroying session.
    DAVIX(posix):  <- opendirpp
    (Davix::HttpRequest) Error: HTTP 401 : Authentication Error

What else has been tested:

What has not been tested

Help, a new version, or hints about any misconfiguration on my parts would be really appreciated. Thanks a lot in advance. If I could be of any help with more testing, just let me know!

mpatrascoiu commented 1 year ago

Hello all,

This problem appeared with the davix v0.8.0 release, which introduced libcurl alongside libneon for the backend. A large refactoring was done to facilitate having two different backends, including introducing a new class hierarchy.

Unfortunately, during that refactoring, the hooks mechanism used with libneon was lost. Previously, hooks would be registered for different scenarios. One such hook would be triggered when the server requests Basic authentication via the response headers.

In branch login_callback_0.8.0, I'm trying to get the hooks mechanism to work again. However, this introduces a memory corruption in the session object management. The corruption does not show via the command line tool (e.g.: davix-get <url>), but if you were to reuse the same Davix object, it will show up sooner or later. For this reason, the branch is not ready to be merged.

Ultimately, we we plan to decommission the libneon backend. Due to this, I gave up on getting the hooks mechanism to work, especially since it is only tied to libneon and not libcurl.

For what concerns the username + password login, the davix + libcurl backend simply does not do it at all. If there's high demand for this functionality, I can add it to the libcurl backend. This means you'd have it only when activating libcurl:

$ DAVIX_USE_LIBCURL=1 davix-get <url>
TGion commented 1 year ago

Hey @mpatrascoiu ,

first of all thanks a lot for your effort and your clarification!

I am a bit confused about your last paragraph (most certainly because of my lack of knowledge): How do you usually connect to a WebDAV Server, because I honestly don't know any other way then by basic authentification (username + password). Iam trying to access my Nextcloud instance via WebDAV and couldnt find another way then by username + password. Long story short, I would think the demand for it is quite high.

If you have the time I would appreciate also some clarification on how other forms of authentification work with davix. I couldnt really find a lot in the documentation which would be useful for my appliance.

Thanks a lot again and please tell me if I could be of any help!

spacehorst commented 1 year ago

Hi @mpatrascoiu,

thanks for investigating on that issue.

Could you please clarify your statement of https://github.com/cern-fts/davix/issues/81#issuecomment-1584726593 Do I understand correct, that it would not be possible to use a WebDAV Server with Basic Authentification by davix >= 0.8.0 any more?

This would be a big drawback for us. We have been using davix for more than 10 years in our CI/CD pipeline. So we would like to encourage you to keep on with basic authentication.

spacehorst commented 1 year ago

As a workaround one could add the missing Authorization Header by hand:

echo -n "$username:$password" | base64 abc....xyz= davix-get -H "Authorization: Basic abc....xyz=" http://your.web/dav

TGion commented 1 year ago

Thanks for your workaround @spacehorst

mpatrascoiu commented 1 year ago

Hello @spacehorst , that's correct. With davix >= 0.8.0, there is no more Basic Authentication. And if we bring it back (I believe we should, it's more a question of effort and planning), it will only be available for the CURL backend.

Just to mention: your workaround should work well, cheers for that