search

pimutils / vdirsyncer

📇 Synchronize calendars and contacts.
https://vdirsyncer.pimutils.org/
Other
1.57k stars 163 forks source link

Sync attempt causes 401 auth required after 0.19.2 -> 0.19.3 update #1138

Open cbirchinger opened 1 month ago

cbirchinger commented 1 month ago

After updating from 0.19.2 to 0.19.3 the server rejects the connection with 401 instead of 207 (-vdebug outputs below)

Nothing other than the update from 0.19.2 to 0.19.3 was changed. Same Python, same configuration, same server setup. Downgrading to 0.19.2 again fixes the issue and the sync works again

Server: Radicale 3.2.3 behind Apache 2.4.62 (mod_proxy) OS: Gentoo Linux (Server and Client) Python: 3.12

Hardcoded password instead of password.fetch was also tried without success.

-vdebug output of 0.19.3 (FAIL)

debug: Fetching value for password.fetch with command strategy.
debug: Found cached value for ['command', '~/.config/vdirsyncer/pass.sh'].
Syncing user_contacts/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.3', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
Syncing user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.3', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
Syncing user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.3', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
debug: 401
debug: <CIMultiDictProxy('Date': 'Thu, 12 Sep 2024 17:44:17 GMT', 'Server': 'Apache', 'WWW-Authenticate': 'Basic realm="Radicale - Password Required"', 'Content-Length': '448', 'Content-Type': 'text/html; charset=iso-8859-1')>
debug: <StreamReader 448 bytes eof>
error: Unknown error occurred for user_contacts/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX: 401, message='Unauthorized', url='https://www.domain.xxx/radicale/joker/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/'
error: Use `-vdebug` to see the full traceback.
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/cli/tasks.py", line 74, in sync_collection
debug:     await sync.sync(
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/sync/__init__.py", line 150, in sync
debug:     b_nonempty = await b_info.prepare_new_status()
debug:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/sync/__init__.py", line 55, in prepare_new_status
debug:     async for href, etag in self.storage.list():  # type: ignore[attr-defined]
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/storage/dav.py", line 672, in list
debug:     response = await self.session.request(
debug:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/storage/dav.py", line 413, in request
debug:     return await http.request(method, url, session=session, **more)
debug:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
debug:   File "/usr/lib/python3.12/site-packages/vdirsyncer/http.py", line 216, in request
debug:     response.raise_for_status()
debug:   File "/usr/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1093, in raise_for_status
debug:     raise ClientResponseError(
debug: 401
debug: <CIMultiDictProxy('Date': 'Thu, 12 Sep 2024 17:44:17 GMT', 'Server': 'Apache', 'WWW-Authenticate': 'Basic realm="Radicale - Password Required"', 'Content-Length': '448', 'Content-Type': 'text/html; charset=iso-8859-1')>
debug: <StreamReader 448 bytes eof>
error: Unknown error occurred for user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX: 401, message='Unauthorized', url='https://www.domain.xxx/radicale/joker/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/'
error: Use `-vdebug` to see the full traceback.

-vdebug output of 0.19.2 (OK)

debug: Fetching value for password.fetch with command strategy.
debug: Found cached value for ['command', '~/.config/vdirsyncer/pass.sh'].
Syncing user_contacts/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.2', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
Syncing user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.2', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
Syncing user_calendar/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX
debug: ====================
debug: PROPFIND https://www.domain.xxx/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/
debug: {'User-Agent': 'vdirsyncer/0.19.2', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'<?xml version="1.0" encoding="utf-8" ?>\n            <propfind xmlns="DAV:">\n                <prop>\n                    <resourcetype/>\n                    <getcontenttype/>\n                    <getetag/>\n                </prop>\n            </propfind>\n            '
debug: Sending request...
debug: 207
debug: <CIMultiDictProxy('Date': 'Thu, 12 Sep 2024 17:43:26 GMT', 'Server': 'WSGIServer/0.2 CPython/3.12.3', 'DAV': '1, 2, 3, calendar-access, addressbook, extended-mkcol', 'Content-Type': 'text/xml; charset=utf-8', 'Content-Encoding': 'gzip', 'Content-Length': '1132')>
debug: <StreamReader>
debug: Already normalized: '/radicale/user/XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX/'

config

[general]
status_path = "~/.local/share/vdirsyncer/"

# ---[ CardDav contacts sync ]--------------------------------------------------------
[pair user_contacts]
a = "user_contacts_local"
b = "user_contacts_remote"
collections = ["XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX"]
metadata = ["displayname"]

[storage user_contacts_local]
type = "filesystem"
path = "~/.contacts/"
fileext = ".vcf"

[storage user_contacts_remote]
type = "carddav"
url = "https://www.domain.xxx/radicale/user/"
username = "user"
password.fetch = ["command", "~/.config/vdirsyncer/pass.sh"]
# -------------------------------------------------------------------------------------

# ---[ CalDav calendar sync ]---------------------------------------------------------
[pair user_calendar]
a = "user_calendar_local"
b = "user_calendar_remote"
collections = ["XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX", "XXXXXXXXXXXXX-XXXXXXXXXXXXXX-XXXXXXXXXXXXXX"]
metadata = ["displayname", "color"]

[storage user_calendar_local]
type = "filesystem"
path = "~/.calendars/"
fileext = ".ics"

[storage user_calendar_remote]
type = "caldav"
url = "https://www.domain.xxx/radicale/user/"
username = "user"
password.fetch = ["command", "~/.config/vdirsyncer/pass.sh"]
# -------------------------------------------------------------------------------------

Apache vhost

RewriteEngine On
RewriteRule ^/radicale$ /radicale/ [R,L]

<Location "/radicale/">
    <IfModule security2_module>
        SecAuditEngine Off
        SecRuleEngine Off
    </IfModule>

    AuthType     Basic
    AuthName     "Radicale - Password Required"
    AuthUserFile "/etc/radicale/users"
    Require      valid-user

    ProxyPass        http://localhost:5232/ retry=0
    ProxyPassReverse http://localhost:5232/
    RequestHeader    set X-Script-Name /radicale
    RequestHeader    set X-Remote-User expr=%{REMOTE_USER}
</Location>
WhyNotHugo commented 1 month ago

cc: @malmeloo

malmeloo commented 1 month ago

I did a little digging and what I think is happening here is aiohttp dropping the Authorization header when it encounters a redirect. Could you post a few log lines from Apache to confirm that that's also what's going on here?

cbirchinger commented 1 month ago

Ah, aiohttp! I think i've found the issue.

Changed in version 3.0: Added support for ~/.netrc file.
Changed in version 3.9: Added support for reading HTTP Basic Auth credentials from ~/.netrc file.

It is using a credential entry in ~/.netrc for the autentication. Because in ~/.netrc only the machine name is configured so it seems to be using it for all requests on that host regardless of the subpath.

The question now is can it be changed so the values in ~/.config/vdirsyncer/config have higher priority than the generic ~/.netrc

EDIT: As far as i understand the aiohttp documentation ~/.netrc should only be used related to proxy authentication. Does it consider a web service behind a reverse proxy such as Apache or nginx needs to be treated like a Squid web cache that requires auth?

WhyNotHugo commented 1 month ago

I’ll take a patch that disables netrc support when invoking aiohttp. It isn’t a supported mechanism for Vdirsyncer.

Hawk777 commented 1 month ago

aiohttp 3.9 added support for using netrc for non-proxy auth.

malmeloo commented 1 month ago

That option should still be off by default though, so I'm not entirely convinced that's the problem here. But I will investigate a little more and see if I can repro.

cbirchinger commented 1 month ago

I don't see an explicit option for this. It's tied to the "trust_env" option in "aiohttp.ClientSession" though

        if auth is None and trust_env and self.url.host is not None:
            netrc_obj = netrc_from_env()
            with contextlib.suppress(LookupError):
                auth = basicauth_from_netrc(netrc_obj, self.url.host)

vdirsyncer currently is using "True": https://github.com/search?q=repo%3Apimutils%2Fvdirsyncer%20trust_env&type=code

Setting it to "False" fixes the issue for me. It also does disable a users ability to set the NETRC environment to have an alternative to the hardcoded ~/.netrc

malmeloo commented 1 month ago

Oh the option is enabled? Right, that explains the issue then. I think disabling it would also disable support for setting proxy details using env vars though, is that desirable?

cbirchinger commented 1 month ago

Not really, a lot more people probably use the "http_proxy" environment variables.

Knowing the root cause of the issue, I have some workarounds for the time being like running "NETRC=/dev/null vdirsyncer sync"

malmeloo commented 1 month ago

If that fixes it we could probably override that env var in vdirsyncer, as a workaround...

The more I learn about aiohttp, the more I'm starting to dislike it.

cbirchinger commented 1 month ago

Yes, please. Having the env var set to something like /dev/null by default would help. vdirsyncer has it's own username and password settings. I see no benefit in using the less flexible and if present probably used for something else ~/.netrc instead.

malmeloo commented 1 month ago

@cbirchinger could you try #1141 to see if it fixes the issue?

cbirchinger commented 1 month ago

@cbirchinger could you try #1141 to see if it fixes the issue?

Yes, that fixes the issue. Thank you.