Test fails against radicale

[smurfix]

$ pytest -sx tests/
tests/test_caldav.py
F

========================================== FAILURES ===========================================
_____________________________ TestScheduling.testInviteAndRespond _____________________________

self = ScheduleInbox(None), client = None
principal = Principal(http://dav.local/tester1/), url = None

    def __init__(
        self,
        client: Optional["DAVClient"] = None,
        principal: Optional[Principal] = None,
        url: Union[str, ParseResult, SplitResult, URL, None] = None,
    ) -> None:
        """
        Will locate the mbox if no url is given
        """
        super(ScheduleMailbox, self).__init__(client=client, url=url)
        self._items = None
        if not client and principal:
            self.client = principal.client
        if not principal and client:
            if self.client is None:
                raise ValueError("Unexpected value None for self.client")

            principal = self.client.principal
        if url is not None:
            if client is None:
                raise ValueError("Unexpected value None for client")

            self.url = client.url.join(URL.objectify(url))
        else:
            if principal is None:
                raise ValueError("Unexpected value None for principal")

            if self.client is None:
                raise ValueError("Unexpected value None for self.client")

            self.url = principal.url
            try:
                # we ignore the type here as this is defined in sub-classes only; require morechanges to
                # properly fix in a future revision
>               self.url = self.client.url.join(URL(self.get_property(self.findprop())))  # type: ignore

caldav/objects.py:1769:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
caldav/lib/url.py:181: in join
    pathAsString = str(path)
caldav/lib/url.py:106: in __str__
    return to_normal_str(self.__unicode__())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <[ValueError('Unexpected value None for self.url_parsed') raised in repr()] URL object at 0x7f5d8919b170>

    def __unicode__(self) -> str:
        if self.url_raw is None:
            if self.url_parsed is None:
>               raise ValueError("Unexpected value None for self.url_parsed")
E               ValueError: Unexpected value None for self.url_parsed

caldav/lib/url.py:112: ValueError

During handling of the above exception, another exception occurred:

self = <tests.test_caldav.TestScheduling object at 0x7f5d8756b770>

    def testInviteAndRespond(self):
        ## Look through inboxes of principals[0] and principals[1] so we can sort
        ## out existing stuff from new stuff
        if len(self.principals) < 2:
            pytest.skip("need 2 principals to do the invite and respond test")
        inbox_items = set(
>           [x.url for x in self.principals[0].schedule_inbox().get_items()]
        )

tests/test_caldav.py:415:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
caldav/objects.py:737: in schedule_inbox
    return ScheduleInbox(principal=self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = ScheduleInbox(None), client = None
principal = Principal(http://dav.local/tester1/), url = None

    def __init__(
        self,
        client: Optional["DAVClient"] = None,
        principal: Optional[Principal] = None,
        url: Union[str, ParseResult, SplitResult, URL, None] = None,
    ) -> None:
        """
        Will locate the mbox if no url is given
        """
        super(ScheduleMailbox, self).__init__(client=client, url=url)
        self._items = None
        if not client and principal:
            self.client = principal.client
        if not principal and client:
            if self.client is None:
                raise ValueError("Unexpected value None for self.client")

            principal = self.client.principal
        if url is not None:
            if client is None:
                raise ValueError("Unexpected value None for client")

            self.url = client.url.join(URL.objectify(url))
        else:
            if principal is None:
                raise ValueError("Unexpected value None for principal")

            if self.client is None:
                raise ValueError("Unexpected value None for self.client")

            self.url = principal.url
            try:
                # we ignore the type here as this is defined in sub-classes only; require morechanges to
                # properly fix in a future revision
                breakpoint()
                self.url = self.client.url.join(URL(self.get_property(self.findprop())))  # type: ignore
            except:
                logging.error("something bad happened", exc_info=True)
                error.assert_(self.client.check_scheduling_support())
                self.url = None
                # we ignore the type here as this is defined in sub-classes only; require morechanges to
                # properly fix in a future revision
>               raise error.NotFoundError(
                    "principal has no %s.  %s"
                    % (str(self.findprop()), error.ERR_FRAGMENT)  # type: ignore
                )
E               caldav.lib.error.NotFoundError: NotFoundError at 'principal has no <?xml version='1.0' encoding='utf-8'?>
E               <C:schedule-inbox-URL xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"/>
E               .  Please consider raising an issue at https://github.com/python-caldav/caldav/issues or reach out to t-caldav@tobixen.no, include this error and the traceback and tell whatserver you are using', reason no reason

caldav/objects.py:1776: NotFoundError
-------------------------------------- Captured log call --------------------------------------
ERROR    root:objects.py:1771 something bad happened
Traceback (most recent call last):
  File "/src/caldav_orig/caldav/objects.py", line 1769, in __init__
    self.url = self.client.url.join(URL(self.get_property(self.findprop())))  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/caldav_orig/caldav/lib/url.py", line 181, in join
    pathAsString = str(path)
                   ^^^^^^^^^
  File "/src/caldav_orig/caldav/lib/url.py", line 106, in __str__
    return to_normal_str(self.__unicode__())
                         ^^^^^^^^^^^^^^^^^^
  File "/src/caldav_orig/caldav/lib/url.py", line 112, in __unicode__
    raise ValueError("Unexpected value None for self.url_parsed")
ValueError: Unexpected value None for self.url_parsed
ERROR    caldav:error.py:34 Deviation from expectations found.  Please consider raising an issue at https://github.com/python-caldav/caldav/issues or reach out to t-caldav@tobixen.no, include this error and the traceback and tell what server you are using
Traceback (most recent call last):
  File "/src/caldav_orig/caldav/objects.py", line 1769, in __init__
    self.url = self.client.url.join(URL(self.get_property(self.findprop())))  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/caldav_orig/caldav/lib/url.py", line 181, in join
    pathAsString = str(path)
                   ^^^^^^^^^
  File "/src/caldav_orig/caldav/lib/url.py", line 106, in __str__
    return to_normal_str(self.__unicode__())
                         ^^^^^^^^^^^^^^^^^^
  File "/src/caldav_orig/caldav/lib/url.py", line 112, in __unicode__
    raise ValueError("Unexpected value None for self.url_parsed")
ValueError: Unexpected value None for self.url_parsed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/src/caldav_orig/caldav/lib/error.py", line 31, in assert_
    assert condition
           ^^^^^^^^^
AssertionError

conf_private.py:

caldav_servers = [
  {
        'enable': True,
        'url': 'http://dav.local',
        'username': 'tester',
        'password': 'wonttellyou',
        'incompatibilities': ['radicale_breaks_on_category_search'],  # doesn't affect the problem
  }
]

rfc6638_users = []
for i in (1, 2, 3, 4, 5):
    sogo = caldav_servers[-1].copy()
    sogo['username'] = 'tester%i' % i
    rfc6638_users.append(sogo)

Radicale versions tested: 3.1.8, 3.2.3

[tobixen]

I'd like to write up a "caldav server compatibility checker"-tool, it's currently a pain that no caldav servers are perfect enough to not break the tests.

Try this

caldav_servers = [
    {
        'url': ...,
        'username': ...,
        'password': ...,
        'incompatibilities': compatibility_issues.redicale
    },
]

[smurfix]

I did that too. tests/test_caldav.py::TestScheduling::testInviteAndRespond - AssertionError breaks anyway.

[tobixen]

That's strange,'no_scheduling' is included in that list.

Hm. When I try, with the master branch, and radicale-3.2.3-1 installed through archlinux, I now get two errors:

FAILED tests/test_caldav.py::TestLocalRadicale::testTodoDatesearch - caldav.lib.error.ReportError: ReportError at '500 Internal Server Error
FAILED tests/test_caldav.py::TestLocalRadicale::testRecurringDateSearch - AssertionError: assert 0 == 1

I will see if I get time to look more into it during the day ... but this is not the same errors as you observe?

[smurfix]

Don't worry, I see those two too. :-/

[tobixen]

I'm a bit swamped those days, but I will try to look into those two during the day at least.

[tobixen]

It's weird, because I don't think it's that long since last I ran the test towards Radicale. Perhaps a new version is out with degraded compatibility.

So the testTodoDatesearch (perhaps the test deserves a better name) causes a 500 Internal Server Error while doing a timestamp search for tasks. It's quite common that calendar servers yields a 500-error when throwing difficult corner cases on them - I would claim this is a server error - but the workaround is quite easy, just add 'no_todo_datesearch' to the compatibility issue list.

The testRecurringDateSearch is more difficult to work around. Radicale is apparently doing server-side recurring event expansion wrongly. The newer versions of the library does support client-side event expansion, but it's currently only done when the server is obviously not supporting expansion.

Possible workarounds/solutions:

• Another compatibility flag for test purposes, disabling this test (or this part of the test).
• Don't do server-side expansion, do it on the client-side always
• Send some compatibility_hints to the client connection object, and use this to decide weather to do server side or client side expansion
• New attribute in the search method to toggle between server-side and client-side expansion

I don't like the second option, as it may cause subtle changes in how the library works in production environments, breaking backward-compatibility. The third option is possibly the best, but requires most work - and it should be well thought through. I think the last option will be too bloated.

[tobixen]

So I'll go for the first option as a temp workaround just to get the tests working (again), and the third option will be for a later release.