`calendar.search`-method with timestamp filters yielding too much

ArtemIsmagilov commented 7 months ago

this code not correct return events.
start= 2023-12-01 00:00:00.793241+03:00

end= 2023-12-01 23:59:58.793241+03:00

start = datetime.now(ZoneInfo("Europe/Moscow")).replace(hour=0, minute=0, second=0)
end = start.replace(hour=23, minute=59, second=58)
print(start, end)
for e in c.search(comp_class=Event, start=start, end=end,  sort_keys=("DTSTART",)):
if e.icalendar_component.get("X-TELEMOST-CONFERENCE"):
    print(Conference(e.icalendar_component, "Europe/Moscow"))

Result -

<Conference uid:'1o1h3lz1v41v9n4agilx3yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>

<Conference uid:'1o1h3lz1v37k5aqz0qnj5yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>

<Conference uid:'1o1h3lz1v37k6id2q50t0yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>

<Conference uid:'1o1h3lz1v32qsp1riff59yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>

<Conference uid:'1o1h3lz1v399ejhoh4em9yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>

<Conference uid:'1o1h3lz1v43nqe7uk4jx6yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T17:48:00+03:00' dtend:'2023-12-01T18:18:00+03:00' >

Problem - not sorted and wrong searching start and stop

tobixen commented 7 months ago

Hm ... this is two issues indeed, problems with sorting and problems with the search. I will raise a separate issue for the sorting problem.

tobixen commented 7 months ago

So you have three events marked that starts midnight MSK 2nd of December and ends midnight MSK 3rd of December (are they really stored with timestamps rather than dates in the calendar? May I see the full icalendar code?). They should not be returned because the end of the search is set to 23:59:59 at the 1st of December (I don't think the caldav standard supports milliseconds in the search timestamps).

One possible reason could be that the search-code handles timezones wrongly. I don't think we have test code exercising timestamp search with timezones. Could you please try and see if it works out correctly when passing UTC timestamps? Something like this:

from datetime import timezone, datetime, timedelta
start = datetime(2023,12,1,0,0,0,tzinfo=timezone.utc)
end = start + timedelta(days=1)

Try with different values for start, midnight MSK is 21:00:00 UTC.

Does yandex offer caldav support? Is it possible to create a yandex account dedicated for running the caldav test suite?

ArtemIsmagilov commented 7 months ago

Yandex calendar works using the CalDAV protocol. Your example did not change the result. Yes, you can create an account in Yandex Calendar. Go to the Yandex ID website https://id.yandex.ru/ Log in or register Go to Security At the very bottom are the Application passwords Click on the CalDAV Calendar and create a password To integrate, you will need a Yandex account login and a previously created password

ArtemIsmagilov commented 7 months ago

my code

import logging, caldav
from caldav import Event
from datetime import datetime, UTC
logging.basicConfig(level=logging.DEBUG)
login, token =  login, token 
with caldav.DAVClient(url=f'https://{login}:{token}@caldav.yandex.ru') as client:
    principal = client.principal()
    c = principal.calendar(cal_id=cal_id)
    start = datetime.now(ZoneInfo("Europe/Moscow")).replace(hour=0, minute=0, second=0)
    end = start.replace(hour=23, minute=59, second=59) 
    print(start, end)
    for e in c.search(comp_class=Event, start=start, end=end,  sort_keys=("DTSTART",)):
        if e.icalendar_component.get("X-TELEMOST-CONFERENCE"):
            print(Conference(e.icalendar_component, "Europe/Moscow"))

you can view full code in an open source project https://github.com/ArtemIsmagilov/mm-yc-notify

tobixen commented 7 months ago

I will try to register my own yandex calendar and play with it, but I cannot tell when I will get time for that.

tobixen commented 7 months ago

I should also make test code testing the search algorithm with various timezones and see if that may be an issue. Again, I cannot tell when I will get time for that.

ArtemIsmagilov commented 7 months ago

What I learned is that freebusy is not supported in Yandex Calendar. Also, this calendar is constantly changing some event attributes, which is why the token changes even if you don’t change the events at all.

tobixen commented 5 months ago

Unable to register account with yandex, it wants to confirm my telephone number, but the SMS doesn't get through :-(

python-caldav / caldav

`calendar.search`-method with timestamp filters yielding too much #351