QA on the basic_usage_examples.py needed

I'd like someone to do a QA on the basic_usage_examples.py from the master branch. Is it easy to understand how to use the library? Can it be improved? Is the language unclear?

This should preferably be done by someone who has not been doing significant contributions to the caldav library already, someone that doesn't know the code, but who has at least a basic understanding of python. :-)

I'm building a Qt task list app using this project, and I am pretty new to python. Here are some things that would be helpful to me for my use case.

Documentation specific to tasks. The previous basic_usage_expamples included some and how to mark it as completed, but its been removed.
Return a premade dictionary of tasks. Right now, I take the raw CalDAV data and create a dictionary myself by iterating over the data. Example of my local JSON file is below. I format the date from 20230201T234709Z to a native python datetime object.
How to edit specific tasks. Right now, I copy the data to a new dict entry, delete the original server version, and create an entire new todo for my edits.
My sync process is very hacky. I pull the server data and compare each task to the local dict. If a change is detected for a UID or a UID is removed, a new server connection is made each time to delete the upstream version and create a new one if it's needed.
In addition, I currently use two separate methods to sync. I pull from the server and modify the local dict to match the server or vice-versa. Every individual modification deletes the whole local dict, modifies it, and saves it. I have a cleaner method in mind, but its what Im doing now.

    "todos": {
        "0": {
            "BEGIN": "VTODO",
            "INCALENDAR": "Appointments",
            "VERSION": "2.0",
            "PRODID": "-//python-caldav//caldav//en_DK",
            "CATEGORIES": "GER 102",
            "DTSTAMP": "20230201T234709Z",
            "DUE": "date(2023, 02, 03)",
            "SUMMARY": "Vokabeln I + II S. 256 & S.265",
            "UID": "a6f05860-a2b4-11ed-affd-50c2e8eec295",
            "END": "VCALENDAR"
        },

Many of my issues are related to my lack of experience, but I still want to give my feedback here since my project relies on this repo.

Here's a link to my project; I'll remove it if you'd like. Not as intent to advertise but for you to see how I implement your project into mine.

Keep up the good work! Thanks!

Documentation specific to tasks. The previous basic_usage_expamples included some and how to mark it as completed, but its been removed.

Oh, really? Unintentional mistake in that case, I will look into it (when I get time for it).

Return a premade dictionary of tasks. Right now, I take the raw CalDAV data and create a dictionary myself by iterating over the data. Example of my local JSON file is below. I format the date from 20230201T234709Z to a native python datetime object.

Hmm ... can you explain it a bit better? What do you want to achieve?

How to edit specific tasks. Right now, I copy the data to a new dict entry, delete the original server version, and create an entire new todo for my edits.

Can search for it using the search-method ... or can fetch all tasks and filter through them ... I'll write up some examples for that (when I get time for it). (One problem though, what works with one calendar server does not always work with another - I've been writing a bit about that at https://github.com/tobixen/plann/blob/master/CALENDAR_SERVER_RECOMMENDATIONS.md)

My sync process is very hacky. I pull the server data and compare each task to the local dict. If a change is detected for a UID or a UID is removed, a new server connection is made each time to delete the upstream version and create a new one if it's needed.

Did you have a look into sync_examples.py? That one hasn't been maintained nor validated for some time though ...

(maybe a list would be more useful than a dict in your case?)

I'd be happy to help you (when/if I have time for it) with whatever questions you have wrg of DavTasks. I'm making my own tool as well - plann (which is more or less a rewrite of my calendar-cli). I'd be happy if you have any comments on https://github.com/tobixen/plann/blob/master/TASK_MANAGEMENT.md

Hmm ... can you explain it a bit better? What do you want to achieve?

I use JSON to store all of my appdata. Link to my code where I pull the dav data.

I take the raw task data "['BEGIN:VCALENDAR', 'VERSION:2.0', 'PRODID:-//python-caldav//caldav//en_DK', 'BEGIN:VTODO', 'CATEGORIES:GER 102', 'DTSTAMP:20230201T234709Z', 'DUE;VALUE=DATE:20230203', 'SUMMARY:Vokabeln I + II S. 256 & S.265', 'UID:a6ffd860-a2b4-11ed-affd-50c2e8eec295', 'END:VTODO', 'END:VCALENDAR', '']" and create a dict with an arbitrary key "i" and make its value a dict containing the key-value pairs from the list given to me above by iterating over each task the server returns in this list format.

By having all of the tasks keyed like this, I can easily reference any specific task by its key and without looping over the whole list. I suppose I could make the UID the key, but I can't assign one client side, so I'd have to push the new task and then pull it again to have its proper UID. From what I understand, dicts are a lot more efficient for my use case. I assign another key "INCALENDAR" for obvious reasons. I still haven't figured out if it's acceptable to connect the server everytime I create/ edit a new todo or if I should do it in bulk like I am now.

A big issue I'm facing is that with my current sync system, I end up iterating over the whole list multiple times when pushing or pulling, making a new server connection for every task that doesn't have a matching server version.

{
    "settings": {
        "URL": "",
        "USERNAME": "",
        "PASSWORD": "",
        "CALENDARS": {
             "Calendar ics link"
        }
    },
    "tags": {
        "MAT 126": {"Color": "#FFFFFF"},
    },
    "todos": {
        "0": {
            "BEGIN": "VTODO",
            "INCALENDAR": "Extracurricular", 
            "VERSION": "2.0",
            "PRODID": "-//python-caldav//caldav//en_DK",
            "CATEGORIES": "BBR",
            "DTSTAMP": "20230131T160642Z",
            "SUMMARY": "org update need phone numbers",
            "UID": "409bb052-a181-11ed-8f64-e0d045c542a2",
            "DUE": "date(2023, 02, 07)",
            "END": "VCALENDAR"
        }
    }
}

As for the date formats, I have encountered three thus far and format each one individually when the task is first pulled and added to my local dict. 20230207T140000, DUE;TZID=America/New_York:20230207T140000, DUE;VALUE=DATE:20230213. I format them all into generic python date before writing to my local dict date(2023, 02, 07). Having a way to get a preformatted date during the connection would be convenient. I deal with it in 20 lines worth of code, so not a big deal whatsoever.

fetch all tasks and filter through them

This is my current process. I've been thinking of ways to make the process more efficient (if it even matters). Thinking about it now, if I pull fresh data from the server, compare it with my local data, and if a change is detected between the fresh and local data per UID, a search at that point would make more sense. Like making a new list of tasks to be modified or created and using that smaller subset of data when connecting to the server.

one calendar server does not always work with another

I only use Nextcloud personally, and it only uses ics from what I can tell. My "main" one I use daily for school and life tasks, and a burner for development. The differences in date formats I referenced earlier are just from these two servers. My project is just meant for me personally unless someone asks me to broaden compatibility. I imagine the only way to cope with the numerous server compatibilities is to write a backend to deal with everyone and inform the user that feature 'x' isn't supported or enforce strict support for one format. For your plann project, that's obviously not a great approach, haha.

"The alternative to using CalDAV is to sync the clients by downloading the full ics feed for the whole calendar every time it's synced, or to keep a local copy of all the calendar events and send/receive calendaring data on event-basis over other protocols, like email attachments. Neither of the solutions are good."

This seems like what I'm looking at.

Did you have a look into sync_examples.py

I have, and it honestly looked too complicated at the time. Having gotten as far as I have with my project, I've gained some experience and will take another look. I wanted to get my project working quickly since I now use it daily, and this is the first usable application I've made.

As I said, I was using Thunderbird, but their task UI is not customizable, has a dated feeling, and is really cramped. I don't utilize the majority of features ics or CalDAV offers. I've basically made a color-coded checklist with due dates that syncs with my phone, but that's all I really want right now.

On a related note to that, I need to learn how to use worker threads for Qt. Every server-related operation freezes the entire GUI until it's finished.

I plan to read through the link for plann's task management page when I have time this weekend. I know basically nothing about CalDAV or best practices, and I definitely appreciate the resource.

Hmm ... can you explain it a bit better? What do you want to achieve?

I use JSON to store all of my appdata. Link to my code where I pull the dav data.

I think you're very right in the comment - you would benefit by accessing the data as obj.icalendar_component rather than obj.data.split('\n'). I would also store the objects received from the caldav calendar in the dict, rather than copy over the data.

By having all of the tasks keyed like this, I can easily reference any specific task by its key and without looping over the whole list. I suppose I could make the UID the key, but I can't assign one client side, so I'd have to push the new task and then pull it again to have its proper UID.

You can set the UID on the client side. Actually, it's meant to be set on the client side. If the UID is missing, the caldav library will set it before pushing it to the server.

From what I understand, dicts are a lot more efficient for my use case.

When I started programming, it really mattered, a simple program could take seconds to run, and if the program was written in some sub-optimal way one would really notice the difference. Nowadays it takes microseconds to search through a full list or a dict by for some specific value (of course, dependent on how long it is), so unless there are millions of rows, chances are that the difference between using a dict and searching through a full array will be neligible. Anyway, if I need to do different lookups, what I often do is that I use multiple dicts instead of only one.

people = [{"name": "alice", "town": "antwerp", ...}, {"name": "bob", "town": "bombay", ...}, ...]
people_by_name = {}
people_by_town = {}
for person in people:
    people_by_name[person[name]] = person
    people_by_town[person[town]] = town

The data itself does not get duplicated - modifying something under people_by_name, and the change will be visible under people_by_town.

Anyway, this is a bit outside the scope for this issue :-)

I assign another key "INCALENDAR" for obvious reasons. I still haven't figured out if it's acceptable to connect the server everytime I create/ edit a new todo or if I should do it in bulk like I am now.

The benefit of doing it in bulk would be if your client is mobile and may be offline. Except for that, I would ship changes to the server immediately. Though, dependent on your server, the communication with the server may take time.

As for the date formats, I have encountered three thus far and format each one individually when the task is first pulled and added to my local dict. 20230207T140000, DUE;TZID=America/New_York:20230207T140000, DUE;VALUE=DATE:20230213. I format them all into generic python date before writing to my local dict date(2023, 02, 07). Having a way to get a preformatted date during the connection would be convenient. I deal with it in 20 lines worth of code, so not a big deal whatsoever.

Timezones can be a source of problems ... unless one happens to live at Iceland or São Tomé (I'm spending most of this winter in Lisbon, that also works out). Why bother reinventing the wheel ... the icalendar library already converts it to a datetime object for you, and it's easily available as obj.icalendar_component.

This is my current process. I've been thinking of ways to make the process more efficient (if it even matters). Thinking about it now, if I pull fresh data from the server, compare it with my local data, and if a change is detected between the fresh and local data per UID, a search at that point would make more sense. Like making a new list of tasks to be modified or created and using that smaller subset of data when connecting to the server.

This may be quite time consuming if it's a huge calendar and if the latency to the calendar server is big. Some years ago I discovered that the caldav integration on my cellphone would efficiently zap the battery - I suppose it was downloading the full calendar regularly instead of using sync-tokens (or probably because sync-tokens weren't well-enoough supported on the server side - I never did any research on that).

I only use Nextcloud personally, and it only uses ics from what I can tell. My "main" one I use daily for school and life tasks, and a burner for development. The differences in date formats I referenced earlier are just from these two servers. My project is just meant for me personally unless someone asks me to broaden compatibility. I imagine the only way to cope with the numerous server compatibilities is to write a backend to deal with everyone

I'm having some ideas for that in the caldav library - discover (or be able to configure) compatibility issues, and then search will fetch the full calendar and filter client side if the server cannot be trusted to do the work properly.

Another thing, sooner or later I will write up some backend that can access other sources and not only CalDAV calendars. In particular, I'm interested in accessing gitlab and github issues through plann.

Documentation specific to tasks. The previous basic_usage_expamples included some and how to mark it as completed, but its been removed.

It has been moved into search_calendar_demo. Perhaps I should make a separate function for dealing with tasks. Anyway, I can find this and it covers the basics:

    ## Tasks can be completed
    tasks[0].complete()

    ## They will then disappear from the task list
    assert not calendar.todos()

    ## But they are not deleted
    assert len(calendar.todos(include_completed=True)) == 1

    ## Let's delete it completely
    tasks[0].delete()

I have added some more functionality in the caldav library that isn't covered by the examples yet:

complete takes the new argument handle_rrule is set to True it can split a task into "completed" and "next" if the task to be completed is a recurring task. I should probably touch that in the example code.
there is a new "uncomplete"-method for reverting a complete. It is probably worth mentioning in the example code.
New methods set_duration, get_due, set_due - I do not want to mention it in the examples yet, because this API is considered to be unstable (with warnings in the code that it may disappear or get changed in version 2.x).

I'll consider to create a separate issue on this, but I think it's not much of a priority - the .complete() is covered in the example code, and that's the most important "extra" there is with regards of tasks.

How to edit specific tasks. Right now, I copy the data to a new dict entry, delete the original server version, and create an entire new todo for my edits.

There is the read_modify_event_demo which demonstrates several ways of editing an event - though, perhaps it's not entirely clear how to get the event in the first place (that is covered in the search_calendar_demo), and perhaps it's not entirely clear that everything that can be done to an event also can be done to a task. Except for that, I think it covers everything ...

How to edit the event using the .vobject_instance property and the vobject library
It mentions the icalendar_instance property, but there is no example code - it's not very suitable for editing anyway.
How to edit the event using the icalendar library and the icalendar_component property.
How to edit the event by setting event.data.
Some warnings on thread safety
A warning that one shouldn't do calendar.save_event(event.data) (seems like an extra "not" had sneaked into the text, I will fix that)
Local changes vs changes pushed back to the calendar

So from my perspective, everything is covered, but maybe it lacks on clarity?

Please tell me more exactly what is missing or how it can be improved, and I will consider it :-)

The clarification in that commit is exactly the information I needed. You hit the nail on the head. I am modifying my code to reflect your explanations, and another small thing came up. It took me a minute to figure out how to set the uid client side (I didn't know I could do it in the first place, I am new, after all :D )

        uidNew = str(uuid.uuid1())
        calendar.add_todo(
            summary="Some Summary",
            due="Some Date",
            categories="Some Category",
            id=uidNew
        )

I thought id was going to be uid, but I traced the references to find a proper example in objects.py. When creating the todo with uid this error is thrown, if that matters:

  File "/usr/lib/python3.10/site-packages/caldav-0.11.0-py3.10.egg/caldav/objects.py", line 2001, in generate_url
    return self.parent.url.join(quote(self.id.replace("/", "%2F")) + ".ics")
AttributeError: 'list' object has no attribute 'replace

Don't send id into the add_todo method, it's uid that should be used. Still, that looks a bit like a bug to me.

The RFC recommends to use uids containing a timestamp, a serial number and a domain, like UID:19970901T130000Z-123405@example.com, but since arbitrary (mobile) clients may not always have a unique domain I think it's better to use the uuid method.

for the modifying an existing event section, a brief overview of converting "a named value from a primitive python type to an icalendar encoded string." would be most helpful. It took me a decent amount of googling to figure out how to convert my new data to a type that will be accepted when editing existing task properties.

For example, to edit the due date for a tag I searched, I need to convert my date to a datetime object, then use vDatetime to convert that string into the datetime ical type.

A quick snippet of what I figured out:

from icalendar.prop import vDatetime

    taskFetched = calendar.search(
        todo=True,
        uid=taskToModify["UID"],
        )

    task = taskFetched[0]

    nDS = datetime.strptime("2023-02-07 00:00:00", "%Y-%m-%d %H:%M:%S")   
    formattedDate = vDatetime(nDS).to_ical()
    task.icalendar_component["due"] = formattedDate

     task.save()

The specific resources I got this method from were from the icalendar.prop source and test_unit_prop.py in the icalendar github repo.

Thanks. I usually access it through the .dt property ... found it out by looking around in the debugger. Hm...

Thanks. I usually access it through the .dt property ... found it out by looking around in the debugger. Hm...

When reading properties, I do use the .dt, but when writing the ical object won't accept a standard datetime object.

reading from an event found by search()

          nDS = upstreamTask["DUE"].dt
          localTask["DUE"] = nDS.strftime("%Y-%m-%d %H:%M:%S")

          category = upstreamTask["CATEGORY"].to_ical().decode()
          localTask["CATEGORY"] = category

Writing to an event found by search()

          nDS = datetime.strptime(localTask["DUE"], "%Y-%m-%d %H:%M:%S")
          formattedDate = vDatetime(nDS).to_ical()
          task.icalendar_component["due"] = formattedDate

          newCatList = []
          newCatList.append(localTask["CATEGORIES"])
          newCat = vCategory(newCatList).to_ical()
          task.icalendar_component["CATEGORIES"] = newCat
          # vCategory expects a list

          task.save()

If you want to see my implementation, here's where I modify a task and here's where I read it.

Based on my experimentation, the same applies for most ical properties, and I used the classes found in the icalendar API reference

When reading properties, I do use the .dt, but when writing the ical object won't accept a standard datetime object.

WFM. I can edit timestamps by assigning a new datetime to ical['DUE'].dt, but there are some caveats ...

One cannot assign a value like that this if it isn't set. Then it's needed to use ical.add, i.e. ical.add('DUE', datetime.datetime.now().astimezone(datetime.timezone.utc)) (though, chances are that you want to set it to some point in the future rather than now())
Many of the DT-types can be either a date or a timestamp. If it was a date and you try to set it to a timestamp, bad things may happen.

Unfortunately I have rather a lot on my task list that is overdue, so I don't have time to do more research on this now. I should probably add some more into the example code ... though, arguably, examples on how the icalendar library works is a bit out of scope of the caldav library, would probably be better to link to some existing example code for that library or documentation.

One cannot assign a value like this if it isn't set. If it was a date and you try to set it to a timestamp, bad things may happen.

so far I haven't had any issue assigning the value that wasn't set when it was created except Validation error in iCalendar: The value type (DATE or DATE-TIME) must be identical for DUE and DTSTART

At least for Nextcloud, I can assign a due date after the task was created only if I also set the start date.

            task.icalendar_component["DTSTART"] = formattedDate
            task.icalendar_component["DUE"] = formattedDate

I'll experiment with creating tasks in various clients and modifying their values from my app.

examples on how the icalendar library works is a bit out of scope of the caldav library

That's fair. Either way, I agree with linking documentation. I am learning the ical library as I use your code, I am sure someone else down the line will be in the same boat.

I don't have time to do more research on this now

I'll fork and make a PR with more example code for you to review later rather than playing tag in an issue to save you some time. Thank you for your thourough replies, I appreciate the elaboration.

python-caldav / caldav

QA on the basic_usage_examples.py needed #256