search

getting-things-gnome / gtg

Getting Things GNOME! trunk
https://wiki.gnome.org/Apps/GTG
GNU General Public License v3.0
555 stars 164 forks source link

Add synchronisation to CalDAV #407

Closed mildred closed 2 years ago

mildred commented 4 years ago

Hello,

I was a user of GTG long ago, and with the new version 0.4, I'd like to use it again. I'm willing to contribute too to the extent that I can afford (with the limited time that I have available). Before using GTG again, I'd like to get working synchronization available using open standards like CalDAV because last time, I just lost all my tasks somewhere in the homedir of an old computer.

I also would like the CalDAV synchronization to be compatible with https://tasks.org which supports many features and arbitrary nesting of tasks.

I'm asking here if there is any leftover of synchronization code left around (I'll look) and what's the best way to proceed.

Thank you.

nekohayo commented 4 years ago

Bonjour Mildred, We didn't delete the old plugins and synch backends, they are simply moved to subfolders to indicate their unmaintained status. Here they are:

If you would like to resurrect one of them (or make a derivative for a different sync service) then we will be happy to include your new contributed feature, as long as it works well (and that you will be able to stay around to maintain it, ideally :)

mildred commented 4 years ago

I take a simple look and I enabled back the synchronization dialog and registered a caldav backend, seems easy so far and it seems I'll just have to implement the talking to the backend part, so I am pleasantly surprised. There is just an issue around the icons where data/icons is not used at all and I worked around by using standard icons.

I'll need some more time to get the backend working and I should be able to maintain it if I know what needs to be done on it.

Thank you for the great work so far

diegogangl commented 4 years ago

Hi Mildred, thanks for working on this!

There is just an issue around the icons where data/icons is not used at all and I worked around by using standard icons.

I think most of the stuff in data/icons isn't even getting exported to the flatpak. We'll need to revisit this once we are using gresources. Using standard icons is the best way for now.

oyren commented 4 years ago

Maybe not the right place, but whatever. I'm planning a setup similar to @mildred, but I'm thinking about whether EteSync would be the better solution. Advantage:

Disadvantage:

diegogangl commented 4 years ago

@oyren We can have multiple backends, so no problem. If you want to discuss implementation details or something, then it's probably best to make a new issue for that backend.

oyren commented 4 years ago

This is already known to me, I just wanted to point out to @mildred that it might be more sensible to put his time into it.

mildred commented 4 years ago

As I understand it, EteSync data model is compatible with CalDav/CardCav but its protocol is different and you need a specific server for it. For me, it can be an additional backend you can choose in addition to CalDAV. it shouldn't be difficult to have both.

mildred commented 4 years ago

I have an issue with the backend, the password never gets saved. is it a security measure to avoid saving it in plaintext? Is there an integration with the gnome-keyring?

edit: yes I saw it and I'm also adding a binding with more modern libsecret

mildred commented 4 years ago

I got something working in #419. it probably needs more testing (in particular early versions managed to clear the CalDAV account of all TODOs. Use with backups for now.

mildred commented 4 years ago

About EteSync, I believe it shouldn't be much more complicated to create a backend, but as I don't have a server to test against, and as I don't have use for it yet (CalDAV over HTTPS self-hosted seems secure enough for me) I didn't investigate further.

All features supported by gtg should be saved to CalDAV. Start date is saved in custom field and hierarchy is saved and compatible with tasks.org. unfortunately start date configured in tasks.org is not saved to CalDAV and it is not possible to have compatibility between the two systems.

nekohayo commented 4 years ago

Fished out the fact that @mildred had made a pull request request for this (awesome!), so linked it to this ticket. Unfortunately I'm so clueless about setting up something (server + phone or something?) to test this, and also so behind because of things keeping me distracted lately, that I haven't found the opportunity to test this myself.

I will call some attention to this on social media, hoping that more people can test it out and report if it works well. Two months ago, as a disclaimer in your pull request, you said this:

This backend is still a bit experimental yet, I managed to wipe all my TODOs a few times, but hopefully it is no longer the case. Debugging issues should be possible with the debug log lines provided.

It's been a while though, and presumably you've been using it yourself daily since then, so... have you encountered other instances of data loss since then, or does the issue seem gone so far?

(to third parties reading this: please help test this, but remember it's always a good idea to make backups just to be sure!)

nekohayo commented 3 years ago

I'm kind of wondering how we could have some indication of how many people have tested this feature and used it successfully, so... I think what we can do is have people give a thumbs up/down on this comment of mine here to indicate if:

That'll at least tell us if it's been battle-tested by a significant amount of people so far.

theunclemez commented 3 years ago

What's needed from us ? More tests and testers of that new implementation? if yes, then please write an how-to that we can use as a base for starting then we can revert with what we find ... positive or not !

Ascl3pi0s commented 3 years ago

Hello,

It is the first time I ever do some testing directly from a pull request, so I'm not sure I proceeded correctly. If this is not the right way to proceed, please let me know.

Here is what I did :

Clone the repo, create a new branch from the pull request, and checkout that branch

git clone https://github.com/getting-things-gnome/gtg.git
cd gtg
git fetch origin pull/419/head:test
git checkout test

Then I lauched gtg using

./laucher.sh -d

It launched the gtg gui. When I clicked on Synchronisation I got error concerning missing python modules dateutil and caldav I manually installed those package using my distro ( manjaro) package manager.

Then I was able to retrieve the information from my nextcloud caldav, then I did a few changes, like adding subtasks, and modifying descriptions. But I wasn't able to sync back the changes onto the server.

Here is the debug log

2020-11-03 13:55:18,628 - DEBUG - application:__init__:86 - Debug output enabled.
2020-11-03 13:55:18,631 - DEBUG - __init__:__init__:60 - Backends found: ['backend_signals', 'backend_localfile', 'backend_caldav']
2020-11-03 13:55:18,631 - ERROR - __init__:__init__:74 - Malformated backend backend_signals: module 'GTG.backends.backend_signals' has no attribute 'Backend'
2020-11-03 13:55:18,796 - DEBUG - keyring:get_password:58 - get_password GTG stored password -backend_caldav@2ea5ce70-df1f-49bf-907f-03490e505113
2020-11-03 13:55:18,809 - DEBUG - keyring:get_password:58 - get_password GTG stored password -backend_caldav@9279c5b5-ffab-4da7-9869-bfde2404d542
2020-11-03 13:55:18,818 - DEBUG - task:add_child:552 - adding child 1@1 to task 0@1
2020-11-03 13:55:18,818 - DEBUG - task:add_child:552 - adding child 2@1 to task 0@1
2020-11-03 13:55:18,818 - DEBUG - task:add_child:552 - adding child 3@1 to task 0@1
2020-11-03 13:55:18,818 - DEBUG - task:add_child:552 - adding child 4@1 to task 0@1
2020-11-03 13:55:18,818 - DEBUG - task:add_child:552 - adding child 5@1 to task 0@1
2020-11-03 13:55:18,818 - DEBUG - task:add_child:552 - adding child 6@1 to task 0@1
2020-11-03 13:55:18,818 - DEBUG - task:add_child:552 - adding child 7@1 to task 0@1
2020-11-03 13:55:18,818 - DEBUG - task:add_child:552 - adding child 8@1 to task 0@1
2020-11-03 13:55:18,823 - DEBUG - task:add_child:552 - adding child 771c9080-e5a2-4d63-9f35-be14415c0d63 to task ba2ad2d9-5b1f-4538-94c2-85cca23d3419
2020-11-03 13:55:18,824 - DEBUG - keyring:set_password:47 - set_password GTG stored password -backend_caldav@2ea5ce70-df1f-49bf-907f-03490e505113
2020-11-03 13:55:18,838 - DEBUG - keyring:set_password:47 - set_password GTG stored password -backend_caldav@9279c5b5-ffab-4da7-9869-bfde2404d542
2020-11-03 13:55:19,072 - DEBUG - application:open_uri_list:203 - Received 2 Task URIs
2020-11-03 13:55:19,073 - DEBUG - application:do_activate:134 - Application activation finished
2020-11-03 13:55:22,783 - DEBUG - sync_engine:_analyze_element:231 - analyze_remote_id(rem=https://<nextcloud-caldav-location>/2a69b788-124a-41d4-a005-188f107b614c.ics, loc=None) -> add
2020-11-03 13:55:22,783 - DEBUG - backend_caldav:_process_todo:198 - GTG<-CalDAV set task url=https://<nextcloud-caldav-location>/2a69b788-124a-41d4-a005-188f107b614c.ics (add True) tid=None
2020-11-03 13:55:22,783 - DEBUG - backend_caldav:_has_local_task:490 - _has_local_task(796e7a2c-74d0-4bbd-b3c9-4b70c46e3c1b) -> True
2020-11-03 13:55:22,783 - DEBUG - sync_engine:_analyze_element:221 - analyze_remote_id(rem=https://<nextcloud-caldav-location>/46gYBEhiQw6aYCIwKAs7-Q%40ismtpd0050p1mdw1.sendgrid.net.ics, loc=796e7a2c-74d0-4bbd-b3c9-4b70c46e3c1b, present) -> update
2020-11-03 13:55:22,783 - DEBUG - backend_caldav:_process_todo:198 - GTG<-CalDAV set task url=https://<nextcloud-caldav-location>/46gYBEhiQw6aYCIwKAs7-Q%40ismtpd0050p1mdw1.sendgrid.net.ics (update True) tid=796e7a2c-74d0-4bbd-b3c9-4b70c46e3c1b
2020-11-03 13:55:22,783 - DEBUG - backend_caldav:_has_local_task:490 - _has_local_task(ba2ad2d9-5b1f-4538-94c2-85cca23d3419) -> True
2020-11-03 13:55:22,783 - DEBUG - sync_engine:_analyze_element:221 - analyze_remote_id(rem=https://<nextcloud-caldav-location>/B5B5A3CB-B04D-4B60-ACEF-815FDA71017A.ics, loc=ba2ad2d9-5b1f-4538-94c2-85cca23d3419, present) -> update
2020-11-03 13:55:22,783 - DEBUG - backend_caldav:_process_todo:198 - GTG<-CalDAV set task url=https://<nextcloud-caldav-location>/B5B5A3CB-B04D-4B60-ACEF-815FDA71017A.ics (update True) tid=ba2ad2d9-5b1f-4538-94c2-85cca23d3419
2020-11-03 13:55:22,784 - DEBUG - sync_engine:_analyze_element:231 - analyze_remote_id(rem=https://<nextcloud-caldav-location>/2607751124110185661.ics, loc=None) -> add
2020-11-03 13:55:22,784 - DEBUG - backend_caldav:_process_todo:198 - GTG<-CalDAV set task url=https://<nextcloud-caldav-location>/2607751124110185661.ics (add True) tid=None
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/core/datastore.py", line 468, in __backend_startup
    backend.start_get_tasks()
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/core/datastore.py", line 653, in start_get_tasks
    self.backend.start_get_tasks()
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/core/interruptible.py", line 38, in new
    return fn(*args)
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/periodic_import_backend.py", line 79, in start_get_tasks
    self._start_get_tasks()
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/periodic_import_backend.py", line 98, in _start_get_tasks
    self.do_periodic_import()
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 188, in do_periodic_import
    self._process_todo(Todo(todo))
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 211, in _process_todo
    self._populate_task(task, todo)
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 278, in _populate_task
    todo_parents = set(todo.get_parents(from_id=lambda parent_url, uid: self._id_to_tid(parent_url, uid)))
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 790, in get_parents
    return [from_id(self.parent_url, rel.value) for rel in self.vtodo.contents['related-to']]
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 790, in <listcomp>
    return [from_id(self.parent_url, rel.value) for rel in self.vtodo.contents['related-to']]
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 278, in <lambda>
    todo_parents = set(todo.get_parents(from_id=lambda parent_url, uid: self._id_to_tid(parent_url, uid)))
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 623, in _id_to_tid
    return self.sync_engine_url.get_local_id(url)
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/core/twokeydict.py", line 207, in _get_primary_key
    return self._key_to_key_bidict._get_by_second(secondary)
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/core/twokeydict.py", line 69, in _get_by_second
    return self._second_to_first[key]
KeyError: 'https://<nextcloud-caldav-location>/357920205205751692.ics'
2020-11-03 13:55:32,786 - DEBUG - sync_engine:_analyze_element:224 - analyze_local_id(loc=ba2ad2d9-5b1f-4538-94c2-85cca23d3419, rem=https://<nextcloud-caldav-location>/B5B5A3CB-B04D-4B60-ACEF-815FDA71017A.ics, present, sync=False) -> lost syncability
2020-11-03 13:55:32,786 - DEBUG - backend_caldav:set_task:356 - GTG->CalDAV set task tid=ba2ad2d9-5b1f-4538-94c2-85cca23d3419 (lost syncability, False) url=https://<nextcloud-caldav-location>/B5B5A3CB-B04D-4B60-ACEF-815FDA71017A.ics
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 1254, in run
    self.function(*self.args, **self.kwargs)
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/generic_backend.py", line 657, in launch_setting_thread
    self.set_task(task)
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/core/interruptible.py", line 38, in new
    return fn(*args)
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 396, in set_task
    self._exec_lost_syncability(tid, todo)
  File "/home/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 458, in _exec_lost_syncability
    meme = self.sync_engine.get_meme_from_local_id(tid)
AttributeError: 'Backend' object has no attribute 'sync_engine'

Now I'm stuck with this, haven't find the time to investigate further but I'm letting you know of this issue.

jaesivsm commented 3 years ago

I tried the patch locally against my Radicale instance. It works wonder ! Congrats mildred ! The translations from caldav collection to GTG tags is a pretty great idea and pretty well done.

I'm gonna use it daily in the coming days and report any bug here.

jaesivsm commented 3 years ago

Got an exception with the code of the backend : 

Exception in thread Thread-189:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jaes/devel/gtg/.local_build/install/lib/python3.8/site-packages/GTG/core/datastore.py", line 837, in quit
    self.backend.quit(disable)
  File "/home/jaes/devel/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/periodic_import_backend.py", line 105, in quit
    super(PeriodicImportBackend, self).quit(disable)
  File "/home/jaes/devel/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/generic_backend.py", line 153, in quit
    threading.Thread(target=self.sync).run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jaes/devel/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/generic_backend.py", line 708, in sync
    self.launch_setting_thread(bypass_quit_request=True)
  File "/home/jaes/devel/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/generic_backend.py", line 657, in launch_setting_thread
    self.set_task(task)
  File "/home/jaes/devel/gtg/.local_build/install/lib/python3.8/site-packages/GTG/core/interruptible.py", line 38, in new
    return fn(*args)
  File "/home/jaes/devel/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 371, in set_task
    self._add_to_cache(todo)
  File "/home/jaes/devel/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_caldav.py", line 315, in _add_to_cache
    self._cached_todos[todo.parent_url]['urls'][todo.url] = todo.uid
TypeError: unhashable type: 'URL'
tasn commented 3 years ago

Lead dev of EteSync here.

Maybe not the right place, but whatever. I'm planning a setup similar to @mildred, but I'm thinking about whether EteSync would be the better solution. Advantage:

* End-to-end encrypted

* [Python API](https://github.com/etesync/pyetesync)

* [Works with Tasks.org](https://tasks.org/docs/etesync_intro.html)

* Desktop Bridge for CalDAV

* [Backend for Akonadi (KDE) and Evolution is planned](https://blog.etesync.com/gsoc-update-fetching-etesync-contacts-in-gnome-and-kde/).

Already done! Included in the next version of KDE, and there's also an EDS module as part of GNOME's gitlab - https://gitlab.gnome.org/GNOME/evolution-etesync There's also a Thunderbird add-on in the work by the maintainer of TbSync!

It's much simpler than caldav. Both setting up a working server, and the implementation itself. It's also end-to-end encrypted

Btw, we just released a new version of the API (first time, and last time for the foreseeable future): https://github.com/etesync/etebase-py Docs: https://docs.etebase.com

jaesivsm commented 3 years ago 
diff --git a/GTG/backends/backend_caldav.py b/GTG/backends/backend_caldav.py
index fcf42a62..52ed59d9 100644
--- a/GTG/backends/backend_caldav.py
+++ b/GTG/backends/backend_caldav.py
@@ -312,8 +312,9 @@ class Backend(PeriodicImportBackend):
     def _add_to_cache(self, todo):
         if todo.parent_url not in self._cached_todos:
             self._cached_todos[todo.parent_url] = {'urls': {}, 'uids': {}}
-        self._cached_todos[todo.parent_url]['urls'][todo.url] = todo.uid
-        self._cached_todos[todo.parent_url]['uids'][todo.uid] = todo.url
+        todo_url = str(todo.url)
+        self._cached_todos[todo.parent_url]['urls'][todo_url] = todo.uid
+        self._cached_todos[todo.parent_url]['uids'][todo.uid] = todo_url

 ###############################################################################
 # Process tasks ###############################################################

will do the trick for the unhashable URL bug

jaesivsm commented 3 years ago

Hello around here, I just finished reworking the CalDAV implementation in #525, dropping the old implementation and rewriting one from scratch. I'm now using it daily but I'd welcome so testing and bug reporting :)

ezickler commented 3 years ago

Thanks a lot for (re)adding this feature. I would really like to use this and help testing. Most important for me is the feature of nested tasks supported by GTG and syncing to my trusted independent mail provider. That so far seams to work.

What i'm struggling with is finding a client for ios which supports caldav and nested task, because the use of syncing without my mobile device is is not that great for me. Does any one have a good solution for this that is maybe even know to work with this or i could test? The other thing i noticed is that my calendar server distinct between calendars to sync via url https://:8443/calendars//<random_cal_id/ but the name could be different from the id in the url. When i enable caldav syncing i have to add a tag with the calendar name to have it synced. It would like to have synced every thing by default. Is that expected behavior or am i doing something wrong?

jaesivsm commented 3 years ago

Thanks a lot for (re)adding this feature.

Thanks for testing !

When i enable caldav syncing i have to add a tag with the calendar name to have it synced. It would like to have synced every thing by default. Is that expected behaviour or am i doing something wrong?

If you're using implementation from #525 (the original implementation was actually using the caldav library the same way), the logic goes like this, when you activate sync (or every 15mn by default) the caldav lib does a listing of the available collections on you server for you. You can reproduce this like that :

>>> import caldav
>>> client = caldav.DAVClient('<your caldav server url>', username='<username>', password='<password>')
>>> principal = client.principal()
>>> principal.calendars()
[Calendar(calendar 1), Calendar(<calendar 2>)]

If that doesn't work with your settings, or you don't see the collections you'd want to see, then there is probably a problem with your CalDAV server.

All the collections are then added as tags prefixed with DAV_ ; all the caldav todo's categories are added as regular tags. Syncing might take a while but they should be added by themselves, try giving it time. If it still doesn't work I'd gladly take a look at some of the output of gtg launched with ./launch.sh -d that you could capture.

Also, please note that I based my code on a running Radicale server. I think you're using sabredav but, while there might be some differences, the caldav python library should work equally well on both.

phdd commented 3 years ago

Thanks a lot for this great contribution! What's the rationale behind the prefix? Can't we just sync everything?

jaesivsm commented 3 years ago

TL;DR: The DAV_ prefix is just a was to visually separate a tag/category from a tag/collection.

Thanks a lot for this great contribution! What's the rationale behind the prefix? Can't we just sync everything?

That's a rationale I would have liked to explain in the sync setting pop up, but I didn't find the time or a way yet to edit that.

Ok so, the DAV_ prefix is an accommodation I made knowing that :

So the logic is to make a special tag which represent your collection. You can chose the collections in which you sync your Todo by adding it as a tag. (The old plug actually threw an error if no collection/tag was selected, but in the new implementation I simply choose not to synchronize such object and leave it to the local storage).

The plug does not support collection creation or deletion (yet).

I'm not bound to that logic, we could go other ways but I personally enjoy being able to synchronize on various collections (some shared) just by switching tags.

phdd commented 3 years ago

Ok, now I see the need. Thanks for your explanation. This may be a configurable Property.

I'd like to contribute after this PR has been merged.

elsiehupp commented 3 years ago

If you still need a CalDAV server to test against, I have a private Nextcloud instance, and I could set up a test account there. I don't know how to make Nextcloud userdata private from the administrator (i.e. me), and I can't guarantee anything approaching reliability, but it wouldn't cost me anything to set up an extra account. (I also have a mail-in-a-box server, which is significantly more reliable because it doesn't strain its VPS's resources.)

elsiehupp commented 3 years ago

Actually, I went ahead and created an extra account on my mail-in-a-box server. You can email me at gtg [dot] issue [dot] 407 [at] elsiehupp [dot] com if you want the login information for it. (You would be able to reset the password.)

BeatLink commented 3 years ago

Any updates on this?

jaesivsm commented 3 years ago

@BeatLink Several things :

TLDR : it works but you'll have to check it out manually, as of now I'm not aware of a major blocker or date of integration

nekohayo commented 2 years ago

Hey, in case folks here haven't noticed, this was merged to master last month 🎉 so give it a try. It seems the essentials are there and documented in the user manual, and it will be included in the upcoming 0.6 release soon.

Is there anything else that remains for this ticket here, or should we consider it "Done", close this ticket, and defer to other caldav-related tickets for potential future enhancements that interested people can work on?

diegogangl commented 2 years ago

I totally forgot this issue existed, closing in favor of specific caldav tickets