search

johannesjo / super-productivity

Super Productivity is an advanced todo list app with integrated Timeboxing and time tracking capabilities. It also comes with integrations for Jira, Gitlab, GitHub and Open Project.
http://super-productivity.com
MIT License
11.75k stars 910 forks source link

Feature Request: WebDav sync (so I can use NextCloud) #159

Closed TobiasDev closed 4 years ago

TobiasDev commented 5 years ago

I see that the app supports Google Drive, I was wondering if there is a chance to add WebDav sync so I can use my NextCloud instance to sync?

johannesjo commented 5 years ago

Good suggestion. Thank you. This will probably happen at some point, if enough people need the feature.

TobiasDev commented 5 years ago

I was thinking about this actually. Would it be possible (until the sync comes in, if it comes in), to just make it possible to choose where to store all the info in the app? I assume at the moment all the tasks, projects etc., is stored in some type of config folder. If I could set this myself, I could place those things in my NextCloud folder, and then when I switch PC it is synced through that folder and I just need to change the app to look in a different location and it automatically is synced.

johannesjo commented 5 years ago

@TobiasDev that's good thinking.

When you start the app via the command line there is the path printed out to where emergency backups are saved. On my system that's ~/.config/superProductivity/backups. Just one folder above (~/.config/superProductivity) is all the data you need.

Maybe this is even better than an integrated solution, because you wouldn't have to put your authentication data inside Super Productivity.

This probably won't work too well, when you are running 2 instances simultaneously, but that's also not the case for Google Drive syncing and probably not a common use case.

theCalcaholic commented 5 years ago

Imo, it would make more sense to fully support CalDAV backends, since this protocol supports todo lists (it would still be WebDAV compatible, as CalDAV is an extension to WebDAV).

This would make it possible to sync superProductivity to many backends and clients which already use this open protocol - e. g. Google Calendar, Nextcloud Calendar/Tasks, OpenTasks and so on

One of the benefits would be having compatible clients for platforms which are not supported by superProductivity (e. G. Android and iOS).

johannesjo commented 5 years ago

@theCalcaholic interesting proposition. I will have to do some research. I had close to no contact with CalDAV.

theCalcaholic commented 5 years ago

@johannesjo I found a few resources concerning this topic:

http://sabre.io/dav/building-a-caldav-client/ https://github.com/mozilla-comm/ical.js/wiki https://developers.google.com/calendar/caldav/v2/guide

johannesjo commented 5 years ago

@theCalcaholic thanks for the info.

Not sure if there is a good way to make this work as not only the tasks need to be saved, but also the jira/git issue data, the configuration, the projects and several other things. While custom fields are possible, there isn't a good way to store arrays and objects. Unclear is also what would be the best way to store the time tracking data as there is DTSTART but not really a proper way to store durations. Also the approach that "the truth is always on the server" might be problematic.

I will think about this a bit more this weekend. Maybe there is another angle to tackle this.

sunjam commented 5 years ago

Perhaps @raimund-schluessler could advise. They are a maintainer of Nextcloud Tasks CalDAV app.

TobiasDev commented 4 years ago

Quick question, is there any chance of seeing something like this? :) Not just for the backups, but so I can keep it synced between computers. I absolutely love the app, it's just a tiny bit of a huzzle since I use it at the office and home for working without the sync. :) Not saying it is impossible to use! It would just be a lot easer. :D

johannesjo commented 4 years ago

@TobiasDev have you tried syncing the whole ~/.config/superProductivity/ folder (or whatever it is on your machine)? I would assume that this works fine as long you are not using multiple instances at the same time and as long you're taking care to start the app fresh when you know the data has changed on the other machine.

TobiasDev commented 4 years ago

@TobiasDev have you tried syncing the whole ~/.config/superProductivity/ folder (or whatever it is on your machine)? I would assume that this works fine as long you are not using multiple instances at the same time and as long you're taking care to start the app fresh when you know the data has changed on the other machine.

Hmm. I have not. Is it possible to change where that folder is stored? So I could "just" re-point the app on all machines I use it?

johannesjo commented 4 years ago

There is a custom startup parameter you can use for that:

superProductivity --user-data-dir=/usr/me/folder-to-use

You need to copy your existing user data manually to that folder though.

TobiasDev commented 4 years ago

Sorry! I haven't had time yet to test this. I'm hoping to test this beginning of January.

johannesjo commented 4 years ago

@TobiasDev No worries! I've been quite busy with other (private) things myself :)

johannesjo commented 4 years ago

Not sure if this is acceptable as is or if we need a dedicated integration. Some feedback would be welcome!

TobiasDev commented 4 years ago

From my side, I would love to rather have it as a setting. :) Since that is a lot easier to deal with. However, I will admit it does work. :) It would just be "nicer" to have it as a setting that I can use instead of adding a starting parameter. :)

johannesjo commented 4 years ago

@TobiasDev thanks for letting me know. This should be much easier to do now, so it will probably come at one point.

LecrisUT commented 4 years ago

There is a custom startup parameter you can use for that:

superProductivity --user-data-dir=/usr/me/folder-to-use

You need to copy your existing user data manually to that folder though.

Is it possible to extend this for the android app as well? I would like to keep the desktop and mobile version synced without using dropbox or google drive.

The CalDav integration would also be appreciated, and I suspect the whole task structure have to be redone to make them compatible. Some use-cases for this integration: Managing tasks from a different application/webserver and syncing the progress across; Having the due tasks appear on a calendar (even if the time-tracking info is not present there); Easier import-export of tasks since many task managers have the option of exporting to CalDav.

johannesjo commented 4 years ago

@LecrisUT Not really sure if this is possible. As far as I know android has a specific folder for every app. I don't think that it is possible to change that. May I ask, how your workflow would look like here?

I also looked into CalDav. Unfortunately, the format is not really able to properly represent things like subtasks and it doesn't seem like a good fit in general for the existing data structures.

LecrisUT commented 4 years ago

For the android version, if you add support for external storage, then other syncing applications can take over. For example I'm using nextcloud app which defaults to syncing the folder Android/media/com.nextcloud.client and I have other apps directly accessing those files like keepass. For the CalDav data, it is hooked via Davx5 and I can control tasks via org.tasks.

I'm not sure how proper support for subtasks you need from CalDav. As it is now, I see that you can have subtasks and be synced across platforms. I have that setup on org.tasks and the nextcloud one.

TobiasDev commented 4 years ago

I wouldn't use CalDav, you can't use attachments and such with that (actually it is lacking quite a few features that Super-Productivity supports). I think the best way would be to use the same type of syncing used for Dropbox and Google Drive but with a webdav folder (like how the note-app "Joplin" does it for Webdav). That way you can sync everything including attachments and all. So when you add something on Android it will sync and show up in the desktop client, or the other way around. :)

Now that there is an Android App, I would love to see this added even more so I can add tasks I remember when I'm taking a walk since that is the time I always come up with new tasks for my projects! :D

LecrisUT commented 4 years ago

Tasks.org and 2Do seem to support attachment, but I haven't tested the synchronization if other apps can read each other. The benefit of a CalDav, even if it is not fully integrated is that we can share tasks with others and sync the progress. But I agree a WebDav would be easier and great for keeping things synchronized, although why not both?

So just to clarify, the CalDav/WebDav Feature Request is a step-by-step implementation, first simply sync the whole ~/.config/superProductivity/ via WebDav, then at least syncing CalDav tasks progress, tags and hierarchy (the nextcloud implementation should be the most compatible), then any other implementations depending on the popularity: sync with calendar entries, attachements, etc.

TobiasDev commented 4 years ago

Tasks.org and 2Do seem to support attachmen

At least for Tasks (I use this to share Shopping and Todo lists with my girlfriend) it does not sync any attachments. I don't believe NextCloud Tasks actually even support attachments from looking at the open issues. :( So I would definitly go for syncing the whole thing through webdav (if possible). :)

MichaelTunnell commented 4 years ago

this is one of the few things that I feel is missing from Super Productivity. If it offered a way to sync without Dropbox or Google Drive then this would be basically perfect for me.

Have you considered a method of sync similar to how Joplin works? I like Joplin's method due to the webdav alone option where I dont need anything but a webserver to use it. It also supports Nextcloud and a bunch of others so maybe it would be work looking into how they implemented it.

btw, so far @johannesjo I love Super Productivity :+1:

johannesjo commented 4 years ago

@MichaelTunnell I've looked at Joplin, but İ think that the manual syncing is not ideal (I myself would excessively hit the sync button but forget to hit it when it makes most sense). Also Joplin just syncs the notes as several different and mostly small files, which work independently while super Productivity uses has bigger interdependent models. But I definitely will have another look.

MichaelTunnell commented 4 years ago

Joplin introduced auto-syncing earlier this year but I dont remember when. I like Joplin's because I can use WebDav on my own server and on a non-public facing section of my server. That is the only thing that I want about Joplin's. I do not need everything as a file and I don't think I even want that as that sounds simple but also adds complexity at the same time. If the method of sync for Super Productivity was the same as it is now but also allowed me to use WebDav as the mechanism where I can store the files on my personal servers then that would be amazing!

I am already a fan of the app and am already using it for most of my todos at this point but if it had the sync thing addressed without dealing with those companies then it would be pretty much perfect for me to implement completely.

I talked about the app on my podcast Destination Linux, episode 190. I sent an email about this but just wanted to say that while I am making a feature request, I already am a fan. This is a great app.

johannesjo commented 4 years ago

@MichaelTunnell very nice podcast. Didn't know that one. Thanks for sharing! EDIT: Just listened to it and wow, this is amazing feedback! I didn't expect that at all! Thank you so much! You really made my day 😊

I think demand is big enough for this to get the attention it deserves :) My current planning looks like this:

  1. Minor & Quick to do bugfixes/improvements
  2. Fix Google Drive Sync if possible (quite a couple of people seem to have an issue at the moment)
  3. Improve data stability by ditching IndexedDB in favor of a custom flat-file storage.
  4. Tackle WebDav

No promises made!!!

If someone is interested in tackling this themselves, please let me know!

johannesjo commented 4 years ago

I came very far today and the feature is ready for testing in the feat/webdav branch.

Unfortunately, a problem popped up, which I am not sure how to best deal with. It seems like that there is no way to make cors work without requiring users to adjust their servers. That's not really ideal. I tested with the webapppassword extension and it seems to work fine, but probably not everybody can use it on their server.

It's also possible to make the requests from outside the browser, but this would leave out the mobile and the web platform. I'd much prefer nextcloud themselves offering a way to securely register web- and desktop apps. Any suggestions are welcome! Upvoting the issue might also help. Thanks for your patience.

codeling commented 4 years ago

@johannesjo I started testing this, and from what I can tell so far, it works very well!

Short step-by-step guide what I did:

When I first added it on a second PC (I first checked the "Enable syncing", before adding the cloud details, and also superproductivity was still open on the first one); superproductivity on the second PC first asked me whether this was a completely new sync - I pressed cancel since it wasn't; then disabled and re-enabled syncing, and then it showed a conflict with remote (no local changes yet, but still understandable, since I had the other still running). I chose the remote, and then I had everything from first on second.

So far I have just one minor usability remark (or rather one comment to make the sync more fool-proof): My first intuition was to enter an existing folder under Sync File Path (on second reading, it says file path, so I just should have reaad more carefully); superproductivity first told me that it found no existing data and asked me whether this was the first sync, and when I answered OK, the sync failed; looking at the error messages I then figured it out, and added just a path with a non-existing file name. Now the synchronisation works like a charm! But maybe there could be check for this situation?

I will keep testing the next few days and will let you know if I come across any actual issues.

johannesjo commented 4 years ago

@codeling Thank you very much for testing. Both improvement suggestions make a lot of sense. I'll add them both and get back to you.

codeling commented 4 years ago

One more thing (more of a question): When the time tracking is running, in the developer console it looks like sync is triggered every second (at least there's a ... __trigger_sync__ task ... entry every second); even though it is configured to only sync every minute. I haven't checked network traffic, so maybe this entry in the console is something unrelated to the sync? Or is this a feature?

johannesjo commented 4 years ago

One more thing (more of a question): When the time tracking is running, in the developer console it looks like sync is triggered every second (at least there's a ... __trigger_sync__ task ... entry every second); even though it is configured to only sync every minute. I haven't checked network traffic, so maybe this entry in the console is something unrelated to the sync? Or is this a feature?

Sync is triggered on every change but executed only at the end of the specified time frame max (apart from some special cases like starting or closing the app). So if you make several changes during the specified time frame it will execute sync at the end. If you don't make any changes it won't execute (but it will check for remote changes when you come back to the app after a while). For more info, you can check this: https://rxjs.dev/api/operators/auditTime

codeling commented 4 years ago

One more minor thing I noticed: When the sync fails, the error message says "Dropbox sync failed" instead of "WebDAV sync failed"; and there could be more information on the actual error (in my case, wrong credentials).

Since I had some data on a third machine already, but started to set up sync and work on two other machines, I now have the situation where I would like to merge past worklog data from two sources - is this somehow possible? The "Conflicting Data" dialog only offers the choice to choose one or the other; but since in my case, the two data stores cover distinct time ranges without overlap, it might be easy to merge? But I have no idea how much effort this would be, and since it's only a few days it's not that important actually, mostly I'm just curious.

Apart from these really minor things, everything is working great, thanks a lot!

johannesjo commented 4 years ago

@codeling Thanks for the input. I will add some more info to the error.

Can you maybe open up a new issue about the merging problem? Maybe we can collect ideas on how to deal with that there. Currently, you would have to merge both manually via their exports, I can help you with that if you like. I previously had also evaluated some automatic sync model, but since this app operates without a server, the only option would be using automerge which would have a terrible performance and very high disk space requirements as well.

johannesjo commented 4 years ago

The feature should now be available for everyone to test in the snap edge channel.

@codeling I made some more updates and merged everything into master. I hope I was able to resolve the issues you mentioned in a good way. If you encounter any other problems, please let me know.

johannesjo commented 4 years ago

I am closing this. Please open up a new issue, if you encounter any problems with the new feature.

@codeling thanks again for testing!