Upgrading to 3.0 broke everything

[kensmith]

Upgrading from 2.6 to 3.0 broke everything in my taskwarrior setup. I'm self hosting taskd and syncing across multiple Linux machines and Android Foreground. Until upgrading, everything was great.

• task list no longer shows any tasks. When I look in $TASKDIR/pending.data, the tasks are all still there. Ditto for /var/lib/taskd/orgs//tx.data. Everything is still there
• task sync yields "No sync.* settings are configured." This is coming from https://github.com/GothenburgBitFactory/taskwarrior/blob/3e41fb604c209e355444a1f0e2f4e15c70d76226/src/commands/CmdSync.cpp#L91 indicating that I need to set up sync.server.origin etc.

I barely have the enthusiasm to unbreak my configuration by trying to figure out the new config to make sync and task listing work again. I'm astonished that this breaking change was released without backward compatibility. I'm likely going to abandon Taskwarrior but in case you want to try to repro my case, here is a redacted .taskrc.

data.location=~/.local/task
include no-color.theme
news.version=2.6.0
report.next.limit=0
report.long.limit=0
report.list.limit=0
report.ls.limit=0
report.minimal.limit=0
report.newest.limit=0
report.oldest.limit=0
report.active.limit=0
report.completed.limit=0
report.recurring.limit=0
report.waiting.limit=0
report.all.limit=0
report.blocked.limit=0
report.full.limit=0
report.next.limit=0
taskd.server=redacted:53589
taskd.credentials=<redacted>
taskd.certificate=redacted.cert.pem
taskd.key=redacted.key.pem
taskd.ca=redacted.cert.pem
taskd.trust=ignore hostname
report.list.sort=tags-,entry+,start-,due+,project+/,urgency-

[kensmith]

I see this is by design, https://github.com/GothenburgBitFactory/taskwarrior/releases/tag/v3.0.0 . Still astonishing. Feel free to close as wontfix but I think you will lose users over this kind of thing. Best of luck to your project.

[polsvoice]

Same issue. Fortunately, I didn't have a huge task list. Regarding this:

Note that this is a breaking upgrade: users must export their task database from 2.x and re-import it into 3.x.

How do you do this?

[polsvoice]

Okay, here's what I did. By looking in ~/.taskrc, I saw that my tasks were stored in ~/.task, so I cd'ed into that folder and did task import backlog.data, and that seems to have resurrected my task list. It now shows my tasks when I do task list. I was able to add a new task, delete a task, and complete a task.

However, whenever I do task list, it also says There are 123 local changes. Sync required. That number changes each time, and it jumped from 118 to 123 just by adding one task! I have no idea what is going on. It does seems to be working again, though.

[felixschurk]

https://taskwarrior.org/docs/upgrade-3/ Here is the documentation for upgrading to taskwarrior 3.0, basically exporting all tasks once and them reimporting them with the updated taskwarrior. It is also linked in the release notes.

https://github.com/GothenburgBitFactory/taskwarrior/blob/v3.0.0/doc/man/task-sync.5.in Corresponding documentation for the update of the sync server.

https://github.com/bgregos/foreground/issues/180 It might take a little bit of time, however I hope that foreground will also adapt to the latest storage backend, unfortunately I am not a Java developer and can not really help in that regard.

https://github.com/GothenburgBitFactory/taskwarrior/issues/3299 Addresses also the issue described here and is working on a solution.

[imran-iq]

Same issue. Fortunately, I didn't have a huge task list. Regarding this:

Note that this is a breaking upgrade: users must export their task database from 2.x and re-import it into 3.x.

How do you do this?

Depends on how you installed taskwarrior. I use arch so pacman keeps a copy of every package in /var/cache/pacman/pkg

So you can do doas pacman -U /var/cache/pacman/pkg/task-2.6.2-2-x86_64.pkg.tar.zst do the export, then install 3.0 and do the import.

Other package managers may do the same, I am not familiar with them. Worst case scenario build the old version from source

[guyjames]

jfc I've wasted so much time on trying to fix this, and now I find it's by design! :-1: It happened to coincide with another issue with file permissions so I thought that was the cause. This upgrade also totally breaks syncing with WingTask (so far the only non-self-hosted solution I've found for syncing TW - and presumably it would have broken self-hosted solutions too). My 'mistake' was to do a simple pacman upgrade. I don't know if there's a way that users can be warned before upgrading without going to the github of every single package in the upgrade (can be well into the hundreds), something along the lines of 'Warning: if you install this upgrade the app will stop working and you will lose access to your data!'. Going to try downgrading now.

[djmitche]

I'm sorry for your frustration. I just finished debugging a sync issue that resulted in an important task mysteriously disappearing while other tasks continued to sync between task instances. This is something we rely on, so the failures tend to occur at a time we need to be doing something else, making the failures all the more aggravating. In fact, this is what led me to start on what became 3.0 about 5 years ago.

I don't have any involvement in any of the downstream distributions that would package Taskwarrior, and I am quite surprised that Pacman blithely upgraded you over a major version bump. In my experience as a user of various distros, typically this is handled either by allowing major versions to be installed side-by-side (e.g., postgres 15 and postgres 16 can both be installed at the same time) or by shipping the major version upgrade in a distro upgrade (so maybe Ubuntu 24.04 has Taskwarrior 3, while the older Ubuntus keep shipping 2.6.2). In fact, homebrew specifically called out this bump although that hasn't been handled yet due to a build error.

[kensmith]

I spent many years managing engineers so let me first say that I totally get the desire to rewrite things from scratch, particularly in Rust. I also want to express my gratitude to the authors of Taskwarrior for their contributions to this project and in general. This is a big project and you all do this as volunteers. Much respect and love.

TW3 is not a major revision, it is a new product. It is a breaking change in storage format, sync backend, and many other things in between. One way this could have been more gracefully released is as a new project v1.0 (eg. taskchampion which seems to be a component of the new architecture) with clear guidance on the benefits of moving to the new project with scripts to manage converting existing configs to new configs including automating away some of the hassle of converting the server config which, tbqh is a lot of work with TW2.6. Rather than force all existing users to take a breaking change, invite them into the new project and then put the existing TW2.6 lineage into maintenance mode. This would also be an opportunity to explain the benefits of the new codebase. What do existing TW users get when they convert their task and server config to the new taskchampion (placeholder for lack of a better moniker) world? Taskwarrior is already a community project so it would be quite fair to leave 2.6 issues as wontfix unless community members provide PRs which address them, freeing the authors of the new platform to concentrate their enthusiasm on the new code tree.

This breakage is particularly onerous because good productivity tools should be so transparent and immediate that they disappear from a person's workflow. Taskwarrior was one such tool. I only started using TW a couple of months ago and one of the things I had enjoyed about it is that the commands are simple and run fast on any of my machines even the super old ones. It had successfully 'disappeared' for me up until 3.0. The fact that it reappeared so vividly and so demandingly of my time to go and take on a new project to spin up on the new workflow was quite unwelcome. I'm quite certain I could 1) maintain my own fork of 2.6 and then handle any issues that come up by patching the C++ myself, 2) port my tasks and server to the new workflow, etc., but I won't. I'm too busy and I differ too far from the philophy of this project given its willingness to impose this level of change on its users. I've put my task list into a vimwiki until I can find a replacement. If the project decides to revert this project to the 2.6 code fork and create a new separate project per my completely unsolicited advice above, I would be quite grateful and would go back to just working with TW for the next months or years until I found the time to take a look at the new project.

It will be a wildly fascinating social experiment to see how many people go through the work to convert all of their tasks into the new world order and persist using the new and improved Taskwarrior. As I said in an earlier comment, I think you'll lose users over this. It violates the principle of least surprise which I hope they still teach to young programmers. It was beaten into my head by my mentors, for whom I remain grateful.

[kensmith]

One other piece of data that I didn't include in my previous comment: like @imran-iq and maybe some of the others, I got tw3 via a pacman upgrade on Arch. One way this issue could potentially be addressed is to change the way tw is released on Arch specifically. For example, it could be split into taskwarrior2 and taskwarrior3 packages, allowing people to opt in to the 3.0 lineage when they want to rather than automatically. Python did this on arch for many years (maybe still) to allow people to run older programs and then slowly opt in to the new world, or even run both revisions in parallel for a while.

If you wanted to be really slick, you could even create a way for the old sync server to sync with the new sync server which could possibly reduce the friction along the migration path. Again, I acknowledge that y'all do this as volunteers so I'm not demanding anything there, just throwing out some ideas.

[ryneeverett]

I think this highlights the somewhat paradoxical wall between satisfied users and developers.

The happiest users will never give any feedback and are thus invisible to developers and unrepresented in decision-making. Anyone who becomes a developer of OSS productivity tools does so to fix their dissatisfaction with said tooling.

On the other hand, satisfied users are blind to the technical debt that their tooling rests on. If they were charged with addressing this debt they would likely be more open to change.

[ryneeverett]

To put it more concretely, the decision likely would have been different if someone with your position had been a participant. But your position might have been different if you had the perspective of a maintainer.

[kensmith]

I would have thought my response convinced you of my empathy for the plight of the maintainers. I get the desire to address big structural bugs with rewrites. I also get the desire to enter into the world of open source to solve a category of issue that isn't solved to one's satisfaction in any of the existing offerings.

Part of why I weighed in with my lengthy comment above is precisely because I absolutely understand how these things happen. I've seen it in many large and small ways in my roles as a leader in engineering teams. If I weren't so busy, I might even join this project to help do some of the things I propose. For what it's worth, I'm working on the Arch side to see if we can split the packages per my previous comment re: task2/task3. But thank you for your comment because I do think that we are speaking about issues of software development philosophy in general and not just this change specifically. You rightly point out some of the tradeoffs in this tension between dissatisfied end users and enthusiastic maintainers.

[ryneeverett]

I didn't mean to accuse you of lack of empathy. I'm more regretting that it seems inherently impossible for a more conservative position such as yours to be represented before breaking your setup.

[pglaser-engineer]

For what it's worth: most of my systems are running on Manjaro, and, being more actively managed with respect to upgrades, tw3 was not even available from any of its repos. So I had to get the tar file from the arch package repo and install manually. In my case, I was fortunate enough to have noticed the instructions for updating before I did so.

When I did a full update on the one machine that I have running Arch, I was really surprised, as expressed by others above, that it upgraded to tw3. I immediately thought about how many users could get a big surprise.

It makes sense to me that the Taskwarrior team can't really control how downstream packagers distribute updated packages.

That said, the one suggestion I might offer is that tw3 could detect when the sql lite file is absent from the data directory and issue an error message to the effect that the data need to be exported in tw2 and reimported under tw3. My guess is that many users are going to start running into this. While this approach does not address the entire problem, at the very least it would help users understand that they need to downgrade to tw2 and either stay there somehow or export and upgrade and import.

Thanks for listening.

[ryneeverett]

That said, the one suggestion I might offer is that tw3 could detect when the sql lite file is absent from the data directory and issue an error message to the effect that the data need to be exported in tw2 and reimported under tw3. My guess is that many users are going to start running into this. While this approach does not address the entire problem, at the very least it would help users understand that they need to downgrade to tw2 and either stay there somehow or export and upgrade and import.

See #3321.

[kensmith]

I didn't mean to accuse you of lack of empathy. I'm more regretting that it seems inherently impossible for a more conservative position such as yours to be represented before breaking your setup.

Thank you for your comment. I respectfully disagree that there is an inherent impossibility between breaking changes and least surprise. Like I said in my long comment above, I view the delta between 2.6 and 3.0 to be more than a major revision. There are many ways to take both the conservative approach and releasing the overhauled version. Two of which I've also outlined above, namely, consider this a new product or release parallel versions so people can opt in to the overhauled version on their own schedule rather than on the project's schedule.

[kensmith]

https://gitlab.archlinux.org/archlinux/packaging/packages/task/-/issues/1

[imran-iq]

Yeah this is more of a ball dropping on arch's part unfortunately. Usually they have a news update for breaking changes like: https://archlinux.org/news/budgie-desktop-1072-6-update-requires-manual-intervention/ but it looks like it was missed this time around. (Thank you to @kensmith for following through on that front).

Just as a more general note though. I do get the frustration around breaking changes, however lets all remember, as per the licence of this software:

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

I'm astonished that this breaking change was released without backward compatibility. I'm likely going to abandon Taskwarrior

Let's not build up entitlement on a free product that was built and shared on nothing but goodwill of the maintainers. It is a good way to burn out maintainers

https://mikemcquaid.com/entitlement-in-open-source/

[djmitche]

As I said earlier in this thread, I appreciate the frustration (and have experienced similar frustration with other TW breakage), and I'm sorry for that. I appreciate the empathetic comments!

To the point of this being a new project, there's some additional background on how the decision not to fork was made here. Availability of maintainers' time was a factor in that decision. No decision is perfect, and others may have made a different decision, especially in different circumstances. But please know that we did take some time to think about this one.

Two data points to the "rewrite it in rust" comment. The goal here was to build a reliable, secure implementation of the task DB, to replace the existing ad-hoc, buggy and insecure implementation. Rust was a good choice for this work, but was not the motivating factor. Second, there are no plans to rewrite the remainder of Taskwarrior in Rust.

If someone would like to maintain the 2.x series going forward, please get in touch. I don't, personally, have the capacity to do so.

On a procedural note, I'm going to convert this from an issue to a discussion as I think that better represents what's going on here.