Investigate whether Tasks plugin affects editing speed in large vaults - Answer: it does, and will be worked on soon.

[claremacrae]

In conversation with @schemar I learned that the way that Tasks tracks edits to Obsidian notes is to reload all files. And that has been fast enough.

There have been some anecdotal comments that maybe Obsidian with the Tasks plugin uses more CPU - and that editing can be laggy.

This issue is to investigate if performance or CPU is a problem, and if it is, then to resolve the issue.

Possible Solution

Refactor the code so that when a note is edited, only that note is re-read, and not the whole vault.

This will need to be sure detect edits made in external editors.

[claremacrae]

Example observation from @elight in Discord

Is it just me or does Tasks eat up a fair amount of CPU? I'm noticing more lag in my data entry in my vault, since enabling it. I'll check more tomorrow AM when I'm on my Mac as opposed to my iPad.

[claremacrae]

From me:

I think it may not be just you, I think it may be worse on iPad and iPhone than on Mac My efforts to get a solid repro have not succeeded, hence wanting to start logging their setups, what plugins they have enabled etc.

TODO Add a note about the size of my vaults - numbers of files and tasks

I must admit that I have not yet thoroughly tested it with only the Tasks plugin enabled, and I'm so dependent on CustomJS as well.

[claremacrae]

@schemar I hope I've represented correctly what you said. If you have a mo, any info about the relevant bits of code would really be appreciated!

[schemar]

Thank you for tracking this, @claremacrae! The details a bit different, though. When a note changes, the cache only refreshes the tasks from that note, not all notes.

Here, the cache subscribes to the vault changes on a per file level and calls indexFile for changes: https://github.com/schemar/obsidian-tasks/blob/09758e51411ca124c4725b00ada023220314eccd/src/Cache.ts#L115

Here, the cache clears all tasks from that file before re-processing the file: https://github.com/schemar/obsidian-tasks/blob/09758e51411ca124c4725b00ada023220314eccd/src/Cache.ts#L221

However, what is currently non-optimized are propagation of changes to the cache's subscribers. For example, each query view subscribes to the cache to be informed of changes to update the view when tasks change. The Cache::notifySubscribers() method always sends all tasks to all subscribers: https://github.com/schemar/obsidian-tasks/blob/09758e51411ca124c4725b00ada023220314eccd/src/Cache.ts#L93

Potential optimization: Only send the tasks of the changed note. For that, all subscribers would need a map of note->tasks to update only partially.

[claremacrae]

Oooh, the internal label is very nice. Thank you.

And thank you for all the info, @schemar!

[elight]

Confirmed this morning that I don't notice any lag on data entry/manipulation on my i7 Mac mini but significant data entry/manipulation lag on my M1 iPad Pro. I'll experiment further on my iPad to attempt to determine if it is in fact the Task plugin or something else. Early signs point toward my recent introduction of the Task plugin to my Vault.

[elight]

And thanks again for reporting, @claremacrae !

[YannickTemplay]

Hello, Tasks (1.6.0) make my quite large vault (21k notes) almost unusable with lags of 5 to 10 seconds in the middle of edits. If I disable tasks then the performance switch back to almost instantaneous. I am on an Apple M1 with 16MB of Ram...

I am not sure how many tasks I have but if counting "dones" then quite a lot.

Maybe you could add an option to the tasks queries are only updated once when a page loads or something ?

The tasks plugin is central to my Obsidian usage but right now I am down to disabling it and reenabling it every time I need to see my tasks :-(

Thanks for any help !

[claremacrae]

@YannickTemplay Sorry to hear that. What version of Tasks were you on previously, and what other plugins do you have enabled?

[claremacrae]

@YannickTemplay any extra info that you can supply would be really helpful. When you say M1 I presume you mean a laptop rather than iPad?

For example:

is it all edits, or edits in files with tasks?

Does it differ if you are Live Preview or Source mode?

Is it worse if your edits are further from the start of the file?

[YannickTemplay]

Yes, Mac M1

All notes (but I think its much worse on notes with tasks queries)

See-> https://www.loom.com/share/c92f3a4b6339451d9ca62b0241d21ff9

[aubreyz]

Yes, Mac M1

All notes (but I think its much worse on notes with tasks queries)

See-> https://www.loom.com/share/c92f3a4b6339451d9ca62b0241d21ff9

Are any of the queries on the page dataview queries on tasks?

I had the same problem with task queries but more prominently with dataview ones. One thing I did not understand was why it had nothing to do with the mode (edit, live preview, reading). So two observations:

a) Why would searches ever refresh (for either plugin) when in edit mode -- but I can confirm that this definitely seemed to be happening). Only one pane open.

b) It seemed to happen when the output of the query was large (i.e. queries that produced many hundreds of hits that had to be rendered), not anything do do with the complexity of the query or the quantity of underlying data (bizarrely even if the result was not actually displayed in HTML). Whatever the case, at least in my hands, the intermittent severe lags on editing are a function of the page contents and not the vault itself.

I cannot see why tasks (or dataview for that matter) should be trying to process anything at all ever except when exiting a page (either by going to another page, or going from edit mode to preview). Arguably it should even do no-processing in live preview. It should certainly not do anything at all at each and every autosave of a file while editing things.

I'm on Windows 10

[YannickTemplay]

The video I sent you has no queries at all. Just a plain new note. And no other notes were opened simultaneously.

Of course, the lags are much worse on pages with a query or multiple queries, but the lag is still there on pages without queries so I think there is indeed a problem with some "refresh" that takes place in the background unnecessarily.

Typical queries I use on my daily note page look like that...

tasks
not done
due on {{date:YYYY-MM-DD}}
group by filename

I often also use in projet notes queries like this...

dataviewjs
const name = dv.current().file.name;
dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

[YannickTemplay]

Could the problem come from the dataview Javascript queries?

I copied this query from Excalidraw Zsolt.

dataviewjs
const name = dv.current().file.name;
dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

In his example, I think he mentioned that he put a comment with a "random number" in every query. I did not understood why so I did not do that. But maybe it's related to dataview refresh or something?

But again those dataview JS queries are not on those empty new pages so I don't understand why it would slow things down and when I turn off TASKS is works again while I keep DATAVIEW active... so it's not that I think.

[aubreyz]

To add to my last message -- the page that caused the most serious problems in my hands -- with editing lags in pure edit mode: Was a page of examples I use which has a host of (dozens of) personal dataview and tasks examples I have collected. This file is almost un-editable (but can be easily previewed/rendered).

I know this sounds bizarre, but there is something fishy going on under the hood.

[claremacrae]

Ah right, knowing that you also have dataview enabled, it would be good to understand if there is any interaction between your dataview and tasks setups.

Is there any chance you could disable dataview and restart Obsidian, and explore the behaviour, please?

[YannickTemplay]

Yes, the lags seem to come when both TASKS and DATAVIEW are enabled together...

Just TASKS = OK

Just DATAVIEW = OK

TASKS + DATAVIEW = lags

[claremacrae]

This is all tremendously helpful!

As much as possible, it would be really good to get approximate measures of timings with just dataview and tasks enabled an nothing else.

(Someone mentioned Excalidraw too. The number of potential interactions between plugins goes up exponentially with the number of plugins enabled.)

[aubreyz]

OK I should qualify my previous comments. The previous file I had containing a large amount of example dataview and task searches, and which (6 weeks ago) produced massive lags in pure edit mode, no longer does that. So perhaps some update to core obsidian has fixed it for me. I also use excalidraw occasionally (not in these pages though).

The page does still have massive problems on editing in live preview mode (much less of a problem, and perhaps understandable, but probably unnecessary). It is not clear why every search on a page should be re-done every few characters of typing if text that is being typed is completely outside of any search.

[claremacrae]

Thanks @aubreyz

So perhaps some update to core obsidian has fixed it for me.

Perhaps., but I'm not sure ... Every time I thought I had a good reproduction of the problem, and restarted Obsidian to verify it, the problem would not occur. And then some time later (hours or so) it would suddenly be slow again.

As a developer, things failing intermittently makes it almost impossible to work reliably on the bug, because you never know whether any individual change you made fixed the problem, if you have to keep testing for multiple hours to even be sure you really fixed it.

I also use excalidraw occasionally (not in these pages though).

Do you mean you turn excalidraw off when you are not using it?

The thing is that some plugins may do work behind the scenes, even if they are not used on the particular page you are editing.... Hence my request for testing to be done with other plugins turned off...

The page does still have massive problems on editing in live preview mode (much less of a problem, and perhaps understandable, but probably unnecessary). It is not clear why every search on a page should be re-done every few characters of typing if text that is being typed is completely outside of any search.

See the comment https://github.com/schemar/obsidian-tasks/issues/697#issuecomment-1140455291 above, from the author of this plugin, for comments about that.

Once we get to a reliable reproduction of this issue on a development machine, we can look at optimising that code.

[aubreyz]

I also use excalidraw occasionally (not in these pages though).

Do you mean you turn excalidraw off when you are not using it?

No I don't switch it off. But no point trying that now, unless I can reproduce the issue again on editing (not on rendering)

[aubreyz]

I think a related issue is the autosave interval in the obsidian core. This is (as far as I know) fixed at about two seconds, and is not changeable.

So if something is triggered on a save during editing, this is massively magnified. It was fairly obvious before when I was having files causing these problems, that editing lag was cycling at about that frequency -- so if I just typed xxxxxxxxxxxxxxxxxxxxxx I got about two seconds of typing in, before having to wait about 10 seconds.

It also creates problems for file versioning (by say Dropbox) when there are just thousands of versions.

It would be quite nice if the Obsidian developers allowed this to be altered within some sensible limits (say up to 30 seconds or when the pane was exited or previewed). This would also help to debug these types of issues because the timing aspect would become more clear.

[elight]

My steps to replicate:

1. Open Obsidian on my M1 iPad Pro.
2. Ensure DataView and Tasks are enabled
3. Open my Daily Note (Contains several Tasks queries and a DataView query)
4. Begin typing into the note,
5. Switch off DV
6. Repeat Steps 1-4 
7. Switch on DV
8. Switch off Tasks
9. Repeat Steps 1-4
10. Switch on Tasks
11. Open a new note with no queries
12. Type into the new note.

Results

Steps	Observations
Steps 1-4	Input lag was noticed. After typing a few characters, input seemed to buffer, rendering approximately every 2 seconds.
Steps 5-6	Same lag was observed.
Steps 7-9	No lag was observed
Steps 10-12	No lag was observed.

This lag has a significant negative impact to usability on iPadOS. I haven't interacted with the iOS version enough lately to comment.

[elight]

I think a related issue is the autosave interval in the obsidian core. This is (as far as I know) fixed at about two seconds, and is not changeable. ...

The observed 2 seconds aligns with my observations on iPadOS.

[YannickTemplay]

I wonder if all of you who have the lag issue use dataview JS queries too ?

Could it be related ?

dataviewjs
const name = dv.current().file.name;
dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

[elight]

I do use dataviewJS but I don't on that particular note, just DV's query language.

On May 31, 2022, Vipercat @.***> wrote:

I wonder if all of you who have the lag issue use dataview JS queries too ?

Could it be related ?

dataviewjs const name = dv.current().file.name; dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

— Reply to this email directly, view it on GitHub https://github.com/schemar/obsidian-tasks/issues/697#issuecomment- 1141707348, or unsubscribe https://github.com/notifications/unsubscribe- auth/AAACPAC3S3AM5HLABHHCASTVMWU55ANCNFSM5XHYPSOQ. You are receiving this because you were mentioned.Message ID: @.***>

[elight]

That is, I'm not doing anything that lends itself to tying up an event loop unusually.

On May 31, 2022, Evan Light @.***> wrote:

I do use dataviewJS but I don't on that particular note, just DV's query language.

On May 31, 2022, Vipercat @.***> wrote:

I wonder if all of you who have the lag issue use dataview JS queries too ?

Could it be related ?

dataviewjs const name = dv.current().file.name; dv.taskList(dv.pages().file.tasks.where(t => !t.completed && t.text.includes(name)));

— Reply to this email directly, view it on GitHub https://github.com/schemar/obsidian-tasks/issues/697#issuecomment- 1141707348, or unsubscribe https://github.com/notifications/unsubscribe- auth/AAACPAC3S3AM5HLABHHCASTVMWU55ANCNFSM5XHYPSOQ. You are receiving this because you were mentioned.Message ID: @.***>

[YannickTemplay]

I just exchange messages with Zsolt, the author of Excalidraw about the issue he had with Dataview’js but it seems unrelated to our problem.

[YannickTemplay]

His problem was that if there is multiple exact same Dataview JS queries in a vault, sometimes it would not update properly.

I definitely have the same Dataview JS query multiple times in my vault and it I don’t have any update problems.

[YannickTemplay]

His solution was to add a random number as a comment in each JS queries to make them “different”.

[YannickTemplay]

To be 100% sure I just made the test and applied the "Zsolt fix" to all my Dataview JS queries by adding a different random number in each of them and this does not change anything related to TASK lags. Wrong track...

[beet]

Found my way here after noticing lag while typing, and having used the performance recording in the developer tools to identify periodic spikes from the Tasks plugin, which seems to correlate with comments above about it being auto-save/reload related.

I can confirm that for me, turning off the Dataview plugin resolves the issue, and I can live without it so it's a feasible workaround (but I can't live without Tasks!)

I'm using Obsidian sync, in case that could be related.

Anything I can do to help debug?

[claremacrae]

@beet thank you. Please could you pop in a note of the versions of Tasks, Plugins and Obsidian? Thanks!

[schemar]

Another potential for optimization came to mind.

Currently, whenever you edit a file, the cache will process it. It will first delete all existing tasks from the file, then it will process the file, and then it will send all tasks to all subscribers.

It will do exactly this for every update.

A potentially easy fix with huge gains would be to change the behavior:

1. Process the file
2. Compare the result to what's currently in the cache
3. Only if it differs, send all tasks to all subscribers

Can later be optimized to only send tasks of the processed file like described above.

[aubreyz]

Another potential for optimization came to mind.

Currently, whenever you edit a file, the cache will process it. It will first delete all existing tasks from the file, then it will process the file, and then it will send all tasks to all subscribers.

It will do exactly this for every update.

I'm not sure I am following that 100% as being related to the problem (albeit it would be a worthwhile optimization). If that were the problem, the lags would be in any file edit (even in files without any tasks at all, or in files with only tasks but without any queries.

At least when I experience the problem (2 second cycling delays) it is always in files with tasks queries alone* (not in files with tasks or in other files). It gets back to the thing I raised before that I don't really understand -- "Why would task queries EVER get processed when a file is in purely edit mode and is not being exited or going to another mode"

*Possibly with dataview queries at the same time

[beet]

@beet thank you. Please could you pop in a note of the versions of Tasks, Plugins and Obsidian? Thanks!

Sure thing:

• Tasks 1.6.0
• Dataview 0.4.26 (currently disabled)
• Obsidian 0.14.15

[claremacrae]

Thanks @beet.

• Tasks 1.6.0
• Dataview 0.4.26 (currently disabled)
• Obsidian 0.14.15

one last question for now… if you downgrade to the previous Takss (1.4.1 IIRC) and restart, s the problem still there?

If so.can you keep going backwards in Tasks versions to identify where it might have happened please?

(I realise you may reach a point in history where Tasks no longer runs with current Obsidian. But any info is helpful.)

[beet]

one last question for now… if you downgrade to the previous Takss (1.4.1 IIRC) and restart, s the problem still there?

If so.can you keep going backwards in Tasks versions to identify where it might have happened please?

(I realise you may reach a point in history where Tasks no longer runs with current Obsidian. But any info is helpful.)

Is there a process for downgrading to a specific version? I tried checking out earlier tags thinking I could copy main.js, manifest.json and styles.css into Obsidian's plugins folder, but it would seem that there's more to it than that.

[claremacrae]

Is there a process for downgrading to a specific version? I tried checking out earlier tags thinking I could copy main.js, manifest.json and styles.css into Obsidian's plugins folder, but it would seem that there's more to it than that.

Yes, the thing to do is to go to the plugin's Releases page, and each release contains the release's Assets, including a obsidian-tasks-.x.y.z.zip.

The .zip contains the 3 files you mentioned, which, as you say, would be saved in the right place inside the plugins folder.

For reliable testing, please always restart Obsidian (or at least, re-open the vault) every time you change the contents of the plugin. (I expect this applies to other plugins too)

[beet]

Is there a process for downgrading to a specific version? I tried checking out earlier tags thinking I could copy main.js, manifest.json and styles.css into Obsidian's plugins folder, but it would seem that there's more to it than that.

Yes, the thing to do is to go to the plugin's Releases page, and each release contains the release's Assets, including a obsidian-tasks-.x.y.z.zip.

The .zip contains the 3 files you mentioned, which, as you say, would be saved in the right place inside the plugins folder.

For reliable testing, please always restart Obsidian (or at least, re-open the vault) every time you change the contents of the plugin. (I expect this applies to other plugins too)

Haven't had a chance to try older versions of Tasks yet, but have noted that disabling the Dataview plugin doesn't actually completely alleviate the issue, just makes it a bit less pronounced. Even with Dataview turned off, I still notice periodic spikes in lagginess when typing, but they're much less noticeable.

[YannickTemplay]

I noticed the same behavior.

If I disable TASKS and keep DATAVIEW there is absolutely no lag.

But if I disable DATAVIEW and keep TASK active, there is LESS lag but it is still slow.

[beet]

After updating to 1.7.0 and restarting yesterday, the performance issues seemed OK, but actually progressively worsened throughout the day.

Reloading Obsidian resolves it, temporarily.

[JoinGitHub1]

I am using Daily notes to track my to-do's throughout thy day – this could be from 1 to 20 tasks every day. Everything not check as done is included in a "backlog" the next day where it collects all tasks in all daily notes except the current. I felt something going wrong a few days ago with about 180 tasks in the backlog over approximately 95 daily notes. Keeping tasks below 150 eliminate the annoying delay in edit mode but is still there if I look for it. The less not done tasks I have, the faster everything feels. It could take 2 seconds to close task 180. but only a millisecond to close task 1. I am on a newer i7 with 32gb ram.

Today I experienced the same thing as @beet. I figured I can make it significantly worse by opening earlier daily notes. If I open 10 daily notes without closing them in between (10x180 = 1800++ tasks?), It would be impossible to write as the cursor will stop for several seconds "every second"... – If I hit Close and reopen one daily note, it is back to "normal".

Disabling Dataview did not make a difference for me.

It feels like there is something updating every second, which also doesn't really feel necessary.

I hope this can help. Let me know if I could provide more info.

Current version: v0.14.15 Installer version: v0.14.6 Dataview: 0.5.34 (enabled) Tasks: 1.7.0

[claremacrae]

@JoinGitHub1 Thank you.

The just-released Tasks 1.8.0 updated a bunch of dependencies, and I would like to understand if that changed the behaviour.

Current version: v0.14.15 Installer version: v0.14.6 Dataview: 0.5.34 (enabled) Tasks: 1.7.0

Please could you repeat that experiment, changing only to Tasks 1.8.0 and report any differences? Many thanks.

[claremacrae]

@JoinGitHub1

I am using Daily notes to track my to-do's throughout thy day – this could be from 1 to 20 tasks every day. Everything not check as done is included in a "backlog" the next day where it collects all tasks in all daily notes except the current.

Please could you share, as text, the querie that you are using for this? Thank you.

I felt something going wrong a few days ago with about 180 tasks in the backlog over approximately 95 daily notes.

What changed a few days ago? Was it only that your backlog suddenly grew in size, or was there any other change on your system?

[JoinGitHub1]

Please could you repeat that experiment, changing only to Tasks 1.8.0 and report any differences? Many thanks.

No noticeable difference. Still huge lags when opening several of these daily notes which include many not done tasks.

I also tested updating to installer version v0.14.15 – same result.

Please could you share, as text, the querie that you are using for this? Thank you.

I include this in every daily note:

not done
path includes a-Daily notes
path does not include 2022-06-13
heading includes Daily IT Meeting
sort by path reverse
hide task count

(currently 10)

not done
path includes a-Daily notes
path does not include 2022-06-13
heading includes Koordineringsmøte
sort by path reverse
hide task count

(currently 8)

not done
path includes a-Daily notes
path does not include 2022-06-13
heading does not include Morning routine
heading does not include Recurring tasks
heading does not include Daily IT Meeting
heading does not include Weekly meeting
heading does not include Koordineringsmøte
heading does not include VM Meeting
sort by path reverse

(currently 168)

What changed a few days ago? Was it only that your backlog suddenly grew in size, or was there any other change on your system?

Can't say for sure. But I have a feeling that this issue got worse the last days due to growing backlog size. Adding more mock tasks for testing got worse than above.

tasks
not done

(Tasks: 743)

Doing this made the note unusable:

done

Tasks: 1370

I am curious what happens if you create a lot of mock data – are you able to reproduce it?

Additional test:

I disabled all community plugins except Tasks and the issue persists. With test data above, I even saw a brief "Tasks loading" on the query's while typing.

[JoinGitHub1]

I did a simple test with dataview vs tasks

Using dataview til list all tasks in vault was without any issues or performance delays.

Using tasks to do the same thing had severe performance delays and made Obsidian unusable.

About 1900 tasks in maybe 120 files in entries below:

'''dataview
TASK FROM ""
'''

'''tasks

'''

[claremacrae]

Thank you everyone. I think I've got enough information to be experimenting on this now, and behind the scenes I've been learning more about how Tasks handles various different ways of modifying tasks.

[flwwlf]

Hi everybody, I am following this issue because I love tasks-plugin for my daily work as a project manager but I also noticed the performance issues described here.

Here is a simple workaround that seems to have solved the issue for me temporarily, but can be reapplied (no long term testing yet though): I replaced the checkboxes of all done tasks („[x]“) with the tag # done to remove done tasks from the task plugins buffer (or whatever it is that clogs up performance), as I usually do never look on old finished tasks. This deleted 3100 finished tasks and now Obsidian seems to run smoothly again (with 730 open tasks in obsidian, which is ~3x more than my normal personal backlog size, so I think this will work for me if I clean up done tasks regularly as described above)

[bwydoogh]

I replaced the checkboxes of all done tasks („[x]“) with the tag # done to remove done tasks from the task plugins buffer (or whatever it is that clogs up performance), as I usually do never look on old finished tasks.

I did the same (and went from +4000 tasks to 255 tasks): the lag while typing in notes is gone. I am really wondering why this plug-in evaluates all tasks, every time again.

Before

Render tasks called for 4221 tasks, state: Warm

After -> editing speed back to normal.

Render tasks called for 255 tasks, state: Warm