Question about sync and duplicate detection

[LemurTech]

Hey @thesamim !

I know there have been lots of recent changes to sync, so perhaps my basis for this question is a little off, but ....

If I create a task in a note (my daily journal, for example), and I tag that task as #ticktick, then that task is synced up to TickTick. Does it then get synced back down to the 'project' list (essentially creating a duplicate within the vault)? Or is it supposed to be that TickTickSync respects and is aware of the originating note and will not sync it back to the 'project' list?

I ask because I've had issues with duplicates being created between a note and the master list (and then detected much later than I'd expect). This monkey business may just be residual. But I thought I'd try to get clear on what you expect to happen. Thank you!

[LemurTech]

One of the things I'm seeing is this:

I have the Tasks plugin set to stamp each task with the Create date, which should put that date (and '+' emoji) before other dates in the task, but outside the area that Tasks considers to be the Description field. That date is getting rearranged (by TickTickSync?) to just before the link to the TickTick task ID. That is, it's getting placed in an area that Tasks considers the Description field.

Here's an example. This is what Tasks expects:

• [x] Jesse needs some adjustment to one our our XDR groups for RunZero Explorers link #ticktick %%[ticktick_id:: 65f1d27a1ea99059c467a70c]%% ⏫ ➕ 2024-03-13 📅 2024-03-13

And then this happens:

• [x] Jesse needs some adjustment to one our our XDR groups for RunZero Explorers ➕ 2024-03-13 link #ticktick %%[ticktick_id:: 65f1d27a1ea99059c467a70c]%% ⏫ 📅 2024-03-13

I'm using this Tasks query to check for these 'bad' dates, where the description contains an emoji:

# These description instructions need to be all on one line:
(description includes 🔺) OR (description includes ⏫) OR (description includes 🔼) OR (description includes 🔽) OR (description includes ⏬) OR (description includes 🛫) OR (description includes ➕) OR (description includes ⏳) OR (description includes 📅) OR (description includes ✅) OR (description includes ❌) OR (description includes 🔁)

# Optionally, uncomment this line and exclude your templates location
path does not include Templates

group by path

[thesamim]

@LemurTech : I've created issue #92 for the created date question.

If I create a task in a note (my daily journal, for example), and I tag that task as #ticktick, then that task is synced up to TickTick. Does it then get synced back down to the 'project' list (essentially creating a duplicate within the vault)? Or is it supposed to be that TickTickSync respects and is aware of the originating note and will not sync it back to the 'project' list?

A file can have a default project. When a task is added to that file, it gets synced to that project/list. If a file does not have a default project, when a task is added to that file it get synced to the "default project". The "default project" is either specified in settings or "Inbox." If the task is updated in TickTick it should update to the same file that it was created in.

I ask because I've had issues with duplicates being created between a note and the master list (and then detected much later than I'd expect). This monkey business may just be residual. But I thought I'd try to get clear on what you expect to happen. Thank you!

So the above is the expected behavior. If you are seeing cases where that's not happening, it would be helpful to get a flow of events:

1. What file was the task originally created in?
2. Did that file have a default project?
3. What list did the task end up in in TickTick?
4. If you update the task in the file, does it update the task correctly in TickTick?
5. If you update the task in TickTick, does it get updated correctly to the right file?

If things go wrong:

1. Did it happen on the sync operation updating to/from Ticktick? (you will see a "File System Modified" log in that case)
2. Did it happen on a subsequent sync operation? (you will see a "File System not modified log, and a list of the files being checked.)

[LemurTech]

Here's what's regularly creating duplicates for me:

I only sync my TickTick 'Work' list. In TickTickSync, my default project is 'Work'.

1. In my daily 'work' note, I create a task, tag it with #ticktick, and set a due date (usually 'today').
2. The task is immediately synced to TickTick and appears in the correct 'Work' list.
3. On the next scheduled sync, the task gets copied back to the project list for 'Work' in Obsidian. It's my understanding that this should NOT happen. TickTickSync should be seeing that the task exists in the daily note.
4. I now have two copies of the task: one it my original daily note and one in the 'Work' project list.
5. On the next sync, TickTickSync complains of duplicates. I have a Javascript query that also identifies that there is a duplicate.
6. If I remove the task from the 'Work' project list in Obsidian, there are no more complaints.

I can replicate this every time.

[thesamim]

Just replicated the exact same scenario in my vault: Works with sync both ways. No duplicates.

I don't know if you're open to this, but I think the only way I'm going to be able to reproduce any of the issues you're encountering would be for you to create a vault where the issues (duplicate tasks, symlink/junction folders showing up as files) with the plugins that you have installed, and their settings, zipping it up, sending it to me, and giving me instructions re: the symlink/junction set up....

[LemurTech]

I understand the difficulty, what with all the ways other plugins might interfere. I may try my hand at enhancing the logging in the javascript, for better visibilty.

"symlink/junction folders showing up as files"

Just to be clear, that's not an issue. I do have folders being seen as files when they are created, but these are not limited to the one junction folder; that happens everywhere.

[thesamim]

I do have folders being seen as files when they are created, but these are not limited to the one junction folder; that happens everywhere.

Ah! Ya, then we're definitely dealing with another plugin affecting the outcomes....

Please keep me posted.

[LemurTech]

I've tried disabling every plugin I could that might be intercepting the identification of folders vs files, including:

Folder Notes Attachment Management DB Folder File Explorer++ Metadata Menu Templater Local Images Plus Linter

None of this had any effect. So I tinkered with the javascript a bit...

At line 20905 you have this bit of code that's supposed to check if we've got a folder, using TFolder:

    const file = this.app.vault.getAbstractFileByPath(filepath);
    if (file instanceof import_obsidian10.TFolder) {
      console.log("Not adding ", filepath, " to Metadata because it's a folder.");
      return null;
    }

This just doesn't work (for me, at least). It seems that TFolder is either not getting imported or else it's just unreliable. So I opted for a simple check for the .md file extension:

    const file = this.app.vault.getAbstractFileByPath(filepath);
    // Check if the filepath ends with '.md', indicating it's a Markdown file (and hence, not a folder)
    if (!filepath.endsWith('.md')) {
      console.log("Not adding ", filepath, " to Metadata because it's a folder.");
      return null;
    }

And this does work, reliably. Now the console will tell me when it's seeing a folder, and exclude it. So I'm not getting any more folders in my metadata!

Curiosly, however, the filepath is always showing up as "Untitled", even though this registrating is coming after I've changed the name to something else:

I added a bit of logging, just below the above section, to see when a new file was added:

    const metadatas = this.plugin.settings.fileMetadata;
    if (metadatas[filepath]) {
      return metadatas[filepath];
    } else {
      metadatas[filepath] = {};
      console.log("Adding ", filepath, " to Metadata.");
    }

These also show up as "Untitled.md":

That oddity aside, data.json appears to populate the metadata with the correct/final name of the file.

I am wondering, too, if this issue is connected with the appearance of duplicates. It's too early to tell...

[LemurTech]

I have not had any more folder metadate entries being created, since the previous file modification. I am still getting duplication, however, but this is only when I create a task in an Obsidian note, and tag it for #ticktick. The task syncs to TickTick, but will eventually find its way back to the project task list in Obsidian. It's still a mystery to me what the mechanism is. Duplicate parsing is somehow going awry for some tasks.

As far as I can see, there's nothing wrong in the formatting of the task:

- [X] Need a definition update to one of our dynamic groups, see JIRA:IS-80185 [link](https://ticktick.com/webapp/#p/65d7ae4b8f0863df68630fbd/tasks/6606e847ed56752634401c2b) #ticktick %%[ticktick_id:: 6606e847ed56752634401c2b]%% 🔼 📅 2024-03-29 ✅ 2024-03-29

And not all tasks created in a note end up being duplicated. In the case of the above task, there was another in the same file that was synced to TickTick, but it hasn't appeared as a duplicate.

I don't understand the javascript well enough. Best guess is that TickTickSync is interating through a task list it has retrieved from the web site, checking for duplicates in the file metadata. But it doesn't see it there, so it creates it. And then the 'maintenance' duplicate check kicks in and the duplicate is detected. What's different about these processes, such that the one would fail to read the file metadata?

[LemurTech]

When I create a task in one of my notes, and check it as done, the 'x' is a small 'x':

- [x] Server x experienced an outage last night or yesterday. Need root cause. [link](https://ticktick.com/webapp/#p/65d7ae4b8f0863df68630fbd/tasks/660ed4ada3fb3514849749c6) #ticktick %%[ticktick_id:: 660ed4ada3fb3514849749c6]%% ⏫ 📅 2024-04-04 ✅ 2024-04-04

When that task gets synced back from TickTick by TickTickSync, the 'X' is a capital:

- [X] Server x experienced an outage last night or yesterday. Need root cause. [link](https://ticktick.com/webapp/#p/65d7ae4b8f0863df68630fbd/tasks/660ed4ada3fb3514849749c6) #ticktick %%[ticktick_id:: 660ed4ada3fb3514849749c6]%% ⏫ 📅 2024-04-04 ✅ 2024-04-04

Technically it is not a duplicate. So it gets copied from TickTick. But it IS a duplicate, so the later check fails.

Tasks plugin seems to want this to be lower case:

But the TickTickSync plugin brings the 'done' tasks in as:

[thesamim]

Good catch! Will make changes.

[LemurTech]

It's funny--or not so funny... I noticed this quirk a very long time ago, but it never occurred to me that it could be the source of the sync issues. I sure hope this is the nail in the coffin!

Please retain the option for the verbose logging. It's been very helpful to see what's happening in the console.

[thesamim]

@LemurTech : I fixed the x vs X completion thing. Please let me know if that takes care of the duplication problems.

[LemurTech]

Thanks @thesamim, will do!

[LemurTech]

@thesamim, I think the manifest file is still showing the old version. My install wasn't upgrading, so I downloaded the files manually. But I noticed the version wasn't updating. Looked in the manifest and see 1.0.23.

{
    "id": "tickticksync",
    "name": "TickTickSync",
    "version": "1.0.23",
    "minAppVersion": "1.0.0",
    "description": "Sync TickTick tasks to Obsidian, and Obsidian tasks to TickTick",
    "author": "thesamim",
    "authorUrl": "https://github.com/thesamim",
    "isDesktopOnly": false
}

[thesamim]

@LemurTech : You are correct. Thanks! Fixed.