Updates on Save

[fletchermoore]

I'm going to start working on automatically updating Anki on save. I can think of two ways to do this. Looking for feedback.

1. After you send a file, the ext writes some metadata to the bottom of the file similar to the way foam does it. Subsequent saves check for the metadata and then automatically retrigger the send.
2. Add some kind of meta.anki file to the root of the directory on send which tracts the changes to Anki and is checked after each save.

If using strategy 1, the user can then prevent further sends to anki by deleting the metadata. 1 is my current strategy. But 2 might be cleaner.

The metadata added would include the note Ids that were created by the send. Then subsequent sends will check to see if the correlates of those notes still exist in the markdown and trigger a delete if they have been removed from the markdown.

I want the delete to occur because let's say I'm working on a topic and decide I need to totally rewrite some markdown files. I don't want tons of duplicate and out of date cards to be left in my Anki.

Hopefully, I think, this effect would be intuitive to people.

[jasonwilliams]

Thanks for this! Is there a reason you’ve ruled out using the workspace storage as a database? I wanted to avoid writing data to each file, users may not like that. Obvious downsides are it getting deleted, but this only happens when the extension gets removed.

if we really have to write data to the file itself I’m guessing it’ll need to be in the front matter. Some ID field which generates a random string

[fletchermoore]

I would like to keep all the information needed for functionality between the individual markdown files and the anki db. My workflow is to edit my foam from multiple computers. So any state needs to be in version control at the minimum.

Anki itself is a pretty bad environment for plugin author metadata if you plan to sync. I've tried various strategies there that don't work well.

The number 2 possibility I mentioned was to make like, a json metadata file for the whole project. So on each save it checks the metadata file for prior state so it can make the diffs correctly. Problem is I don't know if that would be intiutive to users. Suppose they want to stop syncing a file. I figure some users might not like any of this state system written visibly in their directories or files.

On the other hand, I wanted a way to Send All with exlcude/include directories sort of like a typescript config. Or a .gitignore. So maybe a single metadata file for the whole project is best.

I can't decide.

[jasonwilliams]

I just remembered that Anki does give an ID to a note you create. So we can apply that ID to the file. https://github.com/FooSoft/anki-connect/blob/master/actions/notes.md#note-actions

Then it will need to be in the front matter (so it’s not visible to users who are editing md files).

I think we need to start simple and work towards something more feature-rich. There's no reason we can't offer both. I would

1. Start off trying to add the ID returned from Anki to the frontmatter (this is a task in itself).
2. Try implementing the update based on the ID parsed from the frontmatter and checked against the database
3. Once the above is done, try adding the IDs to a JSON file instead and working from there. You can toggle config on which way you want to do it.
4. The same functionality should then be applied to the explorer https://github.com/jasonwilliams/anki#explorer so metadata can be updated and saved (this is easier as they're not markdown)

You could swap 2 and 3 if doing the file is easier to start with.

Potential config i see for this would be:

{
  "anki.updateOnSave": ["file", "ankiDB", false]
}

Potential Issues:

We need to make sure the db file doesn't get too big. Deleting markdown files should also remove them from the db file.

[jasonwilliams]

@fletchermoore how does that sound to you?

[fletchermoore]

Started working on this last night on this branch. https://github.com/fletchermoore/anki/tree/sync_on_save

I went with trying to add [ ] # () style link/comments at the bottom since when I was looking into markdown front matter it seems that is not universal. At least, I think you are referring to a header with --- spacers.

Sort of works, but some issues still to resolve which I listed in my readme on that branch so I don't forget.

[fletchermoore]

Looks like this at the bottom of the file, added after Send if you have KeepSync option enabled.

[fletchermoore]

haha. the markdown editor in github removed it

[//]: # (Autogenerated Anki Metadata -- delete to disable sync)
[note-ids]: # (1611716069019, 1611716069051, 1611716093803)
[//]: # (End Autogenerated Anki Metadata)

[jasonwilliams]

I was looking into markdown front matter it seems that is not universal.

It's understood by a lot of markdown editors, including the mobile ones Foam uses. We should definitely be adding to that (if it exists) rather than creating our own section of metadata. There is a library for adding/parsing it here https://github.com/docs/frontmatter. Maybe some research is needed here? How easy is your metadata to modify once added?

Looks like this at the bottom of the file, added after Send if you have KeepSync option enabled.

Did you look at the config I proposed above? Maybe the sync strategy and updateOnSave need to be separate configurations?

Also, could you try adding it then linking to another markdown file on Foam then running "Janitor". Id be interested if it conflicts with Foam's metadata which gets added to the bottom

[fletchermoore]

Maybe some research is needed here? How easy is your metadata to modify once added?

I'm not married to this. it was just something so I could start working on the actual diffing functionality. I can replace it with the frontmatter parser.

Did you look at the config I proposed above? Maybe the sync strategy and updateOnSave need to be separate configurations?

I think separate is better. I can imagine some people are happy with a one-way push and then want to curate their deck within Anki by other means.

How about

updateOnSave, bool: "Automatically Send to Deck on Save. Enabled per file after initial Send." Not sure if that description is clear.

syncMetaLocation, toggle: "Inline Frontmatter", "Separate File", default?

[fletchermoore]

Other thoughts: possibly use front matter for sync exclusion/inclusion. "Sync: yes|no"

Could use workspace storage by default. My situation may not be broadly applicable.

The word Sync sort of implies two-way which might be confusing. I cannot think of a way to cleanly incorporate Anki cards into multi-file markdown if they are created and/or edited from within Anki. It would be marvelous to do though.

I think ensuring terminology and description is clear is very important. Otherwise people might be quite upset if something in markdown overwrites or deletes something in Anki and they are not expecting it.

[jasonwilliams]

Yeah "Sync" is quite broad and has little meaning in this context. I just remembered that we also have the "Send To Deck" command which users expect to update the card rather than overwrite it or not work. So we can implement this feature using that first also.

Then we can re-use the same mechanism on save.

The point I'm making here, is that the 2 (updating a card, updating on save) aren't mutually exclusive.

updateOnSave, bool: "Automatically Send to Deck on Save. Enabled per file after initial Send." Not sure if that description is clear.

Yeah this is nice, I'm not sure the "enabled per file" is necessary. It sounds like the sort of config you have project wide, even so we should reduce scope for this anyway.

syncMetaLocation, toggle: "Inline Frontmatter", "Separate File", default?

Yeah, the options are quite wordy, I think we can just have "file", or "inline" (then explain what each one does in description).

Could use workspace storage by default. My situation may not be broadly applicable.

I think you made a great point about using different instances of VSCode which would screw up the syncing. Having something that can be committed to Git sounds good to me as a start.

What we default to im not too fussed, we could start with inline. But then there's the question of frontmatter? or what you prototyped at the bottom? I don't know too much about your implementation so far so its hard for me to compare but i know the frontmatter parser can add/remove from a markdown file (or at least it looks like it). A plus for frontmatter is that apps like GitJournal hide any frontmatter that is in the document

Breaking Up the feature

• You can do this updating mechanism without having to worry about the "on save" feature. We have "send to deck" so you can just use that. This should make implementation much easier.
• "updateOnSave" is then as easy as just calling "sendToDeck" on save. If the above is already done this is easy to add

If we go down this path we may want to open up a new issue for "updating an already existing card"

[fletchermoore]

Issue I have run into:

Send To Deck send the active text editor to deck. When I then update the meta of that file, I initially wrote it to the file. This causes the active text editor and file to become inconsistent.

Since the active text editor could represent unsaved data, perhaps Send To Deck should save the file first? I am trying to imagine a scenario where the user wants to use Send To Deck on unsaved data in particular for some reason. Maybe the Send to Deck should prompt to save file? Or just do it without notice. What do you think?

[jasonwilliams]

Maybe the Send to Deck should prompt to save file?

I think this is the best option. But only if the sync feature is on. Sometimes I just make cards to send to Anki. I don’t care about saving them in markdown, that’s why it sends unsaved content. This is useful for quickly writing a card and sending it.

I’m guessing you can find out if the editor is in an “unsaved” state?

[jasonwilliams]

@fletchermoore i might take a look at this today maybe by using a json file to begin with

[fletchermoore]

I have an implementation going on my fork. Doesn't really match all we've discussed so I haven't sent a pull yet.

I have a sync on save going, metadata in files and send all option.

You can see it in the changes I made to subscriptions.ts

[jasonwilliams]

@fletchermoore sounds great did you want to PR your work in progress?

[jasonwilliams]

I've left a review on the PR. The core concept looks good, there's some bugs I've come across but I think its a good implementation so far. The main issue for me is there's too many features bundled into the same PR, and this makes it tricky to review.

I think some things (like sendAll) can be removed and added separately in an upcoming release.

[jasonwilliams]

@fletchermoore was you still looking into this?

[jasonwilliams]

Hey @duguosheng now that you've added send by dirname, what are your thoughts on having a sync feature? So that you can go back to a file and update the fields from a markdown file?

[duguosheng]

I tested the following steps:

1. create a md file for test (I use --- instead of % to separate front/back)

   
   ## this is a test file

front content

---

back content

2. send it to anki
<img width="518" alt="image" src="https://user-images.githubusercontent.com/52968130/175819903-d9b5c011-fbe5-4a8b-8d62-4eaa8d9afdc9.png">

3. use `findNotes` to get noteID
```json
{
    "action": "findNotes",
    "version": 6,
    "params": {
        "query": "this is a test file\n  front content"
    }
}

result:

{
    "result": [
        1656253980018
    ],
    "error": null
}

4. use updateNoteFields to update note

   {
   "action": "updateNoteFields",
   "version": 6,
   "params": {
       "note": {
           "id": 1656253980018,
           "fields": {
               "Front": "<h2>this is a test file</h2>new front content",
               "Back": "new back content"
           }
       }
   }
   }

   

[jasonwilliams]

@duguosheng where are you storing the note id though?

[duguosheng]

I did not store noteid in vscode workspace, but query it every time. However, I forgot front content could be change too. So I just query note title this time.

1. send a test file to anki
2. query noteid

{
    "action": "findNotes",
    "version": 6,
    "params": {
        "query": "\"re:^<h2 id=\"this-is-a-test-file\">this is a test file</h2>\""
    }
}

result:

{
    "result": [
        1656297153119
    ],
    "error": null
}

but when user have multi note with same title in different deck, there will be several id in result

3. update note

   {
   "action": "updateNoteFields",
   "version": 6,
   "params": {
       "note": {
           "id": 1656297153119,
           "fields": {
               "Front": "<h2 id=\"this-is-a-test-file\">this is a test file</h2>new front content",
               "Back": "new back content"
           }
       }
   }
   }

   

I wonder if you think this plan is feasible?

[jasonwilliams]

The problem is any bit of it could change then the query won’t work. The front content could change, the back content could change or the title. There’s no permanent state you can rely on for the query.

what happens if they change the title or front content and you use that to query? It wouldn’t give back any results. I think somewhere you will need to save a mapping of the noteId (which you get when you create a note, or we could grab them from queries) to the file path.

[duguosheng]

If they change the title, it would be send to anki as a new note (same as now). If they change the front/back content, update the note (now: note sending failed). This is my personal opinion, maybe not good~

[jasonwilliams]

If they change the title, it would be send to anki as a new note (same as now).

I wonder if this is expected behaviour, maybe it is. is someone creating a new note or editing a current one? Does it overwrtite the old one? or is the old one still there?

This is my personal opinion, maybe not good~

I think its worth experimenting with this idea and tweaking it as we go

[duguosheng]

I wonder if this is expected behaviour

I think very few people do this. But maybe very few people change the title too. Usually, changes occur in the front/back area. So I think it is acceptable for users to delete extra notes by themselves if they change the title.

[jasonwilliams]

If they change the title, it would be send to anki as a new note (same as now). If they change the front/back content, update the note (now: note sending failed).

Im curious about this, isn't changing the title and front content the same thing? There's no distinction when we send to the API, the title is just html we send as front content. So what exactly do you mean when you say 'title'?

[duguosheng]

sorry, my bad. I usually write notes like this.

## title (short)
front content (many lines)

---

back content (many lines)

so personally, I regard the first line of a card as its title.

but I don't know if more people like to write note like below.

## front content
back content

[jasonwilliams]

Ok So in both cases the title and front content get sent to the API as one string of html. There is no differential between them. As far as the API is concerned..

## title (short)
front content (many lines)

is a single input which goes to the "front" of the card.

If you change any part of the front content you wouldn't be able to query it again. It would be classed as a new card

[duguosheng]

What I think

## this is title
front content (many lines)

---

back content (many lines)

then, we can read the content of a card, and only convert the first line ## this is title to <h2 id="this-is-title">this is title</h2> (we can do this by ourselves without using marked), then use it to query. Also, we can read the deck name according to the user's saving strategy, for example, user use send to dirname deck command and this md file's path is /path/to/workspace/CS/Typescript, then the deck name is CS::Typescript. So the final query is:

"query": "deck:CS::Typescript \"re:^<h2 id=\"this-is-title\">this is title</h2>\""

About save noteid in local file

I don't want this function to change my md file. But if we save a mapping of the noteId in another file, it could be difficult to manage if there're several notes in just one file.

for example, this file is /path/to/workspace/mynote/note.md:

## note1
content1

## note2
content2

we save noteid as:

key=/path/to/workspace/mynote/note.md value=[1623178626, 1236721367]

but what happen if user write a new note between note1 and note2?

[jasonwilliams]

then, we can read the content of a card, and only convert the first line ## this is title to

this is title

(we can do this by ourselves without using marked), then use it to query. Also, we can read the deck name according to the user's saving strategy, for example, user use send to dirname deck command and this md file's path is /path/to/workspace/CS/Typescript, then the deck name is CS::Typescript. So the final query is:

This still doesn't fix the issue of the title being changed though, if the title is edited the query won't work and will return 0 results. I do like the idea of scoping down to deckname though but sadly that doesn't help in this case.

but what happen if user write a new note between note1 and note2?

I agree this is now a problem, before it wasn't because it was 1 file per note but now we can have multiple notes per file we can't use the filename or anything like that.

I think using some sort of tag is unavoidable in this case as there's no other permanent hook to use. People sometimes submit a note into Anki then realise they've made a spelling mistake in the title so often change it and expect it to update, what you proposed wouldn't work because the query wouldn't match the second time.

[duguosheng]

so I say:

If they change the title, it would be send to anki as a new note (same as now).

And the previous wrong notes need to be deleted by themselves in Anki. I think it's acceptable. Of course, maybe others don't think so.