Consider changing note format to something more version control-friendly?

[langfield]

Apologies for another issue imploring you to modify your repository structure.

I was wondering if you'd consider changing the source of your notes over to markdown to facilitate more atomic and trackable edits? Here's an idea of what it could look like:

## Note
nid: 1638097554265
model: Basic-02d89
tags: 04_optionen, checklater
markdown: false

### Front
Wie lautet die <b>Put-Call-Ungleichung</b> für <b>amerikanische
Optionen</b>?

### Back
\(C^{a}-S(0)+D+X \geq P^{a} \geq C^{a}-S(0)+X(1+r)^{-T}\)

I've built a tool for this (that's how I generated the above file) and I'm trying to recruit deck maintainers to try it! Let me know if you'd be interested.

If you still desire to distribute the binaries on GitHub, it would be quite easy to set up a CI/GitHub actions workflow to automatically rebuild the binaries on each commit. Then they could live on GitHub as releases. I believe this is the way many open source projects do distribution of precompiled binaries.

[KarelZe]

@langfield Thanks for pointing me towards your repo. I prefer the concept over the crowd anki approach, as people can contribute without the crowd anki extension. Also I feel like it's easier to follow changes in the markdown files than in json or even binary files.

I'll do some tests tomorrow. 👍

Do you have some GitHub actions workflow to convert the KI markdown files into a binary? So far I didn't spot anything in the docs. Would you mind making a PR?

[langfield]

@KarelZe Fantastic! Definitely let me know if you run into any trouble, or see any areas for improvements!

I can absolutely start working on a PR. We'd use something like this action to build the .apkg files. I apologize for not having anything up on the docs website about this. I've made an issue for it (https://github.com/langfield/ki/issues/52), haha!

It would look something like this, at a high-level:

1. An Ubuntu 22.04 GitHub actions runner installs anki, ki, and all their dependencies.
2. An empty collection.anki2 file is written to disk within a workflow step.
3. This collection file is converted into a ki repository via a ki clone command.
4. The repository (which is to say, KarelZe/anki-decks) is checked-out at the latest commit and copied into a subdirectory within the aforementioned ki repository.
5. These changes are added and committed in the usual way with git.
6. That commit is written to the collection.anki2 file via a ki push command.
7. A standalone python script opens the collection file using anki API, instantiates the AnkiPackageExporter class, and calls AnkiPackageExporter.exportInto(), dumping a .apkg to disk.
8. The .apkg file is published as a release via github-action-publish-binaries.

It may look like a lot, but (1) through (6) are all things that are already done in my existing CI pipeline anyway, so this stuff is easy, and the script would be about a 3-line program.

[KarelZe]

@langfield I made some tests tonight and created a ki branch. Everything went like a charm. 💯 I like the fact that it handles subdecks, nested tags etc. well.

I still have to adjust the readme, so that my fellow students know how to contribute to the repo.

I like your suggestions regarding the github action and noticed that you started working on it already. Would you mind commiting it, once finished?

[langfield]

@langfield I made some tests tonight and created a ki branch (still syncing). Everything went like a charm. 100 I like the fact that it handles subdecks, nested tags etc. well.

I still have to adjust the readme, so that my fellow students know how to contribute to the repo.

I like your suggestions regarding the github action and noticed that you started working on it already. Would you mind commiting it, once finished?

That's great news! I will definitely make a PR once it's working properly!

[langfield]

Update. Steps (1) through (6) are implemented here.

Unfortunately I'm running into a bug I've never seen in the wild before, having to do with how we construct tag lists in ki clone ops. You can follow along at home if you like: https://github.com/langfield/ki/issues/47

When I first wrote the bug report, I thought this was a relatively innocuous thing, but actually it breaks the parser, which is bad. Luckily it looks like a quick and easy fix! :rainbow:

Edit. Progress below.

---

[langfield]

First successful test build of a deck within GitHub actions! Here's the release.

My hubris in thinking it would be a 3-line program is laughable.

"""Compile a deck."""
#!/usr/bin/env python3
import re
import os
import json
import shutil
import zipfile
import argparse
import unicodedata

from beartype import beartype

# pylint: disable=unused-import
import anki.collection

# pylint: enable=unused-import

from anki import hooks
from anki.exporting import AnkiExporter
from anki.collection import Collection

class AnkiPackageExporter(AnkiExporter):
    """Modified ``.apkg`` writer."""

    ext = ".apkg"

    @beartype
    def __init__(self, col: Collection) -> None:
        AnkiExporter.__init__(self, col)

    @beartype
    def exportInto(self, path: str) -> None:
        """Export a deck."""
        with zipfile.ZipFile(
            path, "w", zipfile.ZIP_DEFLATED, allowZip64=True, strict_timestamps=False
        ) as z:
            media = self.doExport(z, path)
            z.writestr("media", json.dumps(media))

    # pylint: disable=arguments-differ
    @beartype
    def doExport(self, z: zipfile.ZipFile, path: str) -> dict[str, str]:
        """Actually do the exporting."""
        # Export into an anki2 file.
        colfile = path.replace(".apkg", ".anki2")
        AnkiExporter.exportInto(self, colfile)
        z.write(colfile, "collection.anki2")

        media = export_media(z, self.mediaFiles, self.mediaDir)

        # Tidy up intermediate files.
        os.unlink(colfile)
        p = path.replace(".apkg", ".media.db2")
        if os.path.exists(p):
            os.unlink(p)
        shutil.rmtree(path.replace(".apkg", ".media"))
        return media

@beartype
def export_media(z: zipfile.ZipFile, files: list[str], fdir: str) -> dict[str, str]:
    media = {}
    for i, file in enumerate(files):
        file = hooks.media_file_filter(file)
        mpath = os.path.join(fdir, file)
        if os.path.isdir(mpath):
            continue
        if os.path.exists(mpath):
            if re.search(r"\.svg$", file, re.IGNORECASE):
                z.write(mpath, str(i), zipfile.ZIP_DEFLATED)
            else:
                z.write(mpath, str(i), zipfile.ZIP_STORED)
            media[str(i)] = unicodedata.normalize("NFC", file)
            hooks.media_files_did_export(i)

    return media

def main() -> None:
    """Compile a top-level deck into a ``.apkg`` binary."""
    parser = argparse.ArgumentParser()
    parser.add_argument("--collection")
    parser.add_argument("--deck")
    args: argparse.Namespace = parser.parse_args()
    col = Collection(args.collection)

    did = col.decks.id(args.deck)

    exporter = AnkiPackageExporter(col)

    # pylint: disable=invalid-name
    exporter.includeSched = False
    exporter.includeMedia = True
    exporter.includeTags = True
    exporter.includeHTML = True
    # pylint: enable=invalid-name
    exporter.cids = None
    exporter.did = did

    deck_name = re.sub('[\\\\/?<>:*|"^]', "_", args.deck)
    filename = f"{deck_name}{exporter.ext}"
    file = os.path.normpath(filename)
    exporter.exportInto(file)

if __name__ == "__main__":
    main()

Only off by a factor of 36! :face_exhaling:

[langfield]

What's left to do?

• [x] Make compile.py python3.9-compatible.
• [x] Add all decks to release workflow.
• [x] Merge the compile-deck-binaries branch of ki.
• [x] Make a PR here.
• [ ] Profit???

---

Having trouble getting that one deck name to resolve, but I'll figure it out!

Edit. Fully-populated releases page in my fork!

[langfield]

I have not tried importing all the built deck binaries, so it's possible there are minor issues that need to be fixed in there. I recommend sampling a few at random and looking at the notes within an Anki client just as a sanity check.

[KarelZe]

@langfield Thanks for your intense work on ki and this issue. What I saw looks really gorgeous. ♥️ I'll finish reviewing everything incl. the decks tomorrow.

Would love to see you add the github action to your ki repo / docs, as it might be interesting for other maintainers as well.

PPS I'd like to donate you a coffee or two for your hard work. Do you have some paypal of ko-fi profile?

[langfield]

@KarelZe I'm happy you're happy! I'll definitely get that action up there, and get some docs up that show people how to build their .apkgs.

Oh that's very kind, but there's no need. I did make a ko-fi profile because perhaps eventually it will be a good idea. Is it all right if I use one of the screenshots I took above of your deck builds? If not I can find something else! :100:

[KarelZe]

@langfield Thanks again for your work. The decks are looking good so far. Both basic decks and LaTeX-based decks seem to work. I only ran into one issue, that is probably related with pushing back to anki for binary generation / not so much to the GitHub action.

For some images the link in Anki is incorrect, so the image can't be loaded. This is however only true for some images. Here is an example from my recommender deck:

Markdown file as in your repo: ✅

## Note
nid: 1616243314081
model: Basic-b122e
tags: 12_relevante_objekte_empfehlen
markdown: false

### Front
Wie sieht die Linkstruktur von einem Hub in einem Web-Graphen aus?

### Back
Ein guter Hub zeigt auf viele Authorities:
<div><img src=
"paste-64c7a314b90f3e9ef1b2d94edb396e07a121afdf.jpg"></div>

HTML in Anki after import: ❌

Ein guter Hub zeigt auf viele Authorities:
<div><img src="%0A" paste-64c7a314b90f3e9ef1b2d94edb396e07a121afdf.jpg""=""></div>

This results in multiple image files not being found:

Should I file a bug report for ki?

Unfortunately I didn't find your profile? Would you mind sharing a link? Sure you can take a screenshot. 👍

[langfield]

Amazing! :1st_place_medal: This is an excellent find, and a horrible lapse on my part. :disappointed: Feel free to file a bug report here, but I will fix it either way!

Thank you very much, in regards to the screenshot! :rainbow:

You can check out my Ko-Fi profile here!

[langfield]

@KarelZe Be advised, the fix was merged to main within the langfield/ki repository, but no changes have been made within the PR I made for KarelZe/anki-decks. The note files within langfield/karelze-anki-decks very likely still have these broken media paths. Working on a patch.