Copytree raises an error on `push` operation

[langfield]

The following error happened on a fresh clone, when adding a deck using convert.py (from quizlet) and possibly as a result of not committing the new directory and contents. Was not able to reproduce.

Traceback (most recent call last):
  File "/home/user/conda/envs/anki/bin/ki", line 33, in <module>
    sys.exit(load_entry_point('ki', 'console_scripts', 'ki')())
  File "/home/user/conda/envs/anki/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/conda/envs/anki/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/user/conda/envs/anki/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user/conda/envs/anki/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/conda/envs/anki/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "<@beartype(ki.push) at 0x7f3683ec6430>", line 10, in push
  File "/home/user/pkgs/ki/ki/__init__.py", line 2082, in push
    head_kirepo: KiRepo = copy_kirepo(head, f"{HEAD_SUFFIX}-{md5sum}")
  File "<@beartype(ki.copy_kirepo) at 0x7f3683ea4dc0>", line 49, in copy_kirepo
  File "/home/user/pkgs/ki/ki/__init__.py", line 226, in copy_kirepo
    ephem: git.Repo = copy_repo(ref, suffix)
  File "<@beartype(ki.copy_repo) at 0x7f3683ea4ca0>", line 49, in copy_repo
  File "/home/user/pkgs/ki/ki/__init__.py", line 192, in copy_repo
    ephem = git.Repo(F.copytree(F.working_dir(repo_ref.repo), target))
  File "<@beartype(ki.functional.copytree) at 0x7f3683f24ca0>", line 49, in copytree
  File "/home/user/pkgs/ki/ki/functional.py", line 71, in copytree
    shutil.copytree(source, target, symlinks=True)
  File "/home/user/conda/envs/anki/lib/python3.9/shutil.py", line 566, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/home/user/conda/envs/anki/lib/python3.9/shutil.py", line 522, in _copytree
    raise Error(errors)
shutil.Error: [('/home/user/collection/.git/objects/2b', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/2b', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/2b'"), ('/home/user/collection/.git/objects/1c', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/1c', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/1c'"), ('/home/user/collection/.git/objects/0e', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/0e', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/0e'"), ('/home/user/collection/.git/objects/12', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/12', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/12'"), ('/home/user/collection/.git/objects/1f', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/1f', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/1f'"), ('/home/user/collection/.git/objects/31', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/31', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/31'"), ('/home/user/collection/.git/objects/11', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/11', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/11'"), ('/home/user/collection/.git/objects/09', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/09', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/09'"), ('/home/user/collection/.git/objects/6a', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/6a', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/6a'"), ('/home/user/collection/.git/objects/01', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/01', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/01'"), ('/home/user/collection/.git/objects/20', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/20', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/20'"), ('/home/user/collection/.git/objects/79/c68fdc13ae8fa9ab6f7038d05d54fd1c8e33c0', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/79/c68fdc13ae8fa9ab6f7038d05d54fd1c8e33c0', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/79/c68fdc13ae8fa9ab6f7038d05d54fd1c8e33c0'"), ('/home/user/collection/.git/objects/79/53f4522b1450fe74bbb96a543c3b7a75ab5a61', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/79/53f4522b1450fe74bbb96a543c3b7a75ab5a61', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/79/53f4522b1450fe74bbb96a543c3b7a75ab5a61'"), ('/home/user/collection/.git/objects/79/3f1448e8c4ff0ab377a7559b60a442f0ef810d', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/79/3f1448e8c4ff0ab377a7559b60a442f0ef810d', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/79/3f1448e8c4ff0ab377a7559b60a442f0ef810d'"), ('/home/user/collection/.git/objects/79/c556f2621ca5e4981d83160a070a0aa087f4a7', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/79/c556f2621ca5e4981d83160a070a0aa087f4a7', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/79/c556f2621ca5e4981d83160a070a0aa087f4a7'"), ('/home/user/collection/.git/objects/79/889f9318978003b0e647037a222509b5f99d81', '/tmp/tmpwe3hpvmd/ki-head-76e525590c8ca7291f647c23f0e8d861/.git/objects/79/889f9318978003b0e647037a222509b5f99d81', "[Errno 2] No such file or directory: '/home/user/collection/.git/objects/79/889f9318978003b0e647037a222509b5f99d81'"), ...

It should be noted that using Quizlet to Anki 2.1 Importer with audio support is a much better way of importing from quizlet than using ki.

[langfield]

This happened again while testing a fix for spurious <br> tags in field HTML. Possibly a race condition.

[langfield]

Unable to reproduce, will reopen if this issue persists.

[lukas-mertens]

@langfield I debugged this error a bit and found out it is a race condition in shutil.rmtree whenever a file is deleted inside a dir during deletion

Here is a pr which tries to fix it: https://github.com/python/cpython/pull/14064

You can reproduce it on macos on non-standard file systems, it happens when I use ki in my /tmp for example

[langfield]

@lukas-mertens Oh wow, perhaps I'll reopen it then! Thanks 😀

Incidentally, have you used this tool much? I'd love to get some feedback, although I am aware of many things that are currently broken. 🙃 I am in the process of a heavy rewrite of the integration tests to make it easier to write new ones.

There is also a branch that contains a Haskell rewrite, and a branch that contains something that parses database diffs from sqldiff. My current plan is to fix the existing bugs in the python version and then jump back to the Haskell branch.

I am still working on it, even though it looks a little dead, haha. Feel free to watch progress on this PR.

[lukas-mertens]

@langfield Haven't used it much yet, but I was a user of CrowdAnki-Diff and developed a tool which allows users to do visual merge request reviews. So far, the idea seems pretty nice. One recommendation though: If possible, it would be nice if Anki could automatically perform a "ki pull" after every sync. This way my anki database is automatically backed up to git.

[langfield]

Oh that's sweet! I'd love to see this tool, if you could send a link!

You say "was", why do you no longer use CrowdAnki-Diff?

Hmmm well it's not really possible, since a pull consists of a fetch + merge, and merge is a manual operation that git can sometimes do automatically, as I'm sure you know. I could make it fetch automatically, but it's out of scope for now as currently there's no interface between the two programs. 🙃

[lukas-mertens]

Oh, that is not a problem. Just do a git fetch origin, then a git merge --no-commit --no-ff origin/<your-branch-name>, check if it is successful and if it is, do a git merge --abort. This way you will check if the pull is going to work without user-intervention ;)

If it isn't, just show the user a little banner in anki, that they have to perform it manually.

Here it is: https://gitlab.com/dodod/crowdanki-diff/-/tree/master

It is awful angular code I wrote in a night without much sleep. But don't waste your time on it for now. It could be adapted to ki as well, but I think a rewrite would be better than forking this tool. And I don't know how important such a thing is, given that Markdown is previewed by many git hosters anyways.

I stopped using CrowdAnki, because it wasn't robust enough. Sometimes it fucked up your database. I had hoped that ki would maybe be better regarding this, which is why I gave it a try.

[langfield]

Well ki will currently also mess up your database (though not too badly I hope, and also the automatic backup on every op is pretty robust), so feel free to wait until it's declared stable!

---

From: Lukas Mertens @.> Sent: Thursday, August 24, 2023 6:29:21 PM To: langfield/ki @.> Cc: langfield @.>; Mention @.> Subject: Re: [langfield/ki] Copytree raises an error on push operation (Issue #12)

Oh, that is not a problem. Just do a git fetch origin, then a git merge --no-commit --no-ff origin/, check if it is successful and if it is, do a git merge --abort. This way you will check if the pull is going to work

Oh, that is not a problem. Just do a git fetch origin, then a git merge --no-commit --no-ff origin/, check if it is successful and if it is, do a git merge --abort. This way you will check if the pull is going to work without user-intervention ;)

If it isn't, just show the user a little banner in anki, that they have to perform it manually.

Here it is: https://gitlab.com/dodod/crowdanki-diff/-/tree/masterhttps://urldefense.com/v3/__https://gitlab.com/dodod/crowdanki-diff/-/tree/master__;!!KGKeukY!y8Mi_BCQDisErvFRDvu3zqiesGj59sKgdDjX-J_WzoLq0UECzqSwgMkmqR25EWjhwvN8tMHfSdeOYThuUChmjxsPqUMasQ$

It is awful angular code I wrote in a night without much sleep. But don't waste your time on it for now. It could be adapted to ki as well, but I think a rewrite would be better than forking this tool. And I don't know how important such a thing is, given that Markdown is previewed by many git hosters anyways.

I stopped using CrowdAnki, because it wasn't robust enough. Sometimes it fucked up your database. I had hoped that ki would maybe be better regarding this, which is why I gave it a try.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/langfield/ki/issues/12*issuecomment-1692496823__;Iw!!KGKeukY!y8Mi_BCQDisErvFRDvu3zqiesGj59sKgdDjX-J_WzoLq0UECzqSwgMkmqR25EWjhwvN8tMHfSdeOYThuUChmjxspnfeohA$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AISQNI3A472MXL6VTWTL4BDXW7IUDANCNFSM53Z5YSWA__;!!KGKeukY!y8Mi_BCQDisErvFRDvu3zqiesGj59sKgdDjX-J_WzoLq0UECzqSwgMkmqR25EWjhwvN8tMHfSdeOYThuUChmjxtNW2ZSaw$. You are receiving this because you were mentioned.Message ID: @.***>

[langfield]

Seems irrelevant with recent scope reduction.