org-fc-awk: concatenate data when too many files with cards (fix #68)

[l3kn]

Ah, I see how this could help.

I think there's a secondary issue in the current implementation where the number of files passed to the shell commands would exceed some OS limit but this might only be relevant for updates (when caching is enabled), not when using awk to index all files.

Is it possible to estimate or know in advance when xargs will perform this split? Maybe there's some kind of chunking of the list of files on the Emacs side.

[mithraen]

It would be very fragile :(

Can you explain where is this second issue, and how I can reproduce it?

[l3kn]

I think you're right about the fragility, we can assume that find/xargs know best when to split.

The second issue might depend on the operating system and version of Emacs, I think I ran into it after updating to Emacs 30 on linux.

It should be possible to trigger it by enabling the org-fc-cache-mode with lots of org files containing flashcards, or just by calling org-fc-awk-index with a long list of paths.

[l3kn]

We could use (apply #'nconc (read (concat "(" output ")"))) instead, to save the calls to the identity function.

For the AWK review stats, I don't think there's a need to switch to this new parsing because only file will be processed.

Another benefit of this modified parsing is that xargs can be replaced with parallel to save some seconds when processing lots of files.

[mithraen]

Perhaps it would be more advantageous to omit the external braces from the AWK script. This approach would negate the need for 'nconc', 'concatenate', or similar transformations. We would only need to concatenate with braces and then read.

[mithraen]

I have implemented this concept in the updated code.

[l3kn]

Very nice, thanks!