media folders with 12k+ files not supported

[hssm]

Originally reported on Google Code with ID 1158

As reported on the Anki forums, if the internal memory or SD card are on a VFAT partition,
there's a limit of about 12k files in a folder, and attempts to write more than that
many files fail. Most desktop users are on NTFS/HFS+/EXT3+ which all scale to that
many files fine, and as iOS also uses HFS+, this issue seems to be limited to old desktops
on VFAT and AnkiDroid.

Perhaps the easiest solution is to provide an option to save the files in a zip and
extract them when they're first referenced on a card, leaving them there. That would
minimize extra writes compared to removing them after a card is displayed. Storing
them in lots of subfolders seems a much more complicated solution: you wouldn't be
able to rely on the html <base> tag for media any longer, and it would make it considerably
harder for users to know which folder media is located in.

Forum thread for reference: https://groups.google.com/forum/?fromgroups#!topic/ankisrs/vrvELhL5V4M

Reported by dae@ichi2.net on 2012-06-04 06:18:35

[hssm]

Thank you Damien,

I actually made a mistake when I said ~12k files; it was actually closer to ~20k files
and strongly dependent on the length of the file names.

That workaround seems like it could be a great solution!

Reported by perceptualChaos on 2012-06-04 06:28:02

[hssm]

Reported by nicolas.raoul on 2012-06-04 09:13:43

• Status changed: Accepted
• Labels added: Priority-Medium
• Labels removed: Priority-Undecided

[hssm]

Reported by inigo.aldana on 2012-07-24 13:42:08

• Labels added: DevelopmentVersion

[hssm]

Thanks for this. Perhaps another workaround would be function in Anki to rename the
media files to shorter file names.

Reported by intercept.x.attack on 2012-08-12 20:16:37

[hssm]

Also see this thread:
https://groups.google.com/forum/?fromgroups#!topic/anki-android/tI_FTtbAqas%5B1-25%5D

As per link in above thread, renaming to shorter files would increase limit to 65,536
which is more than I currently have in my collection.media folder (50,000) but others
may have even more. Additionally the zip file solution has the advantage that it's
MUCH faster to copy the media files to the phone with USB2. As per above, another workaround
(requires root and external SD slot) is to format your SD card as NTFS or EXT4.

Reported by perceptualchaos2 on 2012-08-12 22:00:31

[hssm]

>>As per above, another workaround (requires root and external SD slot) is to format
your SD card as NTFS or EXT4.

Just did this and it worked. Besides root, you also need an app to mount the card as
by default Android will just say it's an unsupported format. The app is called NtfsSD
and it's free in the Play Store.

Thanks for your help!

Reported by intercept.x.attack on 2012-08-12 23:34:39

[hssm]

Issue 1480 has been merged into this issue.

Reported by inigo.aldana on 2012-11-19 09:02:34

[hssm]

One possible option that has not been mentioned yet is ExFAT, which is supported out
of the box on some devices.

Reported by dae@ichi2.net on 2012-11-19 09:07:17

[hssm]

Issue 1469 has been merged into this issue.

Reported by perceptualchaos2 on 2013-01-09 06:39:44

[hssm]

What about mounting the archive like e.g. ubuntu does? So, leave the mp3 in the archive
(zip, tar.gz etc) and access the files that way. Peter.

Reported by wave.root on 2013-01-12 13:47:36

[hssm]

If there's any solution then it'd be good because I'm  not so tech savvy and rooting
is just beyond the scope of my ability :(

Reported by kondeeza.123 on 2013-01-13 08:21:53

[hssm]

Issue 1597 has been merged into this issue.

Reported by inigo.aldana on 2013-01-13 19:53:43

[hssm]

Another approach could be to use sub-folders and sort the files into those by very specific
rules.

Mediawiki does this, when you look at a full-size image on Wikipedia, you have something
like "/X/XY/" in the URL/path with two hex digits X and Y.

I am not sure about the rules, probably some kind of hash on the file name.

Reported by ospalh on 2013-01-15 11:07:58

[hssm]

Is it true that this issue is only relevant for VFAT file systems?

Reported by inigo.aldana on 2013-01-15 11:13:55

[hssm]

>Is it true that this issue is only relevant for VFAT file systems?

Yes, only FAT has this very low limit.

But i tried and couldn't get anything else to work. 

(Might be related to the fact that it's a cheap no-name model...)

Reported by ospalh on 2013-01-15 13:15:07

[hssm]

If you use an external SD card then NTFS is confirmed to be working. It
should also be possible to format the internal SD card as either NTFS or
EXT3/4 with a bit more work, though I don't think anyone here as reported
doing this yet.

Reported by perceptualchaos2 on 2013-01-15 13:21:39

[hssm]

Oh.
I had tried the "NtFS Mounter", *that* din't work.

Now i tried "Paragon NTFS", looks like that works. So i'm just now copying a ton of
media files. :) 

Reported by ospalh on 2013-01-15 13:48:15

[hssm]

Can we get a real solution to this problem within AnkiDroid's code? Nested folders as
suggested by #13 would be perfect. Media zips as suggested by Damien would be ideal
for transfer speeds.

After updating to JB, I can't mount NTFS or EXT3/4 anymore. Which really sucks as I
hit the file count limit way too easily. Managing which media files I need for the
week is nearly impossible.

Since it seems like NTFS is no go for me unless I jump back to ICS, is there any good
workarounds for this? Like a script to sort out all the media on cards that will be
due within the next week, and/or a script to rename all the media to shorter filenames?

Reported by daichi09 on 2013-03-22 21:05:45

[hssm]

It shouldn't be too hard to make an add-on for Anki Desktop which splits the media files
into subfolders and updates the path accordingly on each of the notes. I think it should
work out of the box on AnkiDroid.

You could try giving it a go yourself... or someone here may have already made one;
I think Houssam is doing this.

Reported by perceptualchaos2 on 2013-03-23 01:53:32

[hssm]

No, I'm not working on an add-on. I manually moved files into subfolders. I still had
them in separate folders from before the upgrade, so it was easy to tell which files
belonged to which deck. Then it was simply a matter of changing my template to have
<img src="mySubFolder/{{imgfield}}"/>

If your image field already has the <img> tags in it, you will need to remove them
with the search and replace feature in Anki's browser. It supports regex, and the example
in the manual is exactly like this case, so you can copy it. One problem I ran into
with that: my fields were using double quotes (src="asdf.gif") but Anki was showing
them with single quotes (src='asdf.gif') in the HTML editor (ctrl+shift+x), so the
regex didn't work until I figured that out and changed the quote style.

Reported by Houssam.Salem.Au on 2013-03-23 03:53:44

[hssm]

Ah I see, thanks for the clarification. It sounds very straightforward; maybe I'll add
it to the FAQ.

Reported by perceptualchaos2 on 2013-03-23 04:38:14

[hssm]

#19 daichi09:
> (... I)s there any good workarounds for this? Like (...) a script to rename all >
the media to shorter filenames?

Well, it is very much not a *good* workaround, as at the moment it gives *longer* file
names, but maybe my Dehasihlator add-on attempt could be used as a starting point.
It does go through the collection and renames media files and changes the cards to
use the new file names.
It works reasonably well when you only have one image or video in each field and use
each image/video on only one note. If that is not the case your collection may (or
may not) end up in a mess. (It worked well enough for me, so i just left it there not
really finished after my collection was done.)

Of course to be a solution to this problem, you would have to change the bits that
return the new file name and those that determine what counts as "problem" files.

See http://ospalh.github.com/anki-addons/Dehashilator.html 

I mentioned (#13) the way Mediawiki uses subfolders like "4/41/asdf.gif".
See http://www.mediawiki.org/wiki/Manual:$wgHashedUploadDirectory they use MD5 hashes
of the file name.

Reported by ospalh on 2013-03-23 07:53:50

[hssm]

Issue 1830 has been merged into this issue.

Reported by Houssam.Salem.Au on 2013-09-26 08:28:34

[hssm]

Actually what's are the issues preventing AnkiDroid from fully supporting subfolders?

As far as I can see with my own decks it works fine. I have organized my files into
subfolders recently because having 50k of them in a big heap is cumbersome even with
a good filesystem. I see that Anki desktop has some minor problems with that (it's
dropping files always in the main folder and not seeing files in subfolders when I
use "Check Media") but nothing serious. It surely beats having the tablet slow down
when copying or refuse to accept some files over FTP for mysterious reasons when they're
all together or MTP hang file manager when the folder has too many entries... So why
not?

Reported by cameel2 on 2015-01-16 03:23:24

[hssm]

Export / Import doesn't work with subfolders, and "fully supporting" means
that we would try our best not to make any changes which break subfolders,
which isn't something we'd want to do.

Reported by perceptualchaos2 on 2015-01-16 05:57:21

[hssm]

Is there still no fix for this problem aside from needing to reformat as a workaround?

Reported by leoraphael7@yahoo.com on 2015-07-27 09:30:04

[hssm]

Im not sure but personally I havent had an issue with synchs taking 30
minutes in months, before it used to be an almost daily occurence. I
figured something had happened behind the scenes to fix it. Good luck
fixing your issue.
27 lip 2015 11:30 <ankidroid@googlecode.com> napisał(a):

Reported by jakubwalenczak on 2015-07-27 09:38:15

[jytou]

My main problem is still with Android's media android.process.media process taking loads of time in the media directory and eating up the CPU/battery all the time. So even if I understand that dumping all the media in a single directory may be convenient for the anki developers, I find it poor practice. And I'm a computer engineer, not a "non-computer-friendly" person. It definitely would be best to find another option for this. Either SQL database, either subfolders (which would still not 100% fix the process.media problem, but would somehow be better than only one directory). I consider moving all my media to a web server rather than having to have to duplicate them on all my devices. I understand that this would break the "check media" paradigm, but that can be done differently with a plugin.

[timrae]

Are you using an external sd card? On 31/01/2016 3:05 am, "jytou" notifications@github.com wrote:

My main problem is still with Android's media android.process.media process taking loads of time in the media directory and eating up the CPU/battery all the time. So even if I understand that dumping all the media in a single directory may be convenient for the anki developers, I find it poor practice. And I'm a computer engineer, not a "non-computer-friendly" person. It definitely would be best to find another option for this. Either SQL database, either subfolders (which would still not 100% fix the process.media problem, but would somehow be better than only one directory). I consider moving all my media to a web server rather than having to have to duplicate them on all my devices. I understand that this would break the "check media" paradigm, but that can be done differently with a plugin.

— Reply to this email directly or view it on GitHub https://github.com/ankidroid/Anki-Android/issues/2060#issuecomment-177262908 .

[jytou]

Yes (but at the beginning, I wasn't - in the end it doesn't change a thing). I also tried adding a dot at the beginning of the folder name (some claim that this would "lure" the media scanner into thinking it's a system folder and thus not scan it, but it still gets in it), as well as the regular ".nomedia" file, to no avail. Note that my tablet is not that new, it's a Galaxy Tab 2 10.1, Android 4.1.2. And no I don't plan upgrading it.

[jytou]

... and just letting you know that beside this media library problem, Anki is a wonderful piece of software. :)

[timrae]

I had the same problem with my galaxy S2. I don't have either of the 12k limit or the scanning issues after upgrading to an Xperia, which has a decent internal filesystem.

As far as I'm concerned, they're both legacy issues that can be fixed by upgrading to a newer device, so I don't feel it's worthwhile for us to make any big architectural changes. On 31/01/2016 8:58 am, "jytou" notifications@github.com wrote:

Yes (but at the beginning, I wasn't - in the end it doesn't change a thing). I also tried adding a dot at the beginning of the folder name (some claim that this would "lure" the media scanner into thinking it's a system folder and thus not scan it, but it still gets in it), as well as the regular ".nomedia" file, to no avail. Note that my tablet is not that new, it's a Galaxy Tab 2 10.1, Android 4.1.2. And no I don't plan upgrading it.

— Reply to this email directly or view it on GitHub https://github.com/ankidroid/Anki-Android/issues/2060#issuecomment-177340106 .

[aleksejrs]

My Alcatel smartphone with Android 5.0 supporting only FAT32 on the 64 GB SD card starts having serious issues (frequently unmounting the SD card or refusing to mount it altogether) with less than 300 files with some very long names in a single directory (the names in Anki are shorter, so it worked with over 800 files, but that was just 1 to 3 days of due cards).

PS: Huh, did I unassign the issue or what?

[mikehardy]

Looks like it is unassigned now, but it's also more than 2 years since an update on an issue that's not going to be fixed I don't think...shouldn't matter?

[jytou]

I have found an alternative (I'm actually storing my media on an external server and only have links to it). It's working for me. I still think that forcing everything to be in one single directory is a recipe for trouble.

[mikehardy]

...and a PR that maintains the "flat" appearance of the media for import/export/synchronization but transparently sends and/or moves things like "AA.mp3" to "A/AA.mp3" with no locking, synchronization, or data-loss issues across the entire Android API version and device skew of the installed base should be easy to code up. I'll be happy to review and help QA it... To avoid doubt, I am being a little sarcastic - I mean, that's probably how I'd go about solving this and maybe it would even be fun to code up, but it's not a small thing. And it doesn't get reported much whereas we have some other pretty serious issues to work on.

[github-actions[bot]]

Hello 👋, this issue has been opened for more than 2 months with no activity on it. If the issue is still here, please keep in mind that we need community support and help to fix it! Just comment something like still searching for solutions and if you found one, please open a pull request! You have 7 days until this gets closed automatically

[connwork]

Does anyone know if this is still an issue? It is closed as completed but I cannot find the fix details.

[timrae]

See https://github.com/ankidroid/Anki-Android/issues/2060#issuecomment-177340106. We considered it to be a legacy issue as it's only a problem with FAT32 file system, which we don't really see being deployed in modern devices anymore. Therefore we decided no fix is required.