export fails with filename too long

[ComLock]

We need to have an export format that works on most filesystems.

[GlennRicaud]

There has been an export with a content having a name so long that it caused a "File name too long" on Windows when exporting. The workaround for now is to rename the content or to use the dump.

But so, we could have the export flattened and by node ID.

• We would lose the helpful file structure
• We would have to store the path inside the node
• It makes the import slightly more complex (it could in theory create nodes without parents).

To be discussed

[ComLock]

Im not to certain about flattened either, filesystems have a limit to the number of files in a folder.

With a single large binary file the "only" limitation would be file size?

Perhaps some fast compression could be used to?

I know I have mentioned rsync in the past which would make getting updates from nightly export very quick, but I don't know of anyone actually setting it up.

Perhaps deltas would be even better. Like git...

[sigdestad]

We should set a max limit for node names. I.e. 128chars (utf8). Let's check out what is used by various file systems, and support the smallest size (so export will still work).

[rymsha]

128 utf-8 chars is 512 bytes max (assuming full utf-8 support). Way too much for filesystems like ext4 exFat etc. Most common limitation is 255 bytes.

https://en.m.wikipedia.org/wiki/Comparison_of_file_systems

[sigdestad]

Hmmm. We do require use of utf-8 filsystem for servers. I thought most modern OS' would support this by now?

[rymsha]

They do, this is not a question here.

So most interesting limitations for modern systems are:

• 255 bytes per filename for ext4 and others per filename. (Often 255 chars but sometimes just ~60)
• 255 ucs-2 chars per filename on fat32 (flash drives) - this limitation is not strictly string length one. This one disallows a lot of modern urf (emoji 😬)
• ~32760 utf-16 chars (2 or 4 byte per character) on NTFS/exFAT per full path. This is the most interesting one as it makes the requirement theoretically impossible as it depends on how deep the parent directory already is. But ok, let's be practical and round limit to 32000 utf-16 chars leaving 720 chars for parent path.

[rymsha]

Sooo... In path we can only support character set representable by ucs-2 with max 255 bytes per segment (when utf-8 encoded), maximum 32000 characters for full path.

[rymsha]

Most likely people hit 260 characters per filepath on Windows 10. But there is a workaround via GPO Enable Win32 long paths

[rymsha]

And worth to mention on Windows some filenames (COM1, LPT, etc...) are illegal

[GlennRicaud]

Decide in PAB if we want to have a flatten structure