[Maiko] bus error/segmentation fault when too many versions of a file in a directory

[nbriggs]

The local file code handling versioned files in Maiko smashed memory when there are more than 200 versions of a file in a directory, running unchecked off the end of a fixed size (VERSIONARRAYLENGTH) array.

• this MUST NOT fail like this (at minimum, report an error)
• this SHOULD dynamically allocate the storage as necessary to efficiently handle larger collections of versioned files

[nbriggs]

@rmkaplan - first level of fix is to report an error rather than crash - this is what it would look like if it sends back an EMFILE (too many open files) system error when it encounters more than 200 versions of a file you're operating on:

Next level fix is to make it allocate storage (this is all outside Lisp memory) for the version array as required.

[rmkaplan]

TOO MANY FILES OPEN is not quite the right message, since there may only be one open (e.g. the one being created).

Can you do

TOO MANY VERSIONS OF ?

On Sep 9, 2024, at 1:45 PM, Nick Briggs @.***> wrote:

@rmkaplan https://github.com/rmkaplan - first level of fix is to report an error rather than crash - this is what it would look like if it sends back an EMFILE (too many open files) system error when it encounters more than 200 versions of a file you're operating on: Screenshot.2024-09-09.at.1.38.25.PM.png (view on web) https://github.com/user-attachments/assets/3f80fffb-4131-4c64-aa81-16b08da1fe4a Next level fix is to make it allocate storage (this is all outside Lisp memory) for the version array as required.

— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/1826#issuecomment-2339055886, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJONVWDBOPN5NWPBQ6DZVYCIFAVCNFSM6AAAAABN5HPVNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZGA2TKOBYGY. You are receiving this because you were mentioned.

[nbriggs]

There's no UNIX error that talks about versions -- and all we get to pass up at this point is a Unix error number -- do man errno and tell me if one of those says something you think is more appropriate (EMFILE was the one I chose)

[rmkaplan]

Actually, from the Lisp point of view, it isn’t that there are too many “open” files. The situation I encounter is trying to make a new file FOO when there are maybe only 3 other files actually lisp-open, but the directory is filled up with other files. Is there another message that is a better approximation? Otherwise, this is fine—most people don’t work the way I have been, with all of these Tedit files and versions.

On Sep 9, 2024, at 2:28 PM, Nick Briggs @.***> wrote:

There's no UNIX error that talks about versions -- and all we get to pass up at this point is a Unix error number -- do man errno and tell me if one of those says something you think is more appropriate (EMFILE was the one I chose)

— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/1826#issuecomment-2339177226, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJIFL3522RMLAB23JO3ZVYHJXAVCNFSM6AAAAABN5HPVNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZGE3TOMRSGY. You are receiving this because you were mentioned.

[nbriggs]

Yeah, I know that it's not actually open files, it's trying to do certain operations on a file that has over 200 distinct versions, which causes it to enumerate those versions and attempt to store them in an array that is too small. Please look at the output of man errno in a terminal window and see if you see anything you'd be more comfortable with (e.g., I could go more generic and use EIO "INPUT/OUTPUT ERROR") -- but in the longer term, I think I will be able to just fix the problem of static array allocation.

[rmkaplan]

Here are a few other possibilities...

EIO Input/output error ENOSPC Device out of space. ENOBUFS No buffer space available ENOSTR Not a STREAM.

If the error shows up in a Lisp break window, is there an easy way to catch it with a condition and also print out some advice (e.g. delete some files)?

But the important thing is not to cause a fatal crash in this unusual situation, anything else is a bonus.

On Sep 9, 2024, at 4:41 PM, Nick Briggs @.***> wrote:

Yeah, I know that it's not actually open files, it's trying to do certain operations on a file that has over 200 distinct versions, which causes it to enumerate those versions and attempt to store them in an array that is too small. Please look at the output of man errno in a terminal window and see if you see anything you'd be more comfortable with (e.g., I could go more generic and use EIO "INPUT/OUTPUT ERROR") -- but in the longer term, I think I will be able to just fix the problem of static array allocation.

— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/1826#issuecomment-2339337092, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJLIU46Q76NW6DIGUQTZVYW2DAVCNFSM6AAAAABN5HPVNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZGMZTOMBZGI. You are receiving this because you were mentioned.

[nbriggs]

Using EIO results in a SIMPLE-DEVICE-ERROR being thrown. That should be catchable -- BUT, after this initial fix gets merged I'm going to rewrite a bunch of the code in there so that it will be extremely unlikely to happen again (I think you'd need millions of versions of a file to provoke it)

[nbriggs]

Closed by PR Interlisp/maiko#512