Adjust behavior of `use` statement to improve ease-of-`use`

[rdw20170120]

Pun intended.

In #974, @zzamboni said

@rdw20170120 I'm not sure to which documentation you are referring, but the use of .elv is literally everywhere in the Elvish documentation, so it's pretty easy to pick up.

@zzamboni, I think that we are misunderstanding each other due to a crucial subtlety.

You correctly point out that the .elv file suffix is present in much of the Elvish documentation. I disagree about the use of "everywhere". I think that Elvish would benefit greatly from many more examples of real-world usage that include using actual Elvish script in actual source files. I observe that most of the Elvish documentation describes Elvish from a purely syntactic and language-feature usage, devoid of any context such as source files.

One can find similar mentions of similar file suffixes in the documentation of much software, whether open-source or not. For example, the Fish documentation similarly references the .fish file suffix often.

However, I am specifically referring to documentation regarding the mandatory use of such a file suffix by the software.

For example, the Fish documentation tells us that Fish will lookup the source for a function X by searching its fish_function_path for a file called X.fish, thereby requiring the file suffix. If such a file is not found, then Fish will be unable to source it, and therefore will not have a definition for the function, and Fish will then report an error because it is unable to execute a call to function X. On the other hand, if the user writes a script in a file named script, Fish has no trouble executing it without the .fish file suffix because Fish does not require such a suffix in that use case. Indeed, many if not most files containing Fish source code do not require the .fish file suffix, and Fish works fine without it. Only in a few specific cases does Fish enforce the requirement for the .fish file suffix.

So, yes, the Elvish documentation shows some examples of using the .elv file suffix. As with Fish and other software, those uses of the file suffix are optional. Elvish will happily execute most Elvish source code in files with no file suffix, or with any file suffix that the user cares to apply.

Indeed, it is a common convention that scripts should NOT have a file suffix at all, and they should rely on PATH and the shebang for proper execution using the correct interpreter.

So, when the Elvish use statement requires the .elv file suffix, that is very much worthy of explicit and prominent mention in the documentation. I did not find it to be prominent, and I only found it to be explicit in one place in all the Elvish documentation. It took me several weeks to diligently study ALL of the Elvish documentation until I finally encountered this absolutely essential piece of information.

The understandability of the use statement and its documentation is hampered by the fact that Elvish implements a very unusual (unique?) approach to the mapping between the source file and the statement to use a module. In my forty-three years of software engineering, over many dozens of programming and scripting languages as well as other languages of all kinds, I cannot think of a single example of a similar source-file mapping.

The use statement is intended to load an Elvish "module", whose content stays in memory for the rest of the current session. That module is a file containing Elvish script, and that file is required to have a .elv file suffix. I think all this is fine, up to this point. The problem arises because the use statement requires a pseudo-relative-path reference to that file, but WITHOUT the file suffix. Without the file suffix, such a reference is NOT actually a relative path. With the file suffix, the reference would indeed be a relative path.

When we compare this to the example of a Fish function call, perhaps we can see the significance more easily. A function call is obviously NOT a relative path, or any kind of filesystem path at all. A function call merely references the name of the function. When Fish takes that function name, adds the required .fish file suffix, and then searches the fish_function_path (a list of directories) for a matching file with that name, this is a readily understandable and conventional way for software to behave.

Similarly, when a user writes something like import a.b.c.something in Python, it is readily understandable and conventional that Python transforms the . into directory separators, adds a .py file suffix, and searches for that relative path within PYTHONPATH (list of directories), etc.

Neither of those implementations--nor any other implementation of a similar concept that I can recall in any other language--chooses to obscure the approach by attempting to redefine such a basic concept as "relative path". Likewise, those implementations--and every other one that I can recall--are very careful to explicitly and prominently document the implementation for all to see.

I hope then, @zzamboni, that I have now clarified that Elvish does not actually document this behavior "everywhere". Most of the .elv file suffix uses are optional. I found one place that the file suffix is mandatory. I literally spent several weeks studiously searching for that documentation as I tried to make use work for me. I cannot help but wonder how many other places exist where the file suffix is likewise mandatory but likewise documented poorly or not at all.

I would NEVER have guessed that Elvish behaves this way, requiring a pseudo-relative-path that excludes a file suffix while mapping that to a file that requires the file suffix. When I finally proved the behavior to myself with a local test, I was dumbfounded by such counterintuitive behavior. I could not believe it even though I was staring right at it. This is not the first time that a piece of software surprised me like that, but it is a generally recognized antipattern that should be avoided.

I am particularly concerned that this is somehow worth fussing about and denying. All I am saying is that "I found this to be an obstacle to my adoption of Elvish". As I also said elsewhere, studies and statistics indicate that I am likely far from alone in my experience. Such issues tend to be reported by a few, but experienced by many more users who silently move on. Estimates vary from ten times as many silent users to perhaps a thousand times as many unreported incidents.

So, now I have created an issue for it. May we please have a conventionally-behaving bit of functionality here? If you want to use a "relative path" for the module reference, may we please have a real relative path with the same definition used by the rest of the computer industry? Would you please include the file suffix in the use statement, if the module file has one? I like the .elv file suffix, so I encourage keeping it. I also encourage making a very clear distinction in the documentation for when Elvish requires the file suffix (or any others, if applicable elsewhere in the syntax) versus when they are optional.

[krader1961]

For example, the Fish documentation tells us that Fish will lookup the source for a function X by searching its fish_function_path for a file called X.fish, thereby requiring the file suffix.

That is a reasonable analogy. It seems to me that improving the documentation at relevant points, such as https://elv.sh/ref/language.html#user-defined-modules, is sufficient to resolve your primary point. Yes? In other words, modify the documentation to explain that an implicit .elv extension is used when looking for a relevant file.

[zzamboni]

@rdw20170120 In summary: you think the elvish docs need to be improved. This seems fair, and you have in fact written about this extensively already. I would encourage you you rephrase what you wrote above into the appropriate improvements and contribute a PR to the docs.

[rdw20170120]

Fair enough, gentlemen. Thank you for your feedback. I will look into contributing a PR, though I cannot promise it soon.

I just encountered another documentation issue regarding use that I could also address. Specifically that one cannot introspect the builtin module without first use builtin. There is an example of introspection that does not work because the necessary use statement is not part of the example.

[krader1961]

I think this can be closed. The use documentation has been augmented to make it clear that the file corresponding to a user defined module must have a .elv extension.