Adding operations to load a module from an FS path, and display the loaded modules

[luketpeterson]

Based on a Mattermost thread from @tanksha, I added two new MeTTa library operations:

load-module! takes a file system path and loads the module into the runner print-mods! prints the tree of loaded modules, which can be handy for import debugging

With these operations, there are two ways @tanksha can accomplish what is in the thread:

1. Load the parent directory as a module, and import relative to that.

   !(load-module! ../data)
   !(import! &self data:sample)

2. Load the modules you want explicitly prior to importing them

   !(load-module! ../data/sample.metta)
   !(import! &self sample)

You can use the print-mods! op to display the name of the modules that are loaded.

Let me know if this works for you.

[vsbogd]

Why cannot we just do (import! ..mod_name)? It probably was discussed somewhere but I don't know the reason. Having both load-module! and import! seems confusing to me.

[luketpeterson]

Why cannot we just do (import! ..mod_name)?

import! doesn't take a file system path, it takes a module name (or module name path). Now documented here: https://github.com/luketpeterson/hyperon-experimental/blob/modules/docs/modules_dev.md#module-names--name-paths

But there is currently no ".." token to access the parent module. We could add ".." or I prefer "super", similar to what Rust and some other languages use, but regardless that operator works within the module hierarchy, which is not the same thing as the file system.

Having both load-module! and import! seems confusing to me.

import and load are two different operations. import is like Rust's use; It's a mapping between two modules. But the referenced module must be loaded (or loadable) first. Normally import! has a "load-on-demand" behavior that follows the search path outlined in the documentation. But when there is a desire to bring in a module that isn't findable by the search behavior, it needs to be loaded explicitly.

[luketpeterson]

Just for reference, here is the thread: https://chat.singularitynet.io/chat/pl/f7rcptmi1ib75bocp6k4ep9e7r

Sometimes searching in Mattermost is tricky so I'm mainly adding this for future reference.

[vsbogd]

that operator works within the module hierarchy, which is not the same thing as the file system.

I understand the desire to separate module and file system hierarchy. But they are already mixed because (import! <some-name>) tries to find the <some-name> module in a current working directory. At least it is what MeTTa program author sees. If we don't want to mix them then we should allow excluding current working directory from module hierarchy. And allow excluding it by default as well. We have such option on the level of the Rust API but I believe we don't have it in MeTTa do we?

When I am saying that load-module! is confusing it is because load-module! and import! sound not very different. Also parts of the filesystem are the parts of module hierarchy. Even when user don't do anything special current working directory is a part of the module hierarchy by default (or it behaves like this). Thus make separation of modules and filesystem paths clear to the user is not easy.

In Java for instance module hierarchy is formed by the internal "classpath" variable. Adding folder or package to this variable makes it a part of the module hierarchy. I believe load-module! should work similarly but name is not good. For example default "classpath" is defined via MeTTa script which is loaded on start. And we can modify it using some operator (load-module! with another name).

[luketpeterson]

...But they are already mixed because (import! ) tries to find the module in a current working directory... At least it is what MeTTa program author sees.

My guiding philosophy when architecting any interface that will be used by a human (mainly an API used by a programmer, but it applies to any UI, in this case the set of commands & operations available in MeTTa) is to balance the tension between two competing goals.

1. The interface should provide the best reasonable mix of flexibility, power, and efficiency, for the functionality and abstractions being exposed.
2. The interface should "just work" from the perspective of somebody who doesn't take the time to read the documentation and learn the abstractions.

Fundamentally the module namespace hierarchy is not the file system, and short of just supporting #include file semantics of K&R C, I don't think we want to try and fuse the module namespace and the file system. For example, where does stdlib live in the file system? Do we want the MeTTa author to be responsible for mapping a dependency version requirement to the internal directory structure where the package manager stores packages, etc.?

So in support of goal 2, there are some conveniences, so that a user who has simple needs will usually get the behavior they want / expect, without needing to read any documentation.

• Adding the working directory as a module catalog is one of those conveniences
• Treating directories as modules, that make their contents available as sub-modules is another
• Load-on-demand during import is actually a convenience too. Note Cargo doesn't let you use something that isn't in Cargo.toml, and PkgInfo has a "strict mode" that enforces this behavior too. We probably want to require strict-mode for public publishing of MeTTa modules because it allows for dependency pre-fetching and by extension, network-free operation and strict supply-chain control.

So, I think that accessing modules from the file system in an arbitrary location is a step into an advanced non-standard use case and that warrants understanding the abstractions at work.

When I am saying that load-module! is confusing it is because load-module! and import! sound not very different.

You make a very good point. The verbs both have (linguistic) direct objects in their API forms, but those objects have been elided for brevity as MeTTa commands. load in load-module! means "load into the runner", whereas import in import! means "import into the running context". 100% agree that it's confusing.

I am open to other name choices. Personally I would go with use instead of import, and keep load. But that will involve a lot downstream change to other MeTTa users. If you have an alternative name choice instead of load (maybe register?) I am open to it.

[vsbogd]

I absolutely agree with the philosophy. And I don't think we need to make things more complex for the user. What I meant is that the questions about .. from users demonstrate that users doesn't understand how modules are different from a filesystem. I think we need to explain it clearly and implementation should follow this explanation.

First unclear question to me from the perspective of what I wrote before is whether current working directory is a module which is loaded by default?

[Necr0x0Der]

Maybe, we can use find-module or register-module or something like that instead of load-module. While load-module! might be more precise for what is happening under the hood, if the name will explicitly state that it is not loading or importing a module (into the space), it will be less confusing.

[vsbogd]

As we discussed with @luketpeterson the most confusing part to me is the fact that there are three entities under the hood. Current working directory is a "catalog", we also has a set of loaded/registered modules and script also have imported modules. import! looks up, loads/registers and imports modules from catalog. load-module! loads/registers module without affecting the catalog.

I suggested adding a MeTTa operation to modify catalog. In particular to allow user choosing which default catalog he wants to use: current working directory or some specific path in a filesystem. Or in the case above adding parent directory as a part of the catalog. But Luke explained that he prefer module registration because registering module doesn't affect other metta scripts in runtime in a way which modifying the catalog does. Modifying the catalog is to invasive and it is the reason why Luke avoided of catalog modification by the user.

It is also worth to mention that issue with loading modules from a parent directory can be resolved using a command line argument to add the parent directory into a catalog. For example Rust REPL has an argument -i (or --include-paths) which allows adding .. as an addition include path and with it !(import! &self sample) works without preconditions. @tanksha if you use Rust REPL you can solve the issue in this way. metta.py doesn't have such argument but it also can be added.

[vsbogd]

To summarize, I am not sure whether we need register-module operation. On the one hand this is useful, on the other it can be covered by catalog manipulations. If we don't want implementing catalog manipulation as a MeTTa operation then register-module is useful.

[vsbogd]

Luke, I am happy to merge (if nobody has objections) but we need to rename load-module! into register-module! or add-module!.

[luketpeterson]

Luke, I am happy to merge (if nobody has objections) but we need to rename load-module! into register-module! or add-module!.

Thanks. Done in 41419dc I went with register-module! because I feel like "add` has some of the same issues, ie. "add to what?"

[vsbogd]

Thanks Luke, I have approved. @Necr0x0Der do you have any objections?