importing python modules from subdirectories

[Necr0x0Der]

@luketpeterson , this is a very common blocking issue that Python modules cannot be now imported from subfolder. It should be fixed ASAP. It would also be nice to be able to import modules from parent folders (using ..), although this is not that critical

[luketpeterson]

The test that you had originally implemented will work without changes as soon as I finish up the work for https://github.com/trueagi-io/hyperon-experimental/blob/8540b90d8669484e945fb78d0d11135b68de9233/docs/modules_internal_discussion.md?plain=1#L139

to recap, !(import! &self ext_sub.ext_nested) should import ext_nested into the top-level of the runner, once the Python catalog is implemented. Currently it doesn't work.

*I renamed ext_dir to ext_nested because I didn't want to accidentally load the ext_dir mod one level up and fool myself into thinking the test was passing.

In the meantime, I have tweaked the test by adding a modules.metta file, which imports its sub-module. This can provide you a short-term work-around, as well as another test for nested MeTTa mods.

Finally, !(import! &self ext_sub:ext_nested) will also work when I raise a PR I am currently working on for recursive module loading. This will mean that the modules.metta file doesn't need to explicitly import the sub-module.

So there are 2 more tests I will add soon: !(import! &self ext_sub.ext_nested) (test Python catalog) !(import! &self ext_sub:ext_nested) (test recursive sub-module import from client)

[Necr0x0Der]

I renamed ext_dir to ext_nested

OK

In the meantime, I have tweaked the test by adding a modules.metta file, which imports its sub-module.

Hmm.... It's an interesting way to shape modules in some cases, but it's not much convenient for our case than just putting a lot of symlinks to make the structure shallow. When do you expect the catalog for Python modules to be ready?

[luketpeterson]

When do you expect the catalog for Python modules to be ready?

I am prioritizing getting the recursive sub-module loading working, which I hope to raise in a PR tomorrow. ie. !(import! &self ext_sub:ext_nested)

The Python catalog will take a bit more time because part of that job is to add C and Python bindings to the custom catalog API. So that will take me until the weekend I think.

[Necr0x0Der]

I am prioritizing getting the recursive sub-module loading working, which I hope to raise in a PR tomorrow. ie. !(import! &self ext_sub:ext_nested)

Will it work for the following case? !(import! &self prototyping:assistant_utils:data_getters:data_getter) data_getter.metta is places in prototyping/assistant_utils/data_getters folder with the content like !(import! &self data_getter_extension), where data_getter_extension is a Python module in the same folder as data_getter.metta. So, we just workaround loading Python module !(import! &self prototyping.assistant_utils.data_getters.data_getter_extension) from a subfolder with loading a metta module from a subfolder, which imports the Python module locally? Or it will not find Python module locally, when it is imported by the runner script from another folder?

[luketpeterson]

...with the content like !(import! &self data_getter_extension), where data_getter_extension is a Python module in the same folder as data_getter.metta

import! from within a module searches the module's resource directory first. So that should already work with the code in this PR.

However, the recursive sub-module loading feature will only work if each module name (delimited by ':' in the name path) is itself a valid MeTTa module. Currently a directory is not a valid module without one of several special files inside (e.g. module.metta, etc.) However unlike the situation in the code now, the module.metta doesn't need to import its sub-module itself - an empty file will be enough to designate a directory as a module.

UPDATE: I've been thinking about this and I don't see any harm in treating any directory as a module that is capable of hosting sub-modules, so I will incorporate this change into the next PR.

If you rely on the Python import system to locate the module, e.g. in site-packages or other search behavior specific to Python then the python catalog will be required.

[Necr0x0Der]

@luketpeterson , what is the status of this PR?

[luketpeterson]

@luketpeterson , what is the status of this PR?

I just raised https://github.com/trueagi-io/hyperon-experimental/pull/603 which contains all commits from this PR.

I added the test_extend_recursive_pymod test, which successfully performs this operation: !(import! &self ext_recursive:level-2:ext_nested)