Examining loaded packages in module namespaces/environments

[ElianHugh]

In my spare time, I've been working on a rudimentary module manager ("boxingtape - grab") in the vein of yarn/npm (still extremely WIP, early stages, etc.). One thing I've been trying to implement is the drilling of modules in order to document requisite packages for installation. I have this working to an extent, the main issue being that I can't control namespace leakage, as "grab" has to be loaded in order to drill the modules in the first place.

I've tried a few hacky solutions, but I think the main issue is that I haven't been able to examine the packages in module environments. I've thought of just attaching all the modules to the global environment and then assessing them there, but the issue is then cleaning up the mess caused by that. E.g.:

eval({rlang::parse_expr(glue("box::use({moduleString})"))},
        envir = .GlobalEnv
       )

I suppose this is a rather long-winded way of asking about the ability to assess loaded packages in module namespaces and environments, or ways in which I could assess loaded namespaces in a given module. As above, I can force a module to be loaded in a specific environment, but that doesn't seem to give me access to its contents loaded packages. Is there a way of getting access to what packages they implement? I looked at box::name() which seems to point me in the right direction with regard to namespaces, but I haven't had any luck yet.

Thanks!

...Sorry about the double colon usage

Specifics

Edit: for some more background on the specifics re: package use, I'm listing all files in a local modules directory, and mapping them to box:

fullList <- list.files(
        path = glue("{here()}/modules"),
        recursive = TRUE,
        ignore.case = TRUE,
        pattern = {
            ".r$"
        }
    )

modFiles <- tools::file_path_sans_ext(
        glue(
            '../{fullList[ !grepl("grab", fullList) ] }'
        )
    )

walk(modFiles, ~ suppressMessages(screen_names(.x)))

screen_names <- function(x) {
        splitName <- stri_reverse(stri_split_fixed(stri_reverse(x), "/", n = 2)[[1]])[2:1]
        moduleString <- glue("{splitName[1]}/`{splitName[2]}`")
        eval(
            rlang::parse_expr(glue("box::use({moduleString})")),
# This isn't really necessary, but I've tried a few  
 # variations of setting an environment to call from
            envir = rlang::current_env() 
        )
    }

From here, I can use sessionInfo(), loadedNamespaces(), .packages(), etc., to see what packages are exposed. Doing so, however, means that the global environment is tainted by the function call (as the packages are still visible in loadedNamespaces wherever they are loaded from), and for the same reason gives an inacurate view of what packages a project might use. Ideally, I'd like to access the loaded namespaces of modules only, giving a more accurate view of package usage. This would also have the upside of allowing me to get the exact package dependencies of each module.

Edit 2: Sorry to bombard with edits, just trying to get my head around this! I've been able to access module environments through trial and error, but I haven't been able to assess attached packages, only modules. For instance, if I get the environment of module "grab", I can see the children of grab, but not any package that is loaded (if I assess the environment of a local object that directly loads a package, I can see the package loaded via get()).

[klmr]

I have to admit that I don’t entirely follow what you’re trying to do. I guess ultimately you want to know which packages/modules are loaded by ‘box’, right?

There’s currently no built-in, exposed mechanism for this, but I’ve now gotten several queries to make (static) analysis easier, so I guess this will happen. Related (but not the same): #105.

Modules (but currently not for packages!) are listed inside ‘box’ in the environment loaded_mods. However, I will probably also have to store packages in a similar way to accommodate #178. If this information is useful to others I’m happy to export some functions to access it outside of ‘box’.

The (currently private) function that parses box::use declarations is box:::parse_spec. Currently this only parses a single declaration, as a quoted expression. For example, to parse the declaration box::use(./a[...]), you’d use

box:::parse_spec(quote(./a[...]), '')

This returns an S3 class object containing information about the declaration. Maybe this is helpful?

[ElianHugh]

Sorry - the year caught up with me and I lost track of this a bit! Thank you for replying :)

Yes, sorry, I was interested in examining loaded packages for dependency tracking - rereading the initial post, I can see how you got confused...

I fiddled around with box:::parse_spec(quote(./a[...]), '') but I think I don't fully understand how it works. Am I correct in understanding that the returned mod_spec object doesn't show what packages are loaded by a given mod?

[klmr]

Am I correct in understanding that the returned mod_spec object doesn't show what packages are loaded by a given mod?

That’s correct, yes. The return value is an S3 object that represents a single module/package inside a use() declaration. It doesn’t even locate the actual module yet (i.e. box:::parse_spec(…) works even for non-existent modules, and doesn’t examine the module search path).

At the moment there’s no direct way to retrieve a module’s dependencies. Here’s an outline of what would need to be done:

1. parse the spec
2. locate the module (via box:::find_mod(), with a given spec)
3. parse the located module file, filter out all box::use() calls, and goto 1 for each of their arguments.

Unfortunately there’s a problem: box:::find_mod() requires a caller argument, which is used to resolve local imports (of the form ./foo). This needs to be the environment from which box::use() would have been called, so it can’t be used statically (getting a module’s environment requires actually loading it!).

Instead of box:::find_mod(), one might use box::find_in_path(), which would need to be called either with the global search path or with the current module’s path, depending on whether it’s an absolute or a relative import.

And one more complication: the “global search path” is defined as follows:

https://github.com/klmr/box/blob/d72fb0429a0074b70f68824fe348af19908d97ee/R/paths.r#L16-L19

Note that, once again, it expects a caller, which is used to insert the calling module}s path at the end of the search path (that way, both box::use(sub/mod) and box::use(./sub/mod) work inside a module to refer to a local submodule).

---

All in all this is sufficiently complex that ‘box’ should provide this functionality, … well, out of the box. The question is how the API should look like.

Something like box::static$dependencies(path), where path is the full path to a module’s main file? In particular, I’m wondering what other static analysis API would be useful to have, and move all of that into a common namespace. And ideally the whole thing should be designed forward compatible so that, once we have versioning, that information can be added without breaking backwards compatibility of the static analysis API.