Single repository with multiple modules (or repo with subdir usage)

[posborne]

Internally, we currently have a single repository that includes several components that are part of our "Common Firmware Library." These are all pretty independent pieces of code that would make good candidates for Yotta modularization. In an ideal world, we would like to keep these all in a single repository as we have found it makes maintenance simpler.

The basic layout for the repo would probably look like this:

├── module_a
│   ├── module.json
│   ├── ...
├── module_b
│   ├── module.json
│   ├── ...

Is there a way to handle this use case today? I imagine that publishing these to the registry would be one approach, but that is unlikely to happen with these modules. In the future, a private registry might be an option for us (although I understand that is not on the immediate roadmap). Here's one idea on syntax based on how it is done in pip (https://github.com/pypa/pip/pull/1215/files):

"usefulmodule": "<anything>://somwhere.git#<tag>&subdirectory=sub"

In general, I think having keyword arguments for this URL string makes sense. This is what pip and openembedded/yocto (poky) do. Currently, there is no keyword paired with tag, so I introduced the semicolon to delimit the URL from other arguments as I imagine backwards compatability being important. I have also seen ;, ?, @ and & commonly used as delimiters as they typically do not show up in tag/version names.

This features has been requested in npm but not implemented. I see the argument against not doing it, but I think there are legitimate reasons for not requiring changes be broken out to live at the top-level of repositories.

[ghost]

alternatively one could use read only subtree splits

[autopulated]

I wondered also about yotta searching for module.json files within a repository automatically, until it found one with the right name. I can't think of any obvious drawbacks of that, since there shouldn't normally be two modules with the same name.

[autopulated]

Considering implementing this, I encountered a problem: what to do about tags.

Normally when fetching a named module from a git/hg repo, yotta checks for semver-like tags on the repo (v1.2.3-something+whatever), and uses these to identify which version of the repo to use (if no semver tags are found you'll get the familiar "warning: XXX has no tagged versions, default branch will be used"

(The exception to this are if #namedtag or #namedbranch are used in the VCS url, in which case the tip of the named branch, or the exact named tag are used instead of the latest compatible semver).

If multiple modules exist in the same repository, how should tags be handled?

The simplest option would be to assume that any semver tags apply to all modules in the repository, which would require every module.json file to be updated before creating a tag.

Another would be to ignore tags in certain circumstances: for example if the last part of the repository URL path does not match the requested module name, or if a subdirectory URL query parameter is present.

Neither of these seem particularly appealing. Any ideas?

[ghost]

does npm even handle this use case? projects I'm on tend to solve it with git subtree splits (as mentioned earlier)

[autopulated]

I'm not aware of any npm support for this. In npm the answer would be "publish to the registry" I would think ;) (Or at this point, using their private registry support)

[autopulated]

I've come across various cases recently where this is seeming like a good idea again, even with the drawback that tagging isn't going to work in a straightforward way unless all the modules/targets in a repository share the same version, so I think this should be implemented.