Support Sysl imports across repos

[anzdaddy]

Sysl definitions living in multiple independent repos will need to refer to each other.

Implement a mechanism to refer to and access external repo references. Perhaps take inspiration from go modules. Perhaps delegate the problem to bazel.

[anzdaddy]

https://gohugo.io/hugo-modules/ seems to do this by directly leveraging go's module facility. We should look into this.

[ChloePlanet]

The mechanism of Hugo Modules:

1. ModulesClient handles all the mod subcommands in package commands
2. ModulesClient which is in package modules executes go command or interacts with go.mod/go.sum.

e.g. hugo mod init: Run go mod init <path> hugo mod get: Run go mod get <path> hugo mod tidy: Rewrite go.mod(and go.sum) hugo mod graph: Print module infos hugo mod vendor: Copy all module dependencies into the _vendor directory

[anzdaddy]

I'm curious about how Hugo infers dependencies. Since go mod parses package imports, does a Hugo package need to have a .go file that imports other Hugo packages or does it do something behind the scenes to trick Go's module machinery?

[ChloePlanet]

I think what Hugo Modules does is getting non-Go type of modules' path from the config file(which is manually maintained). And leave most of the work to Go Module cause it can work well with non-Go type of modules(what you need is just a go.mod file).

Here's an example. There's go.mod file in github.com/bep/hugotestmods/mymounts and github.com/bep/hugotestmods/mypartials which imports in the config.toml.

As for adding all the dependencies. Hugo will check _vender directory first. Uses go get if it didn't find the module in the vendor dir. If neither vendor and go get didn't work, then turn to the /theme/path dir(for compatible).

[anzdaddy]

So that may not be exactly the same as what we're looking for, which is to auto-discover external dependencies in the same way that Go does by crawling all the import statements in the repo. We might need to implement a mechanism to populate go.mod.

We should also keep in mind the possibility that go modules aren't a good fit for Sysl.

[ChloePlanet]

Yep, I'll do some researches on how other dependency management tools do to see which mechanism can fit Sysl.

[ghost]

Couple of points to consider:

1) If we're going to model sysl imports like go, perhaps we should stick the import folder into .sysl/vendor? 2) Hooking a mod/download client into the parsing flow is the simplest way to work, either it happens through the afero.FS layer, or (maybe simpler) Hooking into import_foreign which is called for each import line. 3) if we wanted to support something like "sysl mod vendor" it would be simple to walk the currrent dir and grep for import lines without parsing the entire file.

[ghost]

I'm also not entirely convinced about adding the go dependancy on the sysl binary, Adding a url download client and hand rolling mods should be trivial

[anzdaddy]

I'm also not entirely convinced about adding the go dependancy on the sysl binary, Adding a url download client and hand rolling mods should be trivial

Appreciate the sentiment. I've thought along these lines myself and agree that it's worth considering. However, we shouldn't underestimate the scope of the problem. We have to think about git integration (Go fetches repos, not websites), versioning and storage management. We might also have to replicate some of go mod's behaviours, e.g. go mod tidy, which is anything but simple.

[ChloePlanet]

IMO, based on go modules can save time writing code to handle most of the parts.

Concerns and the (possible) solutions:

1. Import private repo: go modules integrates with git config to handle this
2. Import local repo: go modules solves this by using replace
3. Multi-Module Repositories: do not recommend but it can work if needed
4. Versioning: go modules will figure it out through repo's git tag. So do multi-module repos.
5. Which sysl modules command we might need and how to implement it:
   • sysl mod init: similar to go mod init
   • sysl mod tidy: walk the current dir and collect needed sysl files' paths by greping for import lines(as Jonathan said). Download the dependencies to cache and rewrite go.mod and go.sum. Or maybe a separate sysl.mod&sysl.sum would be better? Cause if anyone for any reason runs go mod in the root path, it will trim the sysl module part of go.mod&go.sum.
   • sysl mod vendor: download the dependencies to sysl vendor dir. I think .sysl/vendor is a good idea. In the parsing flow, if there's a vendor dir, then use the modules in vendor dir. Else it could download modules to cache and use them. What's more, dirs without .go files are not copied inside the vendor directory by go mod vendor
   • sysl mod clean: clean the cache.
6. The grammar of import decl:
   • In the same dir: import <filename>, e.g. import foo.sysl
   • In subdir: import /relative/path, e.g. import foo/bar.sysl
   • From different repo or files in upper-level dirs: import /path/to/repo/file, e.g. import github.com/anzx/bff/codegen/baz.sysl

[ghost]

• What would some real imports look like?

import foo.sysl -> foo.sysl in the same folder as the current file import /foo.sysl -> foo.sysl from the sysl root import github.com/anz-bank/examples/demo/bank.sysl -> examples/demo/bank.sysl from the remote? Does the module loader have some knowledge about github.com too know that github.com/anz-bank/examples is the repo path?

• How to handle name collisions?

The import syntax currently supports import filename as X.Y.Z ~mode (everything after the filename is optional), where X.Y.Z and ~mode are used by the foreign importer (i.e direct import swagger files), but could be repurosed to handle renames... import foo.sysl as X would map the X namespace to the foo.sysl module namespace.... how this is supposed to work is very much TBD though.

• Err?

Go packages import at the directory level, do we want to copy that (which means what?), or be explicit with filenames like the above examples?

[ChloePlanet]

• import github.com/anz-bank/examples/demo/bank.sysl

The module name in the go.mod/sysl.mod file would be github.com/anz-bank/examples.

• name collisions

The name collisions syntax is new knowledge to me. How about saving imports as a map[string]Module? And define Module as:

type Module struct {
    path     string
    name     string
    imports  []*Module
}

• import level

I'm also thinking about the directory level things. One of the advantages that we keep importing single file, might be we can avoid parsing unnecessary sysl files. And the good aspect of importing at the directory level, is that we can avoid importing all other files in each file if we want split one file to multiple files to keep it neat and clean and they depends on each other.

[anzdaddy]

Name collisions at the import level are not a thing right now because imports don't have names, they simply merge their elements into the model. This needs to be changed, but it will impact the design of the language and isn't strictly required to implement modules. Raise an issue (if there isn't one already) to address naming as a whole.