Source dependencies between DAML projects.

[ghost]

To improve the user experience for working with multiple DAML projects, we should add the ability to declare dependencies between projects directly, instead of having users manage binary dependencies (DAR files) manually.

The idea is that the user specifies the project folder in dependencies. For example, if you have a project in /home/user/foo that you want to depend on, you would add it to the dependencies field in daml.yaml:

dependencies:
- daml-prim
- daml-stdlib
- /home/user/foo

Then when you run daml build, the compiler will fetch the dependency's DAR file for you, and rebuild the DAR if it needs to be rebuilt. (By contrast, when a user specifies a DAR file directly, nothing can be rebuilt automatically because the sources are not available.)

Limitations:

• This will only be for dependencies not data-dependencies.
• As such, the compiler should error out if the dependency is for a different SDK version. As future work, we need to figure out the best way to support a range of SDK versions for source dependencies. But this is an extra step.

Progress:

• [ ] Add a flag daml build --cached that will skip recompiling a project that doesn't need to be recompiled. This will be used when fetching the DAR file for source dependencies, and prevents unnecessary rebuilds.

• [ ] Handle project folder dependencies in init-package-db.

• [ ] Ensure errors with source dependencies are good and readable. Make sure to handle cases like: dependency has older SDK version, dependency has newer SDK version, dependency has errors, dependency doesn't exist, dependencies are circular (👎), etc.

• [ ] Update the dependencies documentation around working with DAML packages.

[cocreature]

• We almost have daml build --cached with daml build --incremental=yes. There are a couple of caveats however:
  • The main issue is that it still does something if nothing changed. That is mainly an artifact of the usecase we added it for where we were trying to speed up the case where something changed.
  • In addition to that it does not include interface files atm. There is nothing that makes that hard but it’s slow since the zip library in gzip compression seems to have a high per-file overhead.
  • https://github.com/digital-asset/daml/issues/4413 is also somewhat related.
  • It might still make sense to have an “all or nothing” mode in daml build since daml build --incremental=yes is sadly not deterministic wtr to package ids (on the other hand for dependencies that doesn’t really matter).
• If we think of this as a stepping stone towards fetching dependencies from a proper package repository, I wonder if it might make sense to just completely ignore the .daml directory of the referenced project and instead try to always recompile it somewhere in .daml/deps of the current project and only cache that. That is less convenient for references to local folders but it means that there is basically no difference between the project coming from the package repository and a local reference.
• Somewhat related to the previous point is the IDE question. If you reference a local folder it would in principle be possible to work on both of them at the same time and have changes propagate through. However, that is not super easy to accomplish and it doesn’t make sense if you reference something from a package repository so I wouldn’t focus on this for now.

[ghost]

• It might still make sense to have an “all or nothing” mode in daml build since daml build --incremental=yes is sadly not deterministic wtr to package ids (on the other hand for dependencies that doesn’t really matter).

Yes, "all or nothing" is what I was thinking for --cached.

But thinking about how the user-invoked build should also go out and figure out all the source dependencies, transitively, to prevent looping forever when a cycle is encountered ... this means it will also have to determine everything that needs to be rebuilt. So perhaps --cached is a distraction from the real problem, which is to reimplement (a very restricted version of) make. 🙃

• If we think of this as a stepping stone towards fetching dependencies from a proper package repository, I wonder if it might make sense to just completely ignore the .daml directory of the referenced project and instead try to always recompile it somewhere in .daml/deps of the current project and only cache that. That is less convenient for references to local folders but it means that there is basically no difference between the project coming from the package repository and a local reference.

I like this, I think it might be better than what I was thinking. Definitely worth trying.

Though I would still run daml build in the dependency's folder when building the dependency, and just pass a different -o. My reasoning here is to respect the build-options of the dependency, and whatever else may be relevant.

• Somewhat related to the previous point is the IDE question. If you reference a local folder it would in principle be possible to work on both of them at the same time and have changes propagate through. However, that is not super easy to accomplish and it doesn’t make sense if you reference something from a package repository so I wouldn’t focus on this for now.

Agreed ... we can figure that out later.

[cocreature]

My reasoning here is to respect the build-options of the dependency, and whatever else may be relevant.

I would just have copied the daml.yaml over for that but either way that seems like a minor detail that we can figure out later and potentially change if the other option turns out to be more convenient :slightly_smiling_face:

[garyverhaegen-da]

Slightly tangential comment: if we have source dependencies and we aim for git integration, i.e. the ability to specify a dependency as url, sha, path in daml.yaml, we may not need to develop, maintain and host a package repository at all.