[RFC] New dependencies format proposal

[jaredly]

• dependencies are for javascript modules, that npm could manage. They get installed to node_modules. Their hooks (pre/postinstall etc.) are run as normal.
• esy.dependencies are for runtime ocaml dependencies, and are installed to dependencies/default/
• esy.buildDependencies are for build-time ocaml dependencies (they don't have the uniqueness requirement), and are installed to dependencies/build/
• esy.targets.[target-name].dependencies for runtime ocaml dependencies that are only to be built to a specific target, and are installed to dependencies/[target-name]/

Dependencies handled by esy that are prefixed with @opam are installed from the opam repository, and the version specification is expected to follow opam version format, although esy will fallback to converting from an npm-style version to support current versions (e.g. jbuilder: 1.0.0-beta14 is converted to jbuilder: 1.0+beta14).

We can also specify custom opam repositories with @opam:repository-id/packagename, for things like opam-cross-ios.

Alternative for destinations: Instead of installing things to dependencies/build etc, we could just install everything to esy_modules/{name}__{version} (or into a global cache) and esy b would just know to look there, via the lockfile.

{
  "dependencies": {
    "chalk": "*",
  },
  "esy": {
    "dependencies": {
      "@opam/cohttp": "*",
      "ocaml": "^4.6.0"
    },
    "buildDependencies": {
      "@opam/jbuilder": "1.0+beta14",
      "reason": "facebook/reason#esy-awesome",
    },
    "opam-repositories": {
      "opam-cross-ios": "git://github.com:ocaml-cross/opam-cross-ios/"
    },
    "targets": {
      "ios": {
        "@opam:opam-cross-ios/tsdl": "1.0",
        "@opam/cohttp": "2.3"
      }
    }
  }
}

[rauanmayemir]

I like this approach very much and I think it helps to resolve issues mentioned in #152. I have some suggestions for improving the format and things I'd like to push back on.

I'm opposed to doing "@opam:ios/tsdl" and it has to do with long term cross-target support for packages we publish, not just consume. In other words, why don't we just add target support in tsdl's package.json? This will eventually 'unify' the ecosystem and everyone would just be able to pull tsdl package that will have all the needed instructions as to how to build itself in certain targets and toolchains.

Note: I'm not opposing the idea of adding custom opam repositories. esyi could figure out on its own that our custom repository has a target-specific instructions for tsdl (without overriding default opam repository's package).

Here's the rough config example I have in mind:

{
  "name": "mysecretstartup",
  "version": "0.1.0",
  "description": "Secretstartup monorepo",
  "esy": {
    "build": [["jbuilder", "build", "-p", "secret-backend"]],
    "buildsInSource": "_build",
    "opamRepositories": {
      "internal": "git://github.com:mysecretstartup/internal-opam-repos"
    },
    "dependencies": {
      "reason": "facebook/reason#esy-awesome",
    },
    "buildDependencies": {
      "@esy-ocaml/esy-installer": "*",
      "@opam/jbuilder": "1.0+beta14",
      "ocaml": "~4.6.0",
    }
  },
  "target": {
    "ios": { // ideally, this could be a triple resolved automatically by esy
      "toolchain": "ios", // "native" by default
      "build": [["jbuilder", "build", "-x", "ios", "-p", "secret-iphoneapp"]],
      "buildsInSource": "_build",
      "opamRepositories": {
        "ios": "git://github.com:ocaml-cross/opam-cross-ios"
      },
      "dependencies": {
        "@opam/re": "*",
        "@opam/tsdl": "1.0",
        "reason": "facebook/reason#esy-awesome",
      },
      "buildDependencies": { // will be compiled in 'parent' target env
        "@esy-cross/ocaml": "~4.4.0",
        "@opam/jbuilder": "1.0+beta14"
      },
      "sandboxEnv": {
        "ARCH": "amd64",
        "SUBARCH": "x86_64",
        "PLATFORM": "iPhoneSimulator",
        "SDK": "11.2",
        "VER": "11.2"
      }
    }
  }
}

target section might as well be nested inside esy, I'm just not liking too much indentation, and logically esy section itself is a default target.

Let's take a closer look at a target config:

{
  "toolchain": "ios", // "native" by default
  "build": [["jbuilder", "build", "-x", "ios", "-p", "secret-iphoneapp"]],
  "buildsInSource": "_build",
  "opamRepositories": {
    "ios": "git://github.com:ocaml-cross/opam-cross-ios"
  },
  "dependencies": {
    "@opam/re": "*",
    "@opam/tsdl": "1.0",
    "reason": "facebook/reason#esy-awesome",
  },
  "buildDependencies": {
    "@esy-cross/ocaml": "~4.4.0",
    "@opam/jbuilder": "1.0+beta14"
  },
  "sandboxEnv": {
    "ARCH": "amd64",
    "SUBARCH": "x86_64",
    "PLATFORM": "iPhoneSimulator",
    "SDK": "11.2",
    "VER": "11.2"
  }
}

Here, sandboxEnv will only be involved in this particular build target and changing it won't bust the project cache.

buildDependencies, I believe, are perfectly addressing the concerns @jordwalke had about which deps should be built with cross-target toolchain. Build time dependencies could be compiled outside of the toolchain scope. In future, their manifests could be tweaked for build/host/target awareness.

[jaredly]

Ooh, maybe my explanation was confusing @opam:ios/tsdl meant "get tsdl from opam-cross-ios", and had nothing to do with the target. I think we're on the same page

[rauanmayemir]

@jaredly No, I wasn't confused, I understand what you meant.

But I think that if we really want to explicitly specify the source of the package, it should be consistent with how we do it everywhere else and be something like "@opam/tsdl": "opam:ios/tsdl". (though I think, esy should just assume the source prioritising opam repositories in 'LIFO' order.)

We shouldn't mess with package names because esy doesn't make any assumptions about the origin except maybe @opam scope. (@andreypopp ?)

(Sorry for nitpicking, didn't realise that remark will take our attention away from the main point 😓)

[jaredly]

I think I'd like to favor being explicit -- @opam/ means "comes from the opam-repository". Is there a downside to making the name indicate where it's coming from?

[rauanmayemir]

There's nothing wrong with @opam scope as it's used ubiquitously as part of a dependency name. But if you want to support "@opam:opam-cross-ios/tsdl", esy will have to make assumptions and strip the opam-cross-ios part. Otherwise, other packages won't be able to resolve it, fetching the actual @opam/tsdl instead. (i.e they become different packages)

[jaredly]

Yeah, I think I want them to be different packages. Do you want them not to be? Are you looking to override deep dependencies? If so, let's be explicit about that too

[rauanmayemir]

Any particular reason why you want them to be different? (my argument against that is it feels more like a source to get it from: opam/npm, git, link, opamrepos)

[jordwalke]

Great, thanks for kicking this off, @jaredly!

I'd like to come up with some more granular terms that are capable of describing everything you've proposed here. It will help us find any redundancy in concepts. The final package.json form needn't use these granular terms.

• $runtimeTarget: Target that will run the artifacts currently being installed/built for the current package.
• $hostTarget: The target the install/build process itself is running on.
• "Quarantined Dependencies": Dependencies whose build artifacts may only be perceived by the immediate dependers. esy build hides quarantined dependencies' artifacts inteligently, and esy install, takes advantage of that fact by relaxing uniqueness requirements accordingly.

$runtimeTarget changes per package install/build task, and $hostTarget remains constant throughout the process of installing/building the entire package graph.

I've commented up @jaredly's config proposal in terms of $runtimeTarget/$hostTarget: Are the comments correct?

    {
      "esy": {
        // For each of these dependencies,
        // we install/build with runtimeTarget=$runtimeTarget
        "dependencies": {
          "@opam/cohttp": "*",
          "ocaml": "^4.6.0"
        },
        // But for each of these,
        // we install/build with runtimeTarget=$hostTarget!
        // They are also "quarantined" dependencies.
        "buildDependencies": {
          "@opam/jbuilder": "1.0+beta14",
          "reason": "facebook/reason#esy-awesome",
        },
        "targets": {
          // These dependencies are only installed/built
          // *if* $runtimeTarget === "ios"
          "ios": {
            "@opam:opam-cross-ios/tsdl": "1.0",
            "@opam/cohttp": "2.3"
          }
        }
      }
    }

What would an esy config format look like if it was oriented around these granular abstractions?

Note: I'm not actually proposing the following config, but let's use it to see if we have a similar understanding of the problem.

• To customize config based on runtimeTarget, we could allow the entire "esy" config to be built up based on matching predicates in the keys. Matches would be cumulatively merged for each key evaluating to true.
• To implement buildDependencies, we allow configuring dependencies by explicitly specifying what to set their runtimeTarget to when they are installed/built.
• We allow marking dependencies as quarantined.

Insight: buildDependencies are just quarantined dependencies where we set runtimeTarget=$hostTarget.

    {
      "esy": {
        "hostTarget=*": {
          "build": "default-build-command",
          "dependencies(runtimeTarget=$runtimeTarget)": {
            "@opam/cohttp": "*",
            "ocaml": "^4.6.0"
          },
          "dependencies(runtimeTarget=$hostTarget, quarantine=true)": {
            "@opam/jbuilder": "1.0+beta14",
            "reason": "facebook/reason#esy-awesome",
          }
        },
        "runtimeTarget=ios": {
          "build": "ios-build-command",
          "dependencies(runtimeTarget=$runtimeTarget)": {
            "@opam:opam-cross-ios/tsdl": "1.0",
            "@opam/cohttp": "2.3"
          }
        }
      }
    }

Note: the key predicates also are more flexible because they let us specify dependencies based on hostTarget, not just runtimeTarget. (What if you needed some build time tools only on Mac?) The keys also allow specifying build command based on targets.

I do not think this config format is usable - but I think it correctly models the problem most accurately. Now maybe with some sugar/defaults/aliases, that config layout could actually be practical, but I'm just trying to see if I understand the problem and check that we're on the same page.

Does this cover everything? Do you prefer a different terminology?

[jordwalke]

@rauanmayemir I like being able to set the sandboxEnv on a per target basis like you demonstrated. I think that's another long discussion. The sandboxEnv ARCH env var likely needs to get reset when switching to a dependency built with runtimeTarget=$hostTarget. (We can discuss on your existing Issue)

[jaredly]

@rauanmayemir ok I think I'm fine with them being "the same" package, coming from different sources.. @jordwalke Yeah, I think we're on the same page (I'm assuming you meant "runtimeTarget=ios":, not that we're compiling on our phones). I do like the ability to potentially specify different build deps based on host machine. esy b is currently unable to quarantine runtime dependencies, right? But we expect to be able to in the future?

One thing that your proposed format leaves ambiguous is how the build-command works. If I have

"hostTarget=*" {
   "build": "default-build-command"
},
"hostTarget=darwin": {
  "build": "mac-build-command"
},
"runtimeTarget=ios": {
  "build": "ios-build-command",
}

and I'm running on a mac building for ios, which command gets run?

One thing that rauan mentioned to me was about sub-targets, e.g. ios-arm64 vs ios-amd64. These would have a separate build command, but probably not separate dependencies. One way to hide this complexity would be to just have the ios-build-command know about all possible ios architectures, and be able to build for them via a cli flag.

[jordwalke]

@jaredly

"Yeah, I think we're on the same page (I'm assuming you meant "runtimeTarget=ios":, not that we're compiling on our phones)."

Correct. Fixed the example.

esy b is currently unable to quarantine runtime dependencies, right? But we expect to be able to in the future?

Support was recently added in master, though I haven't tried it yet. It uses the sugary buildTimeDependencies field.

About merging of build commands: I see your point. hostTarget=* isn't "special" and so nothing tells it to take priority. Perhaps that's justification for having the format be something like an outer esy config which is like the default, and then with extension based on host or runtime target. This would let the default build command apply, but then be overridden by per-target config. I'd say dependencies shouldn't be clobbered but merged like JS object spread. That's more like your original proposal, but borrows some features from my ugly one (the ability to override pretty much any config in the per-target configs).

"esy": {
  "build": "default-build-command",
  "dependencies": {
    "foo": "0.0.1",
    "bar": "0.0.1"
  },
  "hostTarget=darwin": {
      "build": "mac-build-command",
      "dependencies": {
        "foo": "some-mac-fork"
      },
  },
  "runtimeTarget=ios": {
     "build": "ios-build-command",
  }
}

@rauanmayemir said that there's some precedent in Cargo for specifying per-target config and we might be able to borrow some of that syntax.

Also, our package.json could make use of esy variables to represent runtimeTarget in a single build command that is not overridden. We'd need to add support for runtimeTarget and hostTarget in esy variables.

"esy": {
  "build": "./performBuild --target=#{esy.runtimeTarget}",
  "runtimeTarget=ios": {
    ...
  }
}

[jaredly]

Yeah I did something similar to ${esy.runtimeTarget} in this gist https://gist.github.com/jaredly/c1f4ed5720777ce9ea2b082116663855

[andreypopp]

Closing this now as we can configure esy to use a different sandbox config with a different set of dependencies configured: https://esy.sh/docs/en/multiple-sandboxes.html