Package manager for Koka libraries and/or applications

[Pauan]

Here are some possibilities:

1. Use npm. This has the benefit of being familiar to JavaScript programmers.

   However, there are some pretty major downsides:

   • npm is very slow (but npm5 and yarn help out a lot with this)

   • Every Koka application has to download and build their dependencies individually. In other words, even if a library is used by two different Koka applications, there will be no sharing. This creates a huge amount of wasted space on the harddrive.

   • It's very easy to use npm for JavaScript dependencies, but it's extremely clunky (at best) for .NET dependencies.

   • Koka libraries will need to use peerDependencies for their Koka dependencies, and Koka applications will need to use devDependencies for their Koka dependencies:

     {
     "name": "koka-library-1",
     "peerDependencies": {
      "koka-library-2": "^1.0.0"
     }
     }

     {
     "name": "koka-library-2",
     "peerDependencies": {
      "koka-library-3": "^1.0.0"
     }
     }

     {
     "name": "koka-application",
     "devDependencies": {
      "koka-library-1": "^1.0.0",
      "koka-library-2": "^1.0.0",
      "koka-library-3": "^1.0.0"
     }
     }

     Notice that koka-application had to list koka-library-1, koka-library-2, and koka-library-3 as devDependencies, even though it only intended to use koka-library-1

     In other words, all direct and transitive dependencies must be specified in Koka applications.

     npm and yarn give warnings if you forget to specify a transitive dependency, so it's not completely horrible, but it's still very inconvenient.

     And if a Koka library accidentally uses dependencies or devDependencies rather than peerDependencies, then you can easily end up with compile-time errors, dependency hell, and code duplication.

2. Use Paket. This is similar to the npm approach (with many of the same downsides), except familiar to .NET programmers rather than JavaScript programmers.

3. Use Stack / Hackage / etc. This is great for Haskell dependencies, but Koka targets JavaScript and C#, so this doesn't seem like a viable option.

4. Create a custom package manager for Koka. This solves all of the problems that npm has.

   This is actually a lot easier than it sounds: PureScript uses a very simple custom package manager which works very well. I can give more details if you want.

5. Use Nix. The benefit of Nix is that it is a very powerful package manager, which has many very useful features:

   • Package installation is atomic, and can be rolled back

   • Packages are purely functional, which means they are fully deterministic: no more situations of "it works on my computer but fails on somebody else's computer"

   • Packages are cryptographically hashed to ensure correctness, prevent tampering, and increase security

   • Packages are functions, and therefore they can be parameterized / customized in various ways, which is not possible with most other package managers (including npm)

   • Supports automatic build caching for libraries (so that libraries / applications do not need to be compiled over and over and over again): you can easily download a pre-compiled Koka library or application.

     Unlike most other caching systems, Nix works correctly because it is purely functional

   • Has a global cache which allows for sharing the same library between multiple applications (this is completely safe, even if there are malicious users using the same computer)

   • Supports multiple versions of the same library existing simultaneously

   • Can use Nix, Haskell, npm, and .NET dependencies all within the same application / library.

     You can use all existing npm and .NET libraries even if the library doesn't support Nix: it downloads them straight from the npm / NuGet registries.

   • Cross-platform: Linux and OSX are already supported.

     Windows support is more complicated: Nix does work with Cygwin, and it also works with the Windows Subsystem for Linux.

     However, I do not use Windows, so I cannot verify how easy it is to use Nix on Windows.

   Basically, Nix has already spent a large amount of time and effort solving the very difficult problem of package management.

   I think Nix sounds like a very good option. I am willing to spend some time to investigate whether Nix is viable for Koka or not.

6. Use another existing package manager, like apt, rpm, Snaps, Flatpak, 0install, etc.

   As far as I can tell, the only benefit of this approach over Nix is that certain package managers (like 0install) might have better Windows support compared to Nix.

[Pauan]

I've done some investigation into Nix.

I have successfully managed to create a Nix package for the Koka compiler.

I've also looked into the ability to import npm dependencies in a Nix package. The most promising approach is yarn2nix, but it's too early to be production ready.

I think the biggest problem with Nix is that its Windows support isn't very good.

I think that 0install is the current best cross-platform approach to package management, but Nix has the potential to be better in the long run: it just needs better Windows support.

[siraben]

@Pauan Could you share the Nix package you wrote for Koka?

[Pauan]

@siraben Hey, sorry for the delay, I only recently got access to my NixOS machine again.

The code is obviously quite old and almost certainly doesn't work, but this is what I was using years ago:

{ stdenv, fetchFromGitHub, yarn, buildYarnPackage, haskellPackages }:

#with (import <nixpkgs> {});
#with (import /home/pauan/Programming/nix/yarn2nix {});

stdenv.mkDerivation rec {
  name = "koka-0.8.0-dev";

  src = fetchFromGitHub {
    owner = "Pauan";
    repo = "koka";
    rev = "fac9020716971751c26711c3d07923e475fda6e6";
    sha256 = "0l5ikd6ppxp8gfxrj0rm2amwyp14pymba2qxhgsi0r6qmk8ydjxi";
  };

  outputs = [ "out" "lib" ];

  nodePackages = buildYarnPackage {
    inherit name src;
    packageJson = "${src}/package.json";
    yarnLock = "${src}/yarn.lock";
  };

  buildInputs = [
    yarn

    (haskellPackages.ghcWithPackages
      (haskellPackages: with haskellPackages; [
        random
        text
        parsec
        base
        containers
        directory
        process
        mtl
        alex
      ]))
  ];

  #doCheck = true;

  configurePhase = ''
    ln --symbolic "$nodePackages/node_modules" "node_modules"

    # TODO get rid of this
    HOME="$TMPDIR"
  '';

  buildPhase = ''
    env variant=release yarn run compiler
    #yarn install --frozen-lockfile

    #"$out/bin/koka" --compile --library --outdir="build" --include="lib" --target="js" "toc"
  '';

  checkPhase = ''
    yarn test
  '';

  installPhase = ''
    mkdir "$out"
    mkdir "$out/bin"
    mkdir "$lib"

    # TODO change Koka so that it can put the binary directly into the $out/bin directory
    cp "out/release/$name" "$out/bin/koka"

    cp --recursive "lib/." "$lib"
  '';

  #outputHashMode = "recursive";
  #outputHashAlgo = "sha256";
  #outputHash = "";
}

And this is how I used it in my application:

{ stdenv, koka }:

stdenv.mkDerivation {
  name = "foo-1.0.0";
  src = ./.;
  buildInputs = [ koka ];
  buildPhase = ''
    koka --compile --target=js --outdir=build --host=browser "${koka.lib}/lib/toc.kk" "src/foo.kk"
  '';
}

[SchrodingerZhu]

since there are many solver ready to use (pubgrub-rs，for example), I think it is also nice and manageable to use create a custom manager.

Also, one can consider bootstrap one solver in Koka but may be harder at current stage.

[siraben]

@Pauan I see. In that time I managed to figure out how to build koka and add it to Nixpkgs: https://github.com/NixOS/nixpkgs/pull/117447/files Looks like the build changed a lot too.

[tom-sherman]

I would like to propose a method similar to Deno: https://deno.land/manual@v1.26.2/linking_to_external_code

Basically, don't add a package manager.

See also a similar system in Go: https://go.dev/doc/modules/managing-dependencies

[xialvjun]

I just want to remind you "not use path based imports".

See https://api-extractor.com/pages/setup/configure_rollup/#:~:text=(The%20API%20Extractor,with%20that%20effort.)

Maybe it's not the package manager's duty, it's the language's.

[xialvjun]

not use path based imports is very important.

Node, Deno are running on the wrong way, on the other hand, Carogo/rust do it right.

Why do I emphasize this ? Let's see:

// package: ffmpeg
// file: ffmpeg/utils.js
export const a = xxx;
export const b = xxx;

// file: ffmpeg/index.js
import { a, b } from './utils';
export const encode = xxx;
export const decode = xxx;

// As a package, I want to make just `encode, decode` public, but `a, b` private to other users.
// But in JS, I can't do it. People may just:
import { a, b } from 'ffmpeg/utils';
// You must don't want it

What if we change JS to:

// package user can just import package index file
import * as ffmpeg from 'ffmpeg';

// they can't import other files in that package
import * as utils from 'ffmpeg/utils';
// this should be syntax error

// what if I want to make `ffmpeg/utils` public
// file: ffmpeg/index.js
export * as utils from './utils';
export const encode = xxx;
export const decode = xxx;

// the people can
import { utils } from 'ffmpeg'

So, JUST MAKE THIS SYNTAX ERROR: import * as utils from 'ffmpeg/utils';

Well, we can't change JS, but we should do Koka the right way.

Well, it's just like Java's public protected private, the last problem is "should it default private or protected". I prefer protected in Koka.

If it's default protected, then

// package: ffmpeg
// file: ffmpeg/src/utils.kk
fun a() xxx
fun b() xxx
pub fun c() xxx

// file: ffmpeg/src/lib.kk
pub fun encode()
  utils.a() // just use utils here
// if want to make utils public
pub utils

// package: my_app
// file: my_app/src/main.kk
use ffmpeg
// or
use ffmpeg.{utils, encode}
fun main()
  utils.c()

[tom-sherman]

You mentioned it in your previous comment

Maybe it's not the package manager's duty, it's the language's

Package management and module visibility are two orthogonal problems.

[xialvjun]

Well. yeah, I'll open a new issue.

[Axylos]

I've been looking through this, and I'd like to take a stab at writing a bespoke package manager. One thing to note, though, is that a "package manager" quickly expands into something more like a general build tool and registry.

Rust's cargo comes to mind as a particularly well-crafted tool that wraps the rustc compiler and deals well with dependency resolution, project management, and task runner/building.

So I would lean towards building a rough build tool, package registry, and dependency manager that calls into the existing koka compiler. That's a tall order but thinking about that stuff holistically goes a long way towards cultivating a nice dev experience.

On the other hand, building that all natively in koka right now feels impossible since the current tooling (and especially the lack of a solid unit test infrastructure from what I can tell) is still developing. My thought right now is to build a lot of that stuff in another language, e.g., Rust, and link to it via extern imports with a decent koka interface, and then gradually eat away at the foreign bits as koka continues to evolve.

Koka is a fantastic project, and it would be great to have a working package manager/testing setup to push things forward. I'm happy to start picking at this stuff if it feels halfway sane.

[woojamon]

So is there any great way to share koka code at the moment?

[tom-sherman]

You can get quite far with git submodules. Zig for example got pretty far with this approach.

[TimWhiting]

Yes this is the recommended approach for now. Use the -i option to add more include directories (you can also add compiler arguments in the vscode extension settings) and just use git submodules, monolithic repos with subdirectories for your different packages, or hard code paths to your different cloned packages for now.