newbie question: how do javascript build systems like parcel.js relate to this?

[greglearns]

I've been using elm for years, but I'm new to Nix Flakes, and mkElmDerivation seems very cool!

For building elm, Tailwind, i18n, and pwa assets, I use Parcel.js to handle packaging. How would mkElmDerivation fit into that?

Thanks!

[jeslie0]

The main thing this repository provides is the ability to fetch and build the dependencies of the elm.json file and put them in the right place. The mkElmDervation nix function will compile the elm code that you have and output the result. You can then use that derivation to use the output JS code in your project.

I am not familiar with Parcel.js, but I guess that it is in charge of compiling the elm code. In that case, you want to use the mkDotElmDirectoryCmd function (https://github.com/jeslie0/mkElmDerivation/wiki/mkDotElmDirectoryCmd) to build the .elm directory. If you set the ELM_HOME environment variable to that directory in your build script, the Elm code should build correctly.

[greglearns]

That makes sense. Thanks!

Parcel includes a way to easily bundle elm (https://parceljs.org/languages/elm/) with other javascript assets to create one dist/build_that_includes_elm_and_other_javascript.js along with a dist/final_css_file.css. For instance, I have this as my package.json, which packages my elm code together with some mapping and PWA javascript code that my elm app pulls in via ports and custom-elements:

{
  "name": "elm_parcel_example",
  "version": "1.0.0",
  "description": "Build an elm SPA using parcel",
  "source": "src/index.html",
  "browserslist": "> 0.5%, last 2 versions, not dead",
  "scripts": {
    "dev": "parcel -p 1237 src/index.html",
    "dev:clean": "rm -rf .parcel-cache dist && parcel -p 1237 src/index.html",
    "pwa": "pwa-asset-generator ./assets/img/communication.png -i src/index.html -m ./manifest.webmanifest --favicon --opaque false assets/manifest && npm run format:index",
    "build": "rm -r dist_prod; npm run i18n-generate && NODE_ENV=production parcel build --no-source-maps src/index.html --dist-dir dist_prod --no-cache",
    "i18n-generate": "elm-i18next-gen --source i18n/translations.en.json --target src --overwrite",
    "i18n-generate-watch": "elm-i18next-gen --source i18n/translations.en.json --target src --overwrite --watch"
  },
  "devDependencies": {
    "@parcel/packager-raw-url": "^2.8.3",
    "@parcel/service-worker": "^2.8.3",
    "@parcel/transformer-elm": "^2.8.3",
    "@parcel/transformer-webmanifest": "^2.8.3",
    "elm": "^0.19.1-5",
    "elm-format": "^0.8.7",
    "elm-hot": "^1.1.6",
    "elm-i18next-gen": "^1.1.0",
    "elm-test": "^0.19.1-revision7",
    "node-elm-compiler": "^5.0.6",
    "parcel": "^2.8.3",
    "postcss": "^8.4.23",
    "pwa-asset-generator": "^6.3.0",
    "tailwindcss": "^3.3.2",
    "tailwindcss-rtl": "^0.9.0"
  },
  "dependencies": {
    "@github/clipboard-copy-element": "^1.1.2",
    "leaflet": "^1.9.4",
    "leaflet-geometryutil": "^0.10.2",
    "workbox-cacheable-response": "^6.5.4",
    "workbox-expiration": "^6.5.4",
    "workbox-precaching": "^6.5.4",
    "workbox-routing": "^6.1.2",
    "workbox-strategies": "^6.1.2"
  }
}

[greglearns]

As I'm trying to figure out how to build this, I get the felling that elmMkDerivation will play a part because the elm.json parts still need to be downloaded by parcel.js (even if parcel then handles combining the outputs of elm + the other javascript).

[jeslie0]

Parcel.js looks pretty cool! I will have a play with it when I have time. As for now, I think you want to use this repository to build the .elm directory early in your script. Bundle.js almost certainly calls the elm executable which should realise the required elm dependencies are populated and then try to compile your elm code. You probably want to use buildNpmPackage from nixpkgs to package the rest of your application (see here https://nixos.org/manual/nixpkgs/stable/#javascript-tool-specific).

[greglearns]

Thanks for the buildNpmPackage pointer -- I just burned a few hours trying to figure out (unsuccessfully) how to do it with other tools. Any idea where I would pull in your code in the buildNpmPackage process?

[greglearns]

I've got to admit that I'm pretty stuck. I've got the standard NPM build running well, but it always stops at the end saying that it cannot download Elm. I'm not sure how to inject the mkElmDirvation options. https://gist.github.com/greglearns/a3adf605458ed715df232d0475d7482f.js

[greglearns]

Interesting, another person is trying to do the same thing: build elm with parcel, tried to use buildNpmPackage but got stuck and had to fall back to npm2nix. His flake and conversation talking about the problems with elm trying to download itself: https://discourse.nixos.org/t/error-getaddrinfo-eai-again-github-com-when-using-nix-parcel-and-elm/29605/4 https://github.com/TheOddler/elmder/blob/main/flake.nix

It would be cool if mkElmDerivation could help out since the npm2nix stuff looks sort of ... something not good.

[jeslie0]

Are you still being blocked by the elm executable trying to be downloaded?

This repo replaces just the elm2nix part of the build process. Is the project you are working on online? I can try and take a look at it when I have some time.

[jeslie0]

I've got to admit that I'm pretty stuck. I've got the standard NPM build running well, but it always stops at the end saying that it cannot download Elm. I'm not sure how to inject the mkElmDirvation options. https://gist.github.com/greglearns/a3adf605458ed715df232d0475d7482f.js

This is raw js code for me - it isn't rendering as a gist.

[greglearns]

Oops! Here it is without the ".js" ending: https://gist.github.com/greglearns/a3adf605458ed715df232d0475d7482f

It looks like there is a solution for the elm executable trying to do a download, so the remaining issue is the elm2nix/mkElmDerivation part, if possible.

[jeslie0]

I would try setting preBuild = pkgs.mkDotElmCommand ./elm.json in the buildNpmPackage argument. That /should/ set the environment variable and build the correct .elm directory. Let me know if that works or not.

If that doesn't work, try using "prePatch" rather than "preBuild".

I also don't think your npmBuild will be used by the function. Another nix tip is to try using a let ... in ... block, rather than use a recursive attribute set. Although, if you remove dotelm you can also remove rec!

[greglearns]

Thanks so much for your help! It works with both prePatch and preConfigure (not sure why -- I'm new to this), and thanks for mentioning rec -- I copied that because someone else was doing it, and I googled it but couldn't find any explanation. Another thing I haven't been able to find is ./. means the current directory... I assume that ./.. means the parent directory?

This is the final version that works with ParcelJS + Elm:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    mkElmDerivation.url = github:jeslie0/mkElmDerivation;
  };
  outputs = { self, flake-utils, nixpkgs, mkElmDerivation }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = import nixpkgs {
            inherit system;
            overlays = [ mkElmDerivation.overlays.mkDotElmDirectoryCmd ];
          };

          elmParcelNixFix = {
            # To make Elm, Parcel and Nix work together we need do some some fixes:
            # 1. The Elm node package tries to download the binary when installing, this doesn't work in Nix's sandbox, so instead we add it to the nativeBuildPackages, set the npmFlag to not run install scripts, and alter this (https://github.com/elm/compiler/blob/047d5026fe6547c842db65f7196fed3f0b4743ee/installers/npm/bin/elm#L8-L30) script to instead of downloading the binary to just use the one we installed as the native build package
            nativeBuildPackages = [ pkgs.elmPackages.elm ];
            npmFlags = [ "--ignore-scripts" ];
            preBuild = ''
            substituteInPlace node_modules/.bin/elm \
            --replace 'var binaryPath = path.resolve' 'var binaryPath = "${pkgs.lib.getExe (pkgs.elmPackages.elm // {meta.mainProgram="elm";})}"; runCommand(); return; //'
            '';
            # 2. Generate the .elm directory 
            generateElmJsonFiles = pkgs.mkDotElmDirectoryCmd ./elm.json;
          };

          buildInputs = with pkgs; [
            elmPackages.elm
            elmPackages.elm-live
            elmPackages.elm-test
            elmPackages.elm-format
            nodejs-18_x
          ];
        in
        {
          packages.default = self.packages.${system}.frontend;
          packages.frontend = pkgs.buildNpmPackage {
            name = "parceljs-with-elm-and-nix";
            src = ./.;

            nativeBuildInputs = elmParcelNixFix.nativeBuildPackages;
            npmFlags = elmParcelNixFix.npmFlags;
            npmPackFlags = [ "--ignore-scripts" ]; # The prepack script runs the build script, which we'd rather do in the build phase.
            prePatch = elmParcelNixFix.generateElmJsonFiles;
            # preConfigure = elmParcelNixFix.generateElmJsonFiles; # seems you can do either prePatch or preConfigure (maybe others? IDK -- I'm new to this)
            preBuild = elmParcelNixFix.preBuild;

            npmDepsHash = "sha256-Lpj6we8KXfdFNEQpWxeLwX0vTPeZRV0B5oRoh9Ur+y4=";
            # npmDepsHash = lib.fakeHash;

            npmBuild = "npm run build";

            installPhase = ''
              mkdir $out
              cp -R dist/* $out/
            '';

          };

          devShells.default = pkgs.mkShell {
            inherit buildInputs;
          };

        }
      );
}

[jeslie0]

I'm glad that works, and thank you for posting the working solution. If you're happy, we can close this issue?

The buildNpmPackage definition can be found here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/node/build-npm-package/default.nix

You can see which arguments are taken and passed to what. It also passes all of its arguments to the mkDerivation call, which is useful to know.

Yes, ./. is the "current" directory which contains your flake.nix. ./.. would be the parent, but you have to make sure that the parent is still contained in the repository, otherwise nix won't be able to find it.

Each of the phases are just bash commands that are run at certain times (more info in the nixpkgs manual). As such, you could combine your current build phase with the one I added.

preBuild = elmParcelNixFix.preBuild + "; " + elmParcelNixFix.generateElmJsonFiles;,

or something to that effect.

[greglearns]

Thanks! And you can close this issue. Also, thanks for the reference to the buildNpmPackage definition -- now I can see all of the other definitions out there, and that's a big help. Sometimes it's the basics that are hard to find via google. (I'm still struggling with how to import a file (that is included in the repository) that is in the parent directory above a flake.nix file -- it's really hard to find that! Or, how to see what Nix thinks is the source that it is building... I tried nix show-derivation $(nix-instantiate | sed 's/!.*//') and then looked in the inputSrcs directories, but when you're debugging a flake, it's hard to tell when that gets updated!). Finally, thank you for making mkElmDerivation.