Installation instructions for ansible-collection-to-dhall?

[daniels]

Thanks for an interesting project. I began examining Dhall to hopefully be able to get rid of my (global scope polluting) ansible variables by generating yaml where ansible mostly see static values, but quickly found out that the type system wouldn't easily let me model a list of tasks. It would be great if I could use dhall-ansible to make it possible. But I would need some collections that are not currently in your repos.

I attempted to install ansible-collection-to-dhall to try it out, but had a hard time getting it to work. It would be great if there were installation instructions (and maybe a bit more elaborate usage instructions as well) in the README.

Here is a description of what I tried and where I failed, and what eventually worked for me.

I am not familiar with cabal and shake and was hoping I could use the provided default.nix to get an environment where the commands just worked. First I attempted to create my own shell.nix and reference the default.nix from this repo:

let
  pkgs = import <nixpkgs> {};
  dhall-ansible = pkgs.fetchFromGitHub {
    owner =  "softwarefactory-project";
    repo = "dhall-ansible";
    rev = "5b5008d6dac9fef588cc5ac5deb969e7ec1d55bf";
    sha256 = "130ikhk3zgkiycn6cd2lwk2zg18rimflr4j1k63kd0xd728w7hls";
  };
  ansible-collection-to-dhall = 
    pkgs.callPackage (dhall-ansible + "/ansible-collection-to-dhall") {};
in
  pkgs.mkShell {
    buildInputs = [ ansible-collection-to-dhall ];
  }

It builds a lot of the dependencies but ultimately fails with bash: /nix/store/nzv3m73yi4372wqwk1nh12d1cina6is5-ghc-8.10.7-with-packages: Is a directory. This is probably because I don't understand nix enough.

I then manually cloned the repository and called nix-shell from the ansible-collection-to-dhall directory, and it worked. Good! (Though I'd still prefer to be able to use it from my own shell.nix, to better integrate it in my workflows.)

I had cabal in the shell I was dropped into and could use the command from the README to generate the Dhall-files, pointing it at my nix-installed ansible-doc. Great!

It took some time (I used community.general ...) and generated a lot of dhall-files. But when exploring the generated tree, I noticed that all the package.dhall-files just contained -- to be filled by shake.

So I tried to run shake and found out it wasn't present. I added it to the buildInputs in default.nix and tried again. From the root directory I got the following error:

shake: Could not find `build.ninja'
CallStack (from HasCallStack):
  errorIO, called at src/Run.hs:43:48 in main:Run

And if I instead run it inside the output dir I got this:

Shakefile.hs:6:1: error:
    Could not find module ‘Development.Shake’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
6 | import Development.Shake
  | ^^^^^^^^^^^^^^^^^^^^^^^^

Shakefile.hs:7:1: error:
    Could not find module ‘ShakeFactory’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
7 | import ShakeFactory
  | ^^^^^^^^^^^^^^^^^^^

Shakefile.hs:8:1: error:
    Could not find module ‘ShakeFactory.Dhall’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
8 | import ShakeFactory.Dhall
  | ^^^^^^^^^^^^^^^^^^^^^^^^^

I had no idea what I could do to solve any of these. But I found a notice in Shakefile.hs in the output directory that said I was supposed to run podman run -it --rm -v $(pwd):/data:Z quay.io/software-factory/shake-factory. I tried this (with docker instead of podman) and it downloaded quite a lot of things, but when run from inside the output dir it did build the package.dhall files.

So I guess I'm good to go now. But I'm not sure I did this in the best way, and I'm not sure how to integrate it in my nix/ansible/ansible-dhall workflow.

[TristanCacqueray]

Hi, this is a great feedback!

For the ghc-8.10.7-with-packages: Is a directory issue, I think this is happening because the current default.nix provides a shell derivation when called from nix-shell, thus you end up building a nix-shell using another nix-shell as a build input, and this is not possible. The issue is likely caused by the last if nixpkgs.lib.inNixShell then shellDrv else drv. This trick works great to enable using nix-shell and nix-build with a single default.nix file, but that does not seems good for what you are doing. So here is a fix you can try by changing these fetchFromGitHub attribute:

    rev = "ed0acc275b3fe70efaf4e96b645721df91b236b1";
    sha256 = "sha256-cJvQsT5SDxUzvHuNmfHrZ+ufyOzZw6qIqLbQb7Jmexc=";

Then for the shake part, I put this together to enable some customization after the initial codegen, by using some extra functions to handle schema's defaults, package record and release process. That may be a bit overkill for what you are doing, and perhaps the best solution would be to do the complete codegen with ansible-collection-to-dhall tool. Though for now I think the best would be to add a nix expression for the shake-factory project so that you can run shake directly from your shell.

Once this is done, I'll add the nix instructions to the readme for the full process. Thank you for taking the time to write this issue, this is very useful.

[TristanCacqueray]

I've added a nix expression for the shake-factory project, and here is a complete shell with all the dependencies:

let
  pkgs = import <nixpkgs> {};
  dhall-ansible = pkgs.fetchFromGitHub {
    owner =  "softwarefactory-project";
    repo = "dhall-ansible";
    rev = "ed0acc275b3fe70efaf4e96b645721df91b236b1";
    sha256 = "sha256-cJvQsT5SDxUzvHuNmfHrZ+ufyOzZw6qIqLbQb7Jmexc=";
  };
  shake-factory = pkgs.fetchFromGitHub {
    owner =  "softwarefactory-project";
    repo = "shake-factory";
    rev = "44895e0a005b6cec4722d57e3dc5c37a51b8ee0f";
    sha256 = "sha256-LaUCy6qFiB/iulWFe3vf6R4l1FsFtlLVJd5aoElNyQU=";
  };
  ansible-collection-to-dhall =
    pkgs.callPackage (dhall-ansible + "/ansible-collection-to-dhall") {};
  shake = import shake-factory;
in
  pkgs.mkShell {
    buildInputs = [ ansible-collection-to-dhall ] ++ shake;
  }

This may not be the best nix, but that seems to work. Let me know if this is useful, and I'll add it to the README.

[daniels]

Thank you so much! The nix expression worked flawlessly for me, and did exactly what I was trying to achieve. (I was a bit confused at first that I didn't get cabal in my path only to realize you did one better and added ansible-collection-to-dhall instead.) I hope you didn't take my issue as a request that you do all the work for me, but I'm nevertheless very thankful that you did.

I think it would be a great addition to the README.

[TristanCacqueray]

You are welcome, I'm happy to help, nix can be confusing sometime :-)