feat: use nix flake, std, tullia and cicero

[jbgi]

std-based alternative take to #175.

[hamishmack]

This looks really good. Is there a way to pass evalSystem to everything? I wanted to see if I could run nix flake show on macOS without using remote builder:

I tried the following, but I think the cross compiled versions of the project are still missing the evalSystem somehow.

diff --git a/cells/_automation/hydraJobs.nix b/cells/_automation/hydraJobs.nix
index 5d101b8..ceb7d5c 100644
--- a/cells/_automation/hydraJobs.nix
+++ b/cells/_automation/hydraJobs.nix
@@ -4,7 +4,7 @@
 }: let
   inherit (inputs.nixpkgs) lib;
   inherit (inputs.cells.prelude.library) project extra-compilers;
-  evalSystem = "x86_64-linux";
+  evalSystem = "x86_64-darwin";

   ciProject = project.appendModule {
     inherit evalSystem;
diff --git a/cells/prelude/library.nix b/cells/prelude/library.nix
index d949a48..9ed335a 100644
--- a/cells/prelude/library.nix
+++ b/cells/prelude/library.nix
@@ -41,6 +41,7 @@
     name = "cardano-prelude";
     inherit src;
     compiler-nix-name = lib.mkDefault default-compiler;
+    evalSystem = "x86_64-darwin";
     # Cross compilation support:
     crossPlatforms = p:
       lib.optionals pkgs.stdenv.hostPlatform.isLinux (
@@ -59,7 +60,7 @@
       # deduce package names from the cabal project to avoid hard-coding them:
       projectPackageNames = builtins.attrNames (haskellLib.selectProjectPackages
         (haskell-nix.cabalProject' {
-          inherit (config) name src compiler-nix-name cabalProjectLocal;
+          inherit (config) name src compiler-nix-name evalSystem cabalProjectLocal;
         })
         .hsPkgs);
     in [

[jbgi]

@hamishmack right, forgot to pass evalSystem here: https://github.com/input-output-hk/cardano-prelude/pull/176/commits/bb6e26a8467751c64986dd9ea84446a4e2f50954 nix flake show should work now (with your local modifications).

[jbgi]

@dermetfan tried your suggestion in last commit, but it is still failing:

$ tullia run ci/hydraJobs.x86_64-linux.roots
...
[✗] error  ci/hydraJobs.x86_64-linux.roots                                                                9m11.235325358s
warning: Git tree '/home/jbgi/Dev/iohk/cardano-prelude' is dirty                                                         
warning: Using saved setting for 'allow-import-from-derivation = true' from ~/.local/share/nix/trusted-settings.json.    
warning: Using saved setting for 'extra-substituters = https://cache.iog.io' from ~/.local/share/nix/trusted-settings.jso
n.                                                                                                                       
warning: Using saved setting for 'extra-trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ='
from ~/.local/share/nix/trusted-settings.json.                                                                           
error (ignored): error: SQLite database '/home/jbgi/.cache/nix/eval-cache-                                               
v4/1553df62b97b634bce22160eed0bb4a05e9e574b5e5e752ad9c22245c16fd86e.sqlite' is busy                                      
error: flake 'git+file:///home/jbgi/Dev/iohk/cardano-prelude' does not provide attribute 'packages.x86_64-               
linux.tullia.x86_64-linux.task.ci/hydraJobs.x86_64-linux.roots.nsjail.run', 'legacyPackages.x86_64-linux.tullia.x86_64-  
linux.task.ci/hydraJobs.x86_64-linux.roots.nsjail.run' or 'tullia.x86_64-linux.task.ci/hydraJobs.x86_64-                 
linux.roots.nsjail.run'                                                                                                  
Failed to run /nix/store/i7zh5x24vsbv471zk7vadkgsa55qianz-nix-2.11.0/bin/nix build --json --no-link .#tullia.x86_64-     
linux.task.ci/hydraJobs.x86_64-linux.roots.nsjail.run: exit status 1

Also another issue is that even though I specify only one task, tullia execute all tasks... (almost crashing my machine: is there a way to limit concurrent jobs?).

[dermetfan]

There was an escaping issue in tullia. Fixed that and updated the tullia flake input.

is there a way to limit concurrent jobs?

I added a tullia task for each hydra job so you can now run them separately like this (without ci/ prefix):

nix run github:input-output-hk/tullia -- run hydraJobs.x86_64-linux.ghc924.plan-nix

[jbgi]

@dermetfan I'm now getting

$ nix run github:input-output-hk/tullia -- run x86_64-linux._automation.ciJobs.ghc924.plan-nix
warning: Using saved setting for 'allow-import-from-derivation = true' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-substituters = https://cache.iog.io' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ=' from ~/.local/share/nix/trusted-settings.json.
[✗] error  x86_64-linux._automation.ciJobs.ghc924.plan-nix                                                  12.778438199s
[E][2022-10-17T14:55:32+0000][1] void subproc::subprocNewProc(nsjconf_t*, int, int, int, int, int)():205                 
execve('/nix/store/zspimn40hjy77c6bdmk24lwam87zwmy2-x86_64-linux._automation.ciJobs.ghc924.plan-nix/bin/'x86_64-         
linux._automation.ciJobs.ghc924.plan-nix'') failed: No such file or directory                                            
[F][2022-10-17T14:55:32+0000][1] bool subproc::runChild(nsjconf_t*, int, int, int, int)():448 Launching child process    
failed                                                                                                                   
[W][2022-10-17T16:55:32+0200][802842] bool subproc::runChild(nsjconf_t*, int, int, int, int)():478 Received error message
from the child process before it has been executed                                                                       
[E][2022-10-17T16:55:32+0200][802842] int nsjail::standaloneMode(nsjconf_t*)():256 Couldn't launch the child process     
Failed to run /nix/store/6cka89h4lly1m068a2lv8i99gvp8a0x6-x86_64-linux._automation.ciJobs.ghc924.plan-nix-nsjail/bin/x86_
64-                                                                                                                      
linux._automation.ciJobs.ghc924.plan-nix-nsjail: exit status 255                                                         
4:55PM FTL starting supervisor error="running x86_64-linux._automation.ciJobs.ghc924.plan-nix: Failed to run /nix/store/6cka89h4lly1m068a2lv8i99gvp8a0x6-x86_64-linux._automation.ciJobs.ghc924.plan-nix-nsjail/bin/x86_64-linux._automation.ciJobs.ghc924.plan-nix-nsjail: exit status 255"

[dermetfan]

That's just nsjail having trouble with cgroups when it tries to actually run the task (not yet well tested). Evaluation and building worked fine so it works on cicero, as for cicero the nomad + oci runtime is used instead of nsjail.

[michaelpj]

On reflection, a high-level thought. This PR contains hundreds of lines of nix code providing lots of features, but explains nothing about how to use it or work with it. To put it another way: cardano-prelude needs a CONTRIBUTING, and if we're going to add all this stuff it desperately needs one. What is all this stuff? What is lefthook? What is nixago? What's up with this editorconfig thing? How do I get a shell? How do I update things? What's the CI setup? How do I change the compilers it's testing against? At the moment nobody will know any of this except @jbgi

Yes, that's a lot to write about, but this PR adds a lot of stuff. If we want to keep it minimal instead that would be fine but if we're going to go big IMO we have to document things.

[jbgi]

Sure. Most std stuff can be documented with links to the appropriate std doc page. I'll work on it next week.

[michaelpj]

I think that's good for maintainers of the nix code, but we also need to document things in a form that's comprehensible by people working on the Haskell code, hence CONTRIBUTING.

[jbgi]

@michaelpj added CONTRIBUTING.md in last commit (albeit rather minimal for now, but should be enough to not block any dev workflow).

[jbgi]

squashed commits, removed conventional commit enforcing, switch formatting to use fourmulu (same config as in cardano-ledger).

Should be ready to merge.

[dermetfan]

I pushed a fix for the latest tullia version, maybe squash again @jbgi (didn't want to squash and force push to not confuse anybody)

[coot]

@erikd, @newhoggy, @lehins are you ok with requiring fourmolu formatter?

[lehins]

are you ok with requiring fourmolu formatter?

@coot I encourage requiring it!

[jbgi]

bors r+