Devshell Files Maker

• About
  • Generate
  • Validate
  • Distribute
• Instructions
  • Generating files
• Examples
  • Dogfooding
• Writing new modules
  • Nix lang
  • JSON as NIX
  • Modules
  • As function
  • As object
  • Options
  • Sharing our module
  • Document our module
• Builtin modules
• TODO
• Issues
• See also

About

Helps static file/configuration creation with Nix and devshell.

There is a bunch of ways static file/configuration are hard, this will help you generate, validate and distribute JSON, YAML, TOML or TXT.

Generate

Your content will be defined in Nix Language, it means you can use variables, functions, imports, read files, etc.

The modular system helps layering configurations, hiding complexity and making it easier for OPS teams.

Validate

Your content modules could optionally be well defined and type checked in build proccess with this same tool.

Or you could use Nix as package manager and install any tool to validate your configuration (ie integrating it with existing JSON Schema).

Distribute

Nix integrates well with git and http, it could be also used to read JSON, YAML, TOML, zip and gz files.

In fact Nix isn't a configuration tool but a package manger, we are only using it as configuration tool because the language is simple and flexible.

You can recreate files of a repository directly to your local machine by running nix develop <flake-uri> --build, example:

# copy all my dogfood to your current folder
nix develop github:cruel-intentions/devshell-files --build

With help of Nix and devshell you could install any development or deployment tool of its 80 000 packages.

Instructions

Installing Nix

curl -sSf -L https://install.determinate.systems/nix | sh -s -- install

Configuring new projects:

nix flake new -t github:cruel-intentions/devshell-files my-project
cd my-project
git init
git add *.nix flake.lock

Configuring existing projects:

nix flake new -t github:cruel-intentions/devshell-files ./
git add *.nix
git add flake.lock

Generating files:

nix develop --build

or entering in shell with all commands and alias

nix develop -c $SHELL
# to list commands and alias
# now run: menu 

Examples

Creating JSON, TEXT, TOML or YAML files

# examples/hello.nix
#
# this is one nix file
{
  files.json."/generated/hello.json".hello = "world";
  files.toml."/generated/hello.toml".hello = "world";
  files.yaml."/generated/hello.yaml".hello = "world";
  files.hcl."/generated/hello.hcl".hello   = "world";
  files.text."/generated/hello.txt" = "world";
}

Your file can be complemented with another module

# examples/world.nix
# almost same as previous example
# but show some language feature
let 
  name = "hello"; # a variable
in
{
  files = {
    json."/generated/${name}.json".baz = ["foo" "bar" name];
    toml."/generated/${name}.toml".baz = ["foo" "bar" name];
    yaml = {
      "/generated/${name}.yaml" = {
        baz = [
          "foo"
          "bar"
          name
        ];
      };
    };
  };
}

Content generated by those examples are in generated

# ie ./generated/hello.yaml
baz:
  - foo
  - bar
  - hello
hello: world

Dogfooding

This project is configured by module project.nix

# ./project.nix
{
  # import other modules
  imports = [
    ./examples/hello.nix
    ./examples/world.nix
    ./examples/readme.nix
    ./examples/gitignore.nix
    ./examples/license.nix
    ./examples/interpolation.nix
    ./examples/docs.nix
    ./examples/book.nix
    ./examples/services.nix
    ./examples/nim.nix
    ./examples/nushell.nix
    ./examples/watch.nix
  ];

  # My shell name
  devshell.name = "devshell-files";

  # install development or deployment tools
  packages = [
    "convco"
    # now we can use 'convco' command https://convco.github.io

    # but could be:
    # "awscli"
    # "azure-cli"
    # "cargo"
    # "conda"
    # "go"
    # "nim"
    # "nodejs"
    # "nodejs-18_x"
    # "nushell"
    # "pipenv"
    # "python39"
    # "ruby"
    # "rustc"
    # "terraform"
    # "yarn"
    # look at https://search.nixos.org for more packages
  ];

  # create alias
  files.alias.feat = ''convco commit --feat $@'';
  files.alias.fix  = ''convco commit --fix  $@'';
  files.alias.docs = ''convco commit --docs $@'';
  files.alias.alou = ''
    #!/usr/bin/env python
    print("Alo!") # is hello in portuguese
  '';

  # now we can use feat, fix, docs and alou commands

  # create .envrc for direnv
  files.direnv.enable = true;

  # disabe file creation when entering in the shell
  # call devshell-files instead
  # files.on-call = true;
}

This README.md is also a module defined as above

# There is a lot things we could use to write static file
# Basic intro to nix language https://github.com/tazjin/nix-1p
# Some nix functions https://teu5us.github.io/nix-lib.html
{lib, ...}:
{
  files.text."/README.md" = builtins.concatStringsSep "\n" [
    "# Devshell Files Maker"
    (builtins.readFile ./readme/toc.md)
    (builtins.readFile ./readme/about.md)
    (builtins.readFile ./readme/installation.md)
    (builtins.import   ./readme/examples.nix)
    ((builtins.import  ./readme/modules.nix) lib)
    (builtins.readFile ./readme/todo.md)
    (builtins.readFile ./readme/issues.md)
    (builtins.readFile ./readme/seeAlso.md)
  ];
}

Our .gitignore is defined like this

# ./examples/gitignore.nix
{
  # create my .gitignore copying ignore patterns from
  # github.com/github/gitignore
  files.gitignore.enable = true;
  files.gitignore.template."Global/Archives" = true;
  files.gitignore.template."Global/Backup"   = true;
  files.gitignore.template."Global/Diff"     = true;
  files.gitignore.pattern."**/.data"         = true;
  files.gitignore.pattern."**/.direnv"       = true;
  files.gitignore.pattern."**/.envrc"        = true;
  files.gitignore.pattern."**/.gitignore"    = true;
  files.gitignore.pattern."**/flake.lock"    = true;
}

And our LICENSE file is

# ./examples/license.nix
{
  # LICENSE file creation
  # using templates from https://github.com/spdx/license-list-data
  files.license.enable = true;
  files.license.spdx.name = "MIT";
  files.license.spdx.vars.year = "2023";
  files.license.spdx.vars."copyright holders" = "Cruel Intentions";
}

Writing new modules

Nix lang

Jump this part if aready know Nix Lang, if don't there is a small concise content of Nix Lang.

If one page is too much to you, the basic is:

• : defines a new function, arg: "Hello ${arg}"
• that's why we use = instaed of :, { attr-key = "value"; }
• ; instead of , and they aren't optional
• array aren't separated by , ie. [ "some" "value" ]

JSON as NIX

name	JSON	NIX
null	null	null
bool	true	true
int	123	123
float	12.3	12.3
string	"string"	"string"
array	["some","array"]	["some" "array"]
object	{"some":"value"}	{ some = "value"; }
multiline-string		''... multiline string ... ''
variables		let my-var = 1; other-var = 2; in my-var + other-var
function		my-arg: "Hello ${my-arg}!"
variable-function		let my-function = my-arg: "Hello ${my-arg}!"; in ...
calling-a-function		... in my-function "World"

Module

Modules can be defined in two formats:

As attrset, aka. object (JSON), dict (Python):

{                            #  <|
  imports = [];              #   |
  config  = {};              #   | module info
  options = {};              #   |
}                            #  <|

All those attributes are optional

• imports: array with paths to other modules
• config: object with actual configurations
• options: object with our config type definition

As function:

Functions has following arguments:

• config with all evaluated configs values,
• pkgs with all nixpkgs available.
• lib library of useful functions.
• And may receive others (we use ... to ignore them)

{ config, pkgs, lib, ... }:  #  <| function args
{                            #  <|
  imports = [];              #   |
  config  = {};              #   | module info
  options = {};              #   |
}                            #  <|

Imports

Points to other modules files to be imported in this module

{ 
  imports = [
    ./gh-actions-options.nix
    ./gh-actions-impl.nix
  ];
}

Hint, split modules in two files:

• One mostly with options, where your definition goes
• Other with config, where your information goes

It has two advantages, let share options definitions across projects more easily.

And it hides complexity, hiding complexity is what abstraction is all about, we didn't share options definitions across projects to type less, but because we could reuse an abstraction that helps hiding complexity.

Config

Are values to our options

We can set value by ourself, or use lib functions to import json/toml/text files.

{ lib, ...}:
{
  config.files.text."/HW.txt" = "Hello World!";
  config.files.text."/EO.txt" = lib.concatStringsSep "" ["48" "65" "6c" "6c" "6f"];
  config.files.text."/LR.txt" = (lib.importJSON   ./hello.json).msg; # { "msg": "Hello World!" }
  config.files.text."/LL.txt" = (lib.importTOML   ./hello.toml).msg; # msg = Hello World!
  config.files.text."/OD.txt" = lib.readFile      ./hello.txt;       # Hello World!
}

If file has no options., config. can be ommited.

And this file produce the same result

{ lib, ...}:
{
  files.text."/HW.txt" = "Hello World!";
  files.text."/EO.txt" = lib.concatStringsSep "" ["48" "65" "6c" "6c" "6f"];
  files.text."/LR.txt" = (lib.importJSON   ./hello.json).msg; # { "msg": "Hello World!" }
  files.text."/LL.txt" = (lib.importTOML   ./hello.toml).msg; # msg = Hello World!
  files.text."/OD.txt" = lib.readFile      ./hello.txt;       # Hello World!
}

Options

Options are schema definition for configs values.

Example, to create a github action file, it could be done like this:

{
  config.files.yaml."/.github/workflows/ci-cd.yaml" = {
    on = "push";
    jobs.ci-cd.runs-on = "ubuntu-latest";
    jobs.ci-cd.steps   = [
      { uses = "actions/checkout@v2.4.0"; }
      { run = "npm i"; }
      { run = "npm run build"; }
      { run = "npm run test"; }
      { run = "aws s3 sync ./build s3://some-s3-bucket"; }
    ];
  };
}

This only works because this project has another module with:

{lib, ...}:
{
  options.files = submodule {
    options.yaml.type = lib.types.attrsOf lib.types.anything;
  };
}

But if we always set ci-cd.yaml like that, no complexity has been hidden, and requires copy and past it in every project.

Since most CI/CD are just: 'Pre Build', 'Build', 'Test', 'Deploy'

What most projects really need is something like:

# any module file (maybe project.nix)
{
  # our build steps
  config.gh-actions.setup  = "npm i";
  config.gh-actions.build  = "npm run build";
  config.gh-actions.test   = "npm run test";
  config.gh-actions.deploy = "aws s3 sync ./build s3://some-s3-bucket";
}

Adding this to project.nix, throws an error undefined config.gh-actions, and command fails.

It doesn't knows these options.

To make aware of it, we had to add options schema of that.

# gh-actions-options.nix
{ lib, ...}:
{
  # a property 'gh-actions.setup'
  options.gh-actions.setup = lib.mkOption {
    default     = "echo setup";
    description = "Command to run before build";
    example     = "npm i";
    type        = lib.types.str;
  };
  # a property 'gh-actions.build'
  options.gh-actions.build = lib.mkOption {
    default     = "echo build";
    description = "Command to run as build step";
    example     = "npm run build";
    type        = lib.types.str;
  };
  # a property 'gh-actions.test'
  options.gh-actions.test = lib.mkOption {
    default     = "echo test";
    description = "Command to run as test step";
    example     = "npm test";
    type        = lib.types.str;
  };
  # a property 'gh-actions.deploy'
  options.gh-actions.deploy = lib.mkOption {
    default     = "echo deploy";
    description = "Command to run as deploy step";
    example     = "aws s3 sync ./build s3://my-bucket";
    type        = lib.types.lines;
  };
}

Or using lib.types.fluent

# gh-actions-options.nix
{ lib, ...}:
lib.types.fluent {
  options.gh-actions.options = {
    # defines a property 'gh-actions.setup'
    setup.default  = "echo setup";  #default is string
    setup.mdDoc    = "Command to run before build";
    setup.example  = "npm i";
    # defines a property 'gh-actions.build'
    build.default  = "echo build";
    build.mdDoc    = "Command to run as build step";
    build.example  = "npm run build";
    # defines a property 'gh-actions.test'
    test.default   = "echo test";
    test.mdDoc     = "Command to run as test step";
    test.example   = "npm test";
    # defines a property 'gh-actions.deploy'
    deploy.default = "echo deploy";
    deploy.mdDoc   = "Command to run as deploy step";
    deploy.example = "aws s3 sync ./build s3://my-bucket";
    deploy.type    = lib.types.lines;
  };
}

Now, previous config can be used, but it does nothing, it doesn't create yaml.

It knowns what options can be accepted as config, but not what to do with it.

The following code uses parameter config that has all evaluated config values.

# gh-actions.nix
{ config, lib, ... }:
{
  imports = [ ./gh-actions-options.nix ];
  # use other module that simplify file creation to create config file
  files.yaml."/.github/workflows/ci-cd.yaml".jobs.ci-cd.steps   = [
    { uses = "actions/checkout@v2.4.0"; }

    { run  = config.gh-actions.setup;   }  # 
    { run  = config.gh-actions.build;   }  #  Read step scripts from
    { run  = config.gh-actions.test;    }  #  config.gh-actions
    { run  = config.gh-actions.deploy"; }  # 
  ];
  files.yaml."/.github/workflows/ci-cd.yaml".on = "push";
  files.yaml."/.github/workflows/ci-cd.yaml".jobs.ci-cd.runs-on = "ubuntu-latest";
}

Now it can be imported and set 'setup', 'build', 'test' and 'deploy' configs

# any other module file, maybe project.nix
{
  imports = [ ./gh-actions.nix ];
  gh-actions.setup  = "echo 'paranaue'";
  gh-actions.build  = "echo 'paranaue parana'";
  gh-actions.build  = "echo 'paranaue'";
  gh-actions.deploy = ''
    echo "paranaue 
            parana"
  '';
}

If something that is not a string is set, an error will raise, cheking it against the options schema.

There are other types that can be used (some of them):

• lib.types.bool
• lib.types.path
• lib.types.package
• lib.types.int
• lib.types.ints.unsigned
• lib.types.ints.positive
• lib.types.ints.port
• lib.types.ints.between
• lib.types.str
• lib.types.lines
• lib.types.enum
• lib.types.submodule
• lib.types.nullOr (typed nullable)
• lib.types.listOf (typed array)
• lib.types.attrsOf (typed hash map)
• lib.types.uniq (typed set)

And lib has some modules helpers functions like:

• lib.mkIf : to only set a property if some informaiton is true
• lib.optionals : to return an array or an empty array
• lib.optionalString: to return an string or an empty string

Sharing our module

Now to not just copy and past it everywhere, we could create a git repository, ie. gh-actions

Then we could let nix manage it for us adding it to flake.nix file like

{
  description = "Dev Environment";

  inputs.dsf.url = "github:cruel-intentions/devshell-files";
  inputs.gha.url = "github:cruel-intentions/gh-actions";
  # for private repository use git url
  # inputs.gha.url = "git+ssh://git@github.com/cruel-intentions/gh-actions.git";

  outputs = inputs: inputs.dsf.lib.mkShell [
    "${inputs.gha}/gh-actions.nix"
    ./project.nix
  ];
}

Or manage version adding it directly to project.nix (or any other module file)

{
  imports = 
    let gh-actions = builtins.fetchGit {
      url = "git+ssh://git@github.com/cruel-intentions/gh-actions.git";
      ref = "master";
      rev = "46eead778911b5786d299ecf1a95c9ed4c130844";
    };
    in [
      "${gh-actions}/gh-actions.nix"
    ];
}

Document our module

To document our modules is simple, we just need to use config.files.docs as follow

# examples/docs.nix

{lib, pkgs, ...}:
{
  files.docs."/gh-pages/src/modules/alias.md".modules     = [ ../modules/alias.nix ../modules/alias-complete.nix ];
  files.docs."/gh-pages/src/modules/cmds.md".modules      = [ ../modules/cmds.nix        ];
  files.docs."/gh-pages/src/modules/files.md".modules     = [ ../modules/files.nix       ];
  files.docs."/gh-pages/src/modules/git.md".modules       = [ ../modules/git.nix         ];
  files.docs."/gh-pages/src/modules/on-call.md".modules   = [ ../modules/startup.nix     ];
  files.docs."/gh-pages/src/modules/gitignore.md".modules = [ ../modules/gitignore.nix   ];
  files.docs."/gh-pages/src/modules/hcl.md".modules       = [ ../modules/hcl.nix         ];
  files.docs."/gh-pages/src/modules/json.md".modules      = [ ../modules/json.nix        ];
  files.docs."/gh-pages/src/modules/mdbook.md".modules    = [ ../modules/mdbook.nix      ];
  files.docs."/gh-pages/src/modules/nim.md".modules       = [ ../modules/nim.nix         ];
  files.docs."/gh-pages/src/modules/nushell.md".modules   = [ ../modules/nushell.nix     ];
  files.docs."/gh-pages/src/modules/nush.md".modules      = [ ../modules/nush.nix ../modules/nuon.nix ];
  files.docs."/gh-pages/src/modules/rc.md".modules        = [ ../modules/services/rc-devshell.nix     ];
  files.docs."/gh-pages/src/modules/services.md".modules  = [ ../modules/services.nix    ];
  files.docs."/gh-pages/src/modules/spdx.md".modules      = [ ../modules/spdx.nix        ];
  files.docs."/gh-pages/src/modules/text.md".modules      = [ ../modules/text.nix        ];
  files.docs."/gh-pages/src/modules/toml.md".modules      = [ ../modules/toml.nix        ];
  files.docs."/gh-pages/src/modules/watch.md".modules     = [ ../modules/watch           ];
  files.docs."/gh-pages/src/modules/yaml.md".modules      = [ ../modules/yaml.nix        ];
}

We could also generate a mdbook with it

```nix # examples/book.nix {lib, ...}: let project = "devshell-files"; author = "cruel-intentions"; org-url = "https://github.com/${author}"; edit-path = "${org-url}/${project}/edit/master/examples/{path}"; in { files.mdbook.authors = ["Cruel Intentions <${org-url}>"]; files.mdbook.enable = true; files.mdbook.gh-author = author; files.mdbook.gh-project = project; files.mdbook.language = "en"; files.mdbook.multilingual = false; files.mdbook.summary = builtins.readFile ./summary.md; files.mdbook.title = "Nix DevShell Files Maker"; files.mdbook.output.html.edit-url-template = edit-path; files.mdbook.output.html.fold.enable = true; files.mdbook.output.html.git-repository-url = "${org-url}/${project}/tree/master"; files.mdbook.output.html.no-section-label = true; files.mdbook.output.html.site-url = "/${project}/"; files.gitignore.pattern.gh-pages = true; files.text."/gh-pages/src/introduction.md" = builtins.readFile ./readme/about.md; files.text."/gh-pages/src/installation.md" = builtins.readFile ./readme/installation.md; files.text."/gh-pages/src/examples.md" = builtins.import ./readme/examples.nix; files.text."/gh-pages/src/modules.md" = "## Writing new modules"; files.text."/gh-pages/src/nix-lang.md" = builtins.readFile ./readme/modules/nix-lang.md; files.text."/gh-pages/src/json-nix.md" = builtins.import ./readme/modules/json-vs-nix.nix lib; files.text."/gh-pages/src/module-spec.md" = builtins.readFile ./readme/modules/modules.md; files.text."/gh-pages/src/share.md" = builtins.readFile ./readme/modules/share.md; files.text."/gh-pages/src/document.md" = builtins.import ./readme/modules/document.nix; files.text."/gh-pages/src/builtins.md" = builtins.readFile ./readme/modules/builtins.md; files.text."/gh-pages/src/todo.md" = builtins.readFile ./readme/todo.md; files.text."/gh-pages/src/issues.md" = builtins.readFile ./readme/issues.md; files.text."/gh-pages/src/seeAlso.md" = builtins.readFile ./readme/seeAlso.md; files.alias.publish-as-gh-pages-from-local = '' # same as publish-as-gh-pages but works local cd $PRJ_ROOT ORIGIN=`git remote get-url origin` cd gh-pages mdbook build cd book git init . git add . git checkout -b gh-pages git commit -m "docs(gh-pages): update gh-pages" . git remote add origin $ORIGIN git push -u origin gh-pages --force ''; } ```

And publish this mdbook to github pages with book-as-gh-pages alias.

Builtin Modules

Builtin Modules are modules defined with this same package.

They are already included when we use this package.

• files.alias, create bash script alias
• files.cmds, install packages from nix repository
• files.docs, convert our modules file into markdown using nmd
• files.git, configure git with file creation
• files.on-call, connfigure file to created only when devshell-files command is called, not on shell start
• files.gitignore, copy .gitignore from templates
• files.hcl, create HCL files with nix syntax
• files.json, create JSON files with nix syntax
• files.mdbook, convert your markdown files to HTML using mdbook
• files.nim, similar to files.alias, but compiles Nim code
• files.nus, similar to files.alias, but runs in Nushell
• files.nush, similar to files.nus, but for subcommands
• files.services, process supervisor for development services using s6
• files.rc , WIP, process supervisor for development services using s6-rc
• files.spdx, copy LICENSE from templates
• files.text, create free text files with nix syntax
• files.toml, create TOML files with nix syntax
• files.watch, create an alias and service to run command when file changes using inotify-tools
• files.yaml, create YAML files with nix syntax

Our documentation is generated by files.text, files.docs and files.mdbook

TODO

• Add modules for especific cases:
  • ini files
  • most common ci/cd configuration
• Verify if devshell could add it as default

Issues

This project uses git as version control, if your are using other version control system it may not work.

See also

• Nix the tool
• DevShell the framework
• Digga similar project
• Makes similar project
• Nixago similar project
• Home Manager similar project (for user home configs)
• NixOS similar project (for system configs)
• Nix Ecosystem more projects using same tool
• Nix Mindmap of how nix ecosystem works
• Nix.Dev
• Nixology

Don't look at

• *nix general definition of Unix/Linux
• Nixos.com NSFW
• Nixos.com.br furniture