nixinator commented 4 years ago

Nixos and Mgmt Config might be combination for the following reasons (there are many more).

Keeping secrets out of the /nix/store
Coordinating and running changes to /etc/nixos/configuraton.nix (and import files). (on a git commit)
Setting up tests infrastructure
Integrating with Nixops

purpleidea commented 4 years ago

Happy to have a package, but no resources available to build one atm. Contributions welcome!

nixinator commented 4 years ago

I'm working on it :-). It may not be possible currently because the way dependencies are built into the project. Nixos likes to build from golang projects from source (it is a build system after all). It can handle go modules, with buildGoModule.

There a few other 'nixos' tools to gather dependencies , mgmt seems to use a combination of 'go get' , 'go submodules', which may make it a little more difficult to get it compile in a nix enviroment. I may need to pick your brains on this build process (but not much).

The main goal is to get all the source dependencies into the source tree before nix builds them, as once the build has start no network access is allowed (to help with a deterministic build :-) ).

dep2nix converts a Gopkgs.lock file into a deps.nix file which is understood by nixpkgs's go abstraction thanks to go2nix effort.

go2nix is best suited for Go apps written before dep or modules were introduced.

vgo2nix Convert go.mod files to nixpkgs buildGoPackage compatible deps.nix files.

It's a matter of unpicking you build system, and see if any of these can help. I'll have a hack to see where i can get.

nixos declarative configuration + mgmt config, might be an interesting combination.

purpleidea commented 4 years ago

If this would make your life easier (and ours too perhaps)

We like building everything from git master to keep things fresh for now, but for packaging it's an issue. So I wanted to extend (via shell script I suppose) a "tag" script that saves a lock file with the currently used dep versions. So that it can be restored if you want to build a specific tag.

You can use the standard golang module lock files if you'd like, they just need to get committed and stored in a subdir and not the root.

If you can contribute that, it would be appreciated! I think it would then make building your nix package much easier. Ping if you'd like more details.

nixinator commented 4 years ago

Ping!!! :-). Well if there is a coherent way to make the build more 'nix' friendly, then it's worth exploring. If it builds in nix/nixos, then would you be willing to add a default.nix and shell.nix (and maybe even a flake.nix) to the root of the project??

Nix is a bit alien when you first see it but....as they say in The Hitch Hikers Guide to the Galaxy... DON'T PANIC!.

as example, ipfs is buildable with the following nix expression.

https://github.com/NixOS/nixpkgs/blob/d1183332d224cbbdd46e709d20753efe6ff1ff05/pkgs/applications/networking/ipfs/default.nix

it builds

https://github.com/ipfs/go-ipfs/tree/v0.6.0

can this project be laid out like this and avoid the 'make deps' stage all together?

I'll working on fixing up the scripts and see if can get a compile with

'nix-shell -p go git'

go version go1.14.8 linux/amd64

(is 1.14.8 too new)

purpleidea commented 4 years ago

default.nix and shell.nix (and maybe even a flake.nix) to the root of the project

Not allergic to it AFAICT. I don't know enough about nix. If the files need to be in the root for nix to work, then it's probably okay. Otherwise they can go in misc/ happily.

can this project be laid out like this and avoid the 'make deps' stage all together?

How do you avoid it? Are you still going to work on the pinning of modules thing?

(is 1.14.8 too new)

Probably not. Just bumped some deps to 1.13 as well. Tests also run on 1.14 -- We'll see if I missed anything shortly.

nixinator commented 4 years ago

Yes, I'm going to look at pinning etc when i get my head around the build process.

nix likes shell script files to shebang to be portable friendly.

https://www.cyberciti.biz/tips/finding-bash-perl-python-portably-using-env.html

so changing

!/bin/bash

to

!/usr/bin/env bash

some projects refuse to do this, if your not adverse to it , i can make a PR.

Also, would you say setting CGO_ENABLE=0 (to output a static binary) a problematic thing with mgmt.

purpleidea commented 4 years ago

!/usr/bin/env bash

some projects refuse to do this, if your not adverse to it , i can make a PR.

I'll make you a deal... Once the go pinning stuff is merged, I'll accept or write the patch to switch this stuff for you if you like.

Also, would you say setting CGO_ENABLE=0 (to output a static binary) a problematic thing with mgmt.

Is this what you want? https://github.com/purpleidea/mgmt/blob/76ede10e0a0433d8aae6b3b4e132ca9dcce5ca75/Makefile#L159

nixinator commented 4 years ago

again, it's just nix weirdness, where golang is trying to install a library into a the readonly /nix/store

Building: mgmt, os/arch: linux-amd64, version: 0.0.21-92-ge99dd74-dirty... go build runtime/cgo: copying /run/user/1000/go-cache/a8/a8c43676c0bd05331491ea4b76385e1daa149d053f9a1f55a25589358611653c-d: open /nix/store/y2nrzzz2yz7k36xvwr7bbfx76cb8jg16-go-1.14.8/share/go/pkg/linux_amd64/runtime/cgo.a: read-only file system

make: *** [Makefile:175: build/mgmt-linux-amd64] Error 1

setting CGO_ENABLED=0 seems to clear that up for now...

Stuff like this is probably why developers run a mile from nix when they first see it, then run back when they realise how handy it is.

There seems to be a number of weird and wonderful ways to do dependency management with golang, and i'm just trying to see what approach works well with nix. Doing a build on ubunutu bionic maybe at least look what a sane build looks like.

I may accept this deal ... however.

https://www.youtube.com/watch?v=WpE_xMRiCLE

;-)

purpleidea commented 4 years ago

it's just nix weirdness

We can always add a new build target for nix if needed. But maybe try setting GOPATH to something totally writable and see if that helps.

There seems to be a number of weird and wonderful ways to do dependency management with golang

For your patch it should probably be a bit of bash combined with the new standard go modules.

I may accept this deal ... however.

I'm not wearing your clown shoes.

nixinator commented 4 years ago

Ok, I have mgmt compiling and runnng on Nixos within a Nixshell. This is the first step to getting a full nixos package done. :-)

I had to do the following things.

patch the shebang !#/bin/bash to a portable #!/usr/bin/env bash

to avoid

go build runtime/cgo: copying /run/user/1000/go-cache/a8/a8c43676c0bd05331491ea4b76385e1daa149d053f9a1f55a25589358611653c-d: open /nix/store/y2nrzzz2yz7k36xvwr7bbfx76cb8jg16-go-1.14.8/share/go/pkg/linux_amd64/runtime/cgo.a: read-only file system

make: *** [Makefile:175: build/mgmt-linux-amd64] Error 1

I had to remove the -i from

time env GOOS=${GOOS} GOARCH=${GOARCH} go build -i -ldflags=$(PKGNAME)="-X main.program=$(PROGRAM) -X main.version=$(SVERSION) ${LDFLAGS}" -o $@ $(BUILD_FLAGS); \

this seems to be related to this

https://github.com/NixOS/nixpkgs/issues/52706

and upstream here

https://github.com/golang/go/issues/24674

cat shell.nix
with import <nixpkgs> {};

pkgs.mkShell rec {

  name = "mgmtconfig";

  buildInputs = with pkgs; [
    go git pkgconfig augeas libvirt libxml2 
  ];

  shellHook = ''

    mkdir $HOME/gopath
    export GOPATH=$HOME/gopath
    export GOCACHE=$HOME/go-cache
    export CGO_ENABLED=1

    [ -z "$GOPATH" ] && mkdir ~/go/ || mkdir -p $GOPATH/src/github.com/purpleidea/
    cd $GOPATH/src/github.com/purpleidea/ || cd ~/go/
    git clone --recursive https://github.com/purpleidea/mgmt/
    cd $GOPATH/src/github.com/purpleidea/mgmt/ || cd ~/go/src/github.com/purpleidea/mgmt/

    export PATH=$PATH:$HOME/gopath/bin

     for i in misc/*.sh
    do
        substituteInPlace $i \
        --replace "#!/bin/bash" "#!/usr/bin/env bash"
    done

    #go get golang.org/x/tools/...
    go get -v -d ./...
    go get github.com/blynn/nex                             # for lexing
    go get golang.org/x/tools/cmd/goyacc                    # formerly `go tool yacc`
    go get golang.org/x/tools/cmd/stringer                  # for automatic stringer-ing
    go get golang.org/x/lint/golint                         # for `golint`-ing
    go get golang.org/x/tools/cmd/goimports         # for fmt
    go get github.com/tmthrgd/go-bindata/go-bindata # for compiling in non golang files
    go get github.com/dvyukov/go-fuzz/go-fuzz               # for fuzzing the mcl lang bits

    TMPDIR=/tmp

  '';

}

to run

nix-shell
make

./mgmt version
NAME:
   mgmt - next generation config management

USAGE:
   mgmt [global options] command [command options] [arguments...]

VERSION:
   0.0.21-98-g76ede10-dirty

COMMANDS:
   run, r     run
   deploy, d  deploy
   get, g     get
   help, h    Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --license      prints the software license (default: false)
   --help, -h     show help (default: false)
   --version, -v  print the version (default: false)

dantefromhell commented 2 years ago

FYI I've started work on a nix derivation using the buildGoModule builder and the following nix expression

  # Currently pinned to cloned repo owned by 'dantefromhell' to fix some build issues.
  mgmt-owner   = "dantefromhell";
  mgmt-rev     = "7e90b0da1c93c3dbfda5c45bc74f14edc085623c";
  mgmt-sha256  = "08qj0nrfy1f665k477zzhrv2qr6yw61j2klfjj08lbxr54i1nlh4";   # Computed using 'nix-prefetch-git'
  mgmt-vsha256 = "sha256-K7YUW68bc+jbM3fanfe/LEy8WpmVHwCDukk6WBQns18=";
#  mgmt-vsha256 = pkgs.lib.fakeSha256;

  mgmt = pkgs.buildGoModule {
    name = "mgmt-${mgmt-rev}";

    src = pkgs.fetchFromGitHub {
      owner  = mgmt-owner;
      repo   = "mgmt";
      rev    = mgmt-rev;
      sha256 = mgmt-sha256;
    };
    # Use 'pkgs.lib.fakeSha256' to create error message that displays correct value.
    vendorSha256 = mgmt-vsha256;

    nativeBuildInputs = [ pkgs.pkg-config ];
    buildInputs = [ 
      pkgs.augeas
      pkgs.libpcap
      pkgs.libvirt
      pkgs.libxml2
      pkgs.ragel
    ];
  };

Some patches were needed to improve dependency handling via go modules, see https://github.com/dantefromhell/mgmt/tree/nixos-packaging

~~Currently wrangling nix special behaviour around library paths, build process is not able to find dependencies like libvirt and libxml2.~~ UPDATE: This was me being confused about buildInputs vs nativeBuildInputs.

Next issue is figuring out how to dynamically create the bindata.go file [1]. It seems buildGoModule is not using make on the inside.

[1] https://github.com/purpleidea/mgmt/blob/master/lang/funcs/Makefile

dantefromhell commented 2 years ago

OK, after a bunch of wranling I came up with this

  # A lexer package (nex) is required as buildInput for mgmt to generate go
  # code during compilation process. Management via 'go.mod' is not working
  # so let's build it first.
  go-nex = pkgs.buildGoPackage {
    name = "go-nex";
    src = pkgs.fetchFromGitHub {
      owner  = "blynn";
      repo   = "nex";
      rev    = "1a3320dab988372f8910ccc838a6a7a45c8980ff";
      sha256 = "00wckdllcnv9bc35wx1f7mxb9c3x35bfxbjgixwn0krlgxbn9lhf";
    };
    goPackagePath = "https://github.com/blynn/nex";

    nativeBuildInputs = [ pkgs.gotools ];

    # From tacky/README: "Tacky is a tool to help with US tax forms."
    # It's unnecessary for the lexer functionality and comes with unproperly
    # handled depencies. Let's skip.
    excludedPackages = [ "tacky" ];
  };

  # Currently pinned to cloned repo owned by 'dantefromhell' to resolve some 
  # dependency issues.
  mgmt-owner   = "dantefromhell";
  mgmt-version = "0.0.22-41-g7e90b0d";
  mgmt-rev     = "7e90b0da1c93c3dbfda5c45bc74f14edc085623c";
  mgmt-sha256  = "08qj0nrfy1f665k477zzhrv2qr6yw61j2klfjj08lbxr54i1nlh4";   # Computed using 'nix-prefetch-git'
  # Use 'pkgs.lib.fakeSha256' to create error message that displays correct value.
  mgmt-vsha256 = "sha256-K7YUW68bc+jbM3fanfe/LEy8WpmVHwCDukk6WBQns18=";

  mgmt = pkgs.buildGoModule {
    name = "mgmt";
    version = mgmt-version;

    src = pkgs.fetchFromGitHub {
      owner  = mgmt-owner;
      repo   = "mgmt";
      rev    = mgmt-rev;
      sha256 = mgmt-sha256;
    };
    vendorSha256 = mgmt-vsha256;
    nativeBuildInputs = [
      go-nex
      pkgs.go-bindata
      pkgs.gotools
      pkgs.pkg-config
      pkgs.ragel
    ];
    buildInputs = [
      pkgs.augeas
      pkgs.libpcap
      pkgs.libvirt
      pkgs.libxml2
    ];
    preBuild = ''
#TODO: Specific path for 'dantefromhell' fork, because of non-upstreamed
#      https://github.com/dantefromhell/mgmt/commit/299ea4e16322d8b01c2ee8c7ba9f464c49156d43
      substituteInPlace lang/Makefile        --replace "/usr/bin/env bash"   ${pkgs.bash}/bin/bash
      substituteInPlace lang/types/Makefile  --replace "/usr/bin/env bash"   ${pkgs.bash}/bin/bash
      substituteInPlace lang/types/Makefile  --replace "unset GOCACHE && "   ""
      substituteInPlace misc/header.sh       --replace "/usr/bin/env bash"   ${pkgs.bash}/bin/bash
      cat lang/Makefile
      cat lang/types/Makefile
      make -C lang
      make -C lang/funcs
      go run `find lang/funcs/funcgen/ -maxdepth 1 -type f -name '*.go' -not -name '*_test.go'` -templates=lang/funcs/funcgen/templates/generated_funcs.go.tpl
    '';

    ldflags = [
      "-s -w"
      "-X main.program=mgmt"
      "-X main.version=${mgmt-version}"
    ];

    # Exclude tests that dont compile
    excludedPackages = [ "test/shell" ];

    # Disable check phase - it tries to connect to a docker image
    doCheck = false;
  };

Haven't tested further than a

$ mgmt --help
NAME:
   mgmt - next generation config management

USAGE:
   mgmt [global options] command [command options] [arguments...]

VERSION:
   0.0.22-41-g7e90b0d

COMMANDS:
   run, r     Run code on this machine
   deploy, d  Deploy code into the cluster
   get, g     Download code from the internet
   help, h    Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --license      prints the software license (default: false)
   --help, -h     show help (default: false)
   --version, -v  print the version (default: false)

but we're off to a start...

Next steps

[ ] Some real-life testing w/ MCL definitions
[ ] Figuring out how to commit to

purpleidea commented 2 years ago

@dantefromhell I'd like to apologize for not replying earlier. I haven't been receiving github notifications. It's a bug they had once with my mailserver but it is apparently back. If there's anything I'm blocking, please lmk and I can patch in mgmt. Cheers

dantefromhell commented 2 years ago

@purpleidea No rush, I've only get to work on it every now and than myself.

I believe I got some possible upstream changes related to #693 worked out in the process of creating the NixOS build, but also broke some of the automated testing along the way [1].

I still need into fixing those tests (might have done some other breaking changes in my fork too). In case any of this is useful for you, please re-use as it makes sense. And once I have capacity to fix the testing you can check what of the changes make sense upstream.

[1] https://github.com/dantefromhell/mgmt/actions

nixinator commented 2 years ago

I'm super busy with my startup right now, but i still got a place for mgmt-config in my heart!

purpleidea commented 2 years ago

@dantefromhell I quite recentl updated a bunch of go mod stuff so everything builds again. This is coincidentally around the exact same time you made your comment AFAICT.

LMK if rebasing to latest master helps, and anything else, please let me know and I'll patch or merge.

urandom2 commented 2 years ago

hope I am not stepping on your toes @dantefromhell with my pr, but thank you for your inspiration! I built mine from the ground up, so there are a few things missing from your latest version, thus please feel free to test, contribute, or give feedback, as I did not do much more than get it to compile and run a few basic commands to ensure it actually did compile.

@purpleidea, any chance we can get a release tagged, so that we can have a real version like 0.0.22?

dantefromhell commented 2 years ago

@urandom2 No toe-stepping on my end. I'm actually super happy about your PR. I haven't learned that aspect of NixOS yet this greatly helps to simplify my flake!

purpleidea commented 2 years ago

@urandom2 You're saying getting a newly tagged release would be helpful to the nix folks? Please confirm, and then sure, I can do that.

Is there anything else in the diff for packaging that we can merge upstream here to make this easier? Do you want to push the packaging stuff upstream here?

Cheers

dantefromhell commented 2 years ago

Is there anything else in the diff for packaging that we can merge upstream here to make this easier?

@urandom2 I'm wondering how helpful it could be to have a flake.nix in this repo defining a devShell with requirements for hacking mgmt?

urandom2 commented 2 years ago

@urandom2 You're saying getting a newly tagged release would be helpful to the nix folks? Please confirm, and then sure, I can do that.

nixpkgs does not need a new release, I just prefer to package against releases, since the version strings look better and it guarantees that we are not going to ship wip features. that said, I have done this in nixos/nixpkgs#198963 because 0.0.21 is so far out of date that i could not trivially get it to build. end of the day, you should release on the cadence you are comfortable with, but yeah a new tagged release would be awesome!

Is there anything else in the diff for packaging that we can merge upstream here to make this easier? Do you want to push the packaging stuff upstream here?

generally nix is a special case, so I do not want you to bend over backwards to support us. go as a toolchain, very much requires authors to check in generated code (if you want go install to work). generally advice is to commit the code and then have a ci check for "is checked in code up to date". this would go a long way to ensuring that not just nix, but go works seamlessly. that said, I can totally understand if you are uncomfortable with this approach.

@urandom2 I'm wondering how helpful it could be to have a flake.nix in this repo defining a devShell with requirements for hacking mgmt?

a flake.nix would certainly be cool, but that is a choice for @purpleidea, as to whether and how they want to maintain dev tooling; generally nixpkgs is enough for most everybody. I would strongly caution against taking too tight a dependency on nix, since it will make go and other non-nix customers have a harder time.

purpleidea commented 2 years ago

generally nix is a special case, so I do not want you to bend over backwards to support us.

Propose anything that you think makes sense. If I don't think it's a good for mgmt I won't merge it.

go as a toolchain, very much requires authors to check in generated code (if you want go install to work). generally advice is to commit the code and then have a ci check for "is checked in code up to date".

Unfortunately we do a good deal of code generation, and I realize this makes a few things harder. Maybe we'll switch away from this in the future. But go generate as a paradigm which is allowed breaks this anyways since it isn't run on go install.

a flake.nix would certainly be cool

I have no idea what this is, but patches welcome!

purpleidea commented 2 years ago

After fixing up the mgmt make scripts , from !/bin/bash to !/usr/bin/env bash

Send patches for this if you want.

Also -i stuff I think it's deprecated now.

HTH

urandom2 commented 2 years ago

Unfortunately we do a good deal of code generation, and I realize this makes a few things harder. Maybe we'll switch away from this in the future. But go generate as a paradigm which is allowed breaks this anyways since it isn't run on go install.

yep, and no shame about code generation; the standard library does plenty and they use go generate too; the difference with mgmt is you .gitignore the generated code and thus break go install. the "fix" I am proposing is to generate code just like you are today, but remove the .gitignore lines and check those files in. (this could cause drift between the generation logic and generated code checked in, thus it is recommended to add a ci step that regenerates the code and validates nothing has changed.)

I will submit what I am thinking about as a pr, and validate that go install works, but as I am unclear on how your ci integrates, I will skip the "keep stuff in sync step", though I am happy to help as needed.

I have no idea what this is, but patches welcome!

there is something on the way, so I will open it up when I can manage to get it building.

purpleidea commented 2 years ago

the "fix" I am proposing is to generate code just like you are today, but remove the .gitignore lines and check those files in.

I understood, and while it is a good idea, I still don't love the idea of committing generated code when there isn't a major win. go install working is a non-goal for me.

purpleidea / mgmt

Package for Nixos #610

!/bin/bash

!/usr/bin/env bash

!/usr/bin/env bash