search

NixOS / nixops

NixOps is a tool for deploying to NixOS machines in a network or cloud.
https://nixos.org/nixops
GNU Lesser General Public License v3.0
1.87k stars 364 forks source link

Release 2.0 old tracking issue #1242

Open grahamc opened 4 years ago

grahamc commented 4 years ago

features to delete or move to plugins:

domenkozar commented 4 years ago

If we get first half of that list done for 2.0 I'd say that's enough :)

Nekroze commented 4 years ago

If we get first half of that list done for 2.0 I'd say that's enough :)

Down to the state back ends, I think would be fantastic! Nice list.

nixos-discourse commented 4 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update/6525/1

arianvp commented 4 years ago

moved to https://github.com/NixOS/nixops/issues/1295#issuecomment-616646047 by @grahamc

adisbladis commented 4 years ago

This is a change I'd really like to get in NixOps 2.0: https://github.com/NixOS/nixops/pull/1275.

It makes plugin development so much nicer to only work with structured data.

nixos-discourse commented 4 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-2/6808/1

PsyanticY commented 4 years ago

For backups removal, for AWS backend at least, we can use DLM instead: https://github.com/NixOS/nixops-aws/pull/17

We already been using it in prod for a while now

cc @grahamc

PsyanticY commented 4 years ago

For auto-luks as well we can use the AWS CMK instead, https://github.com/NixOS/nixops-aws/pull/20 The conflict in the 2 PR is pretty strait-forward to fix if we are interested in merging these PR

jonringer commented 4 years ago

nixops is poised to being marked vulnerable in nixpkgs https://github.com/NixOS/nixpkgs/issues/101964

EDIT: invulnerable -> vulnerable

tewfik-ghariani commented 4 years ago

Hello

Are there any plans to officially release 2.0 soon? ^^

Thanks, cc @grahamc @adisbladis

jonringer commented 4 years ago

Once https://github.com/NixOS/nixpkgs/pull/104726 gets merged and back-ported, people will have to explicitly add:

     {
       permittedInsecurePackages = [
         "python2.7-cryptography-2.9.2"
       ];
     }

for nixops to work.

adisbladis commented 4 years ago

Hello Are there any plans to officially release 2.0 soon? ^^ Thanks,

The main blockers at the moment are:

@tewfik-ghariani @grahamc (and also anyone else interested): Would you be up for a review call either this week or the next and try to push NixOps 2.0 across the finish line? There really isn't much blocking us at the moment and it's a shame that those PRs have gone unmerged for so long.

tewfik-ghariani commented 4 years ago

@jonringer I don't think that would be an issue when using nixops 2.0 as that package has been upgraded in nixops-gcp : https://github.com/nix-community/nixops-gce/blob/bc94673f9aa6abd9949ecbba86dbbec05f5f7242/pyproject.toml#L12

@adisbladis Thanks for summary! Is there a chance we can postpone the 'State Backends' changes a bit? Let's say move it to 2.1 release? I believe it's better to not rush it and test it properly and the nixops 2.0 contains many ground breaking changes that the community would like to take advantage of the soonest possible

Regarding the "Testing" PR, I agree that scheduling a review call would be the best approach to unblock the merge and the release of 2.0

cc @AmineChikhaoui

jhillyerd commented 4 years ago

Given 1.7 is somewhat broken, I'd also like to see 2.0 sooner. I expect a number of people try 1.7, realize it's out of date, come to this repo to try and use 2.0, and give up.

jonringer commented 4 years ago

@jonringer I don't think that would be an issue when using nixops 2.0 as that package has been upgraded in nixops-gcp : https://github.com/nix-community/nixops-gce/blob/bc94673f9aa6abd9949ecbba86dbbec05f5f7242/pyproject.toml#L12

Yes, but all versions of nixops in nixpkgs are based on the 1.7 python2 build. I'm aware the the current dev branch of nixops uses python3, but that means that people will be required to package this outside of the repo for them not to be broken (or inconvenienced and vulnerable)

EDIT: Since you don't think this will be an issue, I will move foward with marking the package as vulnerable on nixpkgs

mkg20001 commented 4 years ago

In light of the package being vulnerable and outdated I feel like https://github.com/NixOS/nixpkgs/pull/83548 should get finished up and merged, so people wanting to try the version 2.0 can at least do so without further fuzz (the package could also be mentioned on the readme of this repo)

calbrecht commented 3 years ago

There is more to do for the nixops-hetzner plugin than just merging NixOS/nixops-hetzner#9 to be compatible with nixops 2.0. I managed to get the subcommands info ssh and check running in calbrecht/nixops-hetzner#master. I guess deploy will work too, but would not verify yet. Also i managed to get two plugins work together within a single nixops derivation on the latest nixpkgs master in calbrecht/nixops-overlay#main/default.nix, fyi.

dotlambda commented 3 years ago

Once NixOS/nixpkgs#104726 gets merged and back-ported, people will have to explicitly add:

     {
       permittedInsecurePackages = [
         "python2.7-cryptography-2.9.2"
       ];
     }

for nixops to work.

A similar thing is the case with https://github.com/NixOS/nixpkgs/pull/111322.

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-7/11552/1

Mic92 commented 3 years ago

nix-community is switched to nixopsUnstable now due to this: https://github.com/nix-community/infra/pull/57

deepfire commented 3 years ago

This gets excitingly close!

devhell commented 3 years ago

@deepfire Right? I'm really excited too!

roberth commented 3 years ago

Some important issues around the new state handling and locking. Fixing these results in breaking changes, so must be part of this major release.

Making remote state and locking usable (non-breaking, but quite needed)

roberth commented 3 years ago

The generated option docs are missing. They have been for a long time. Might these be a victim of the plugin architecture and repo split? This is definitely a blocker.

JanCVanB commented 3 years ago

Will NixOps 2.0 support macOS?

roberth commented 3 years ago

Project board

https://github.com/NixOS/nixops/projects/4

Please add (or mention here so we can add):

talyz commented 3 years ago

The state backends and the general workflow still needs work done and decisions need to be made on how they should work. Quoting my comment on #1264:

Yeah, the legacy backend was broken by this, since it now ignores the nixAttrs state attribute. I've solved this by using the require attribute to import the deployment files from the network file. Specifying the network file is however cumbersome, since the --network flag currently expects a directory, not a file. To make it easier, I've opened #1480.

With that in place, I think we could essentially drop deployments altogether without losing granularity: instead of running nixops deploy -d my_deployment you'd get the same result by running nixops deploy -n my_deployment.nix, as long as my_deployment.nix requires the appropriate files and specifies the backend to use. Note however that this strategy doesn't work with the current legacy backend, since it doesn't allow choosing where the state is stored. A simple json file backend based on import/export, like @roberth is suggesting, would solve this.

Dropping deployments would mean people would have to split their deployments into separate networks, but I think it would reduce complexity and confusion immensely.

dzmitry-lahoda commented 1 year ago

I tried nixos-generators + terranix combo, looked great, so complicated (created/commented some issues on terranix to improve). I could be interesting if nixops to abandon to write is own Python provision layer and just go with Terraform plugins (run them). So NixOps would concentrate doing nix magic of wiring things (I wire generate image as variable into terraform and deploy new VM after change of nixos config - sure I would optimize later just use Cachix Deploy or SSH to switch config only, but that is later - may be I will mount config via terraform into VM :) ).

Also would be greate if NixOps abstracts away pull vs push deployment somehow. And use (like home manager, nixos-generator, nixos itself full end to end module system (if not yet). As of now I see it only partially so.

Oh, we used NixOps from master for a while. So it harder to handle then combo above as i feel.

UPDATE:

just made end to end example fully in nix, from nixos custom amazon images, to domain name system to vm insances and then nixos-rebuild nixos updates of nixified server nodes releases https://github.com/GoldenGateGGX/golden-gate/pull/45 .

from pain points which are different from nixops, these so can be fixed. how to connect output of terraform to nix input. so that commands which run nixos-rebuild know what to call (actually generating nixos configuration using terraform output). part of issues can be solved running nix as hooks under terraform nix -> terraform -> nix https://developer.hashicorp.com/terraform/language/resources/provisioners/syntax

another missing peace is reading terraform lock file and getting all plugins mounted (plugins can be mounted via https://github.com/nix-community/nixpkgs-terraform-providers-bin, but need reader).

and sure what is lacking, evaluating nix code druing terraform build. HCL is like 2 languages, those that evaluates into JSON spec(that what nix works well), and another one which runs on resource/data attributes to produce attributes to be used for other resources. when terraform binary starts, it has no all data, so cannot evaluate on start. so some plugs of terraform for loops and maps are used as strings. so in theory can be solved by having converter of nix to hcl functions during derivation.

> resources.dns_zone is created before domain name
resources.domain.nameservers = [for n in resources.dns_zone.ns : name => n )]

in terranix

resources.domain.nameservers = "\${[for n in resources.dns_zone.ns : name => n )]";

so this is not type checked and is injection of foreign language.

does nixops allows to evaluate nix expression based on output response from cloud?

basically, can nix have https://developer.hashicorp.com/terraform/language/expressions/references#values-not-yet-known unkown values system like?

nixos-discourse commented 1 year ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/unknown-values-propagation-in-nix-like-in-hcl/26743/1