WIP

[ixxie]

@Kiwi - as promised, a WIP commit so you can see what I am working on.

@grahamc has been helping out, and I've been following this guide he linked on #nixops.

As per @grahamc's request, I am keeping a log of the process and the challenges I encounter through it in this gist. Main goal of this is to help the NixOps team improve their documentation, although it can be useful for the purpose of this PR.

[ixxie]

Rebased onto the volumes branch.

[ixxie]

Currently stuck, see comment on the gist: https://gist.github.com/ixxie/36c3830e931bcf8e03f86f893f79bf99#gistcomment-3362063

[ixxie]

@Kiwi - regarding nixos-infect vs alternatives, it seems there is relatively fresh work supporting builds of a NixOS image specifically for Digital Ocean - https://github.com/NixOS/nixpkgs/pull/66978

Quoting @infinisil on this Reddit thread:

To use it, include this in some default.nix file:

{
  imports = [
    <nixpkgs/nixos/modules/virtualisation/digital-ocean-image.nix>
  ];
}

then run this to build the image:

nix-build '<nixpkgs/nixos>' --arg configuration ./default.nix -A config.system.build.digitalOceanImage

So we could use a CI/CD pipeline to build such an image and host it as a release on Github, and then leverage python-digitalocean's Image submodule to allow users to build an NixOS image.

[ixxie]

Plugin now seems to work with the new architecture:

+-------------------+
| Installed Plugins |
+-------------------+
|    digitalocean   |
+-------------------+
(nixops-digitalocean-8Il532fe-py3.7) 

:tada:

[ixxie]

Next we need to fix the broken auth-token and test the new volume feature.

[ixxie]

@kshaa (going by chiiba on Freenode) reported an issue with nixos-infect when attempting to use the plugin.

Chat log:

13:33 < chiiba> While doing a NixOps deployment to DigitalOcean, it seems nixos-infect failed at `source ~/.nix-profile/etc/profile.d/nix.sh`, because `curl -L https://nixos.org/nix/install 
                | $SHELL` must've failed. Anyone ever encountered this? Deployment output: https://pastebin.com/HA9WJWpw
13:47 < chiiba> Also nix-store was missing. I manually ssh'd in, downloaded and successfully ran nix/install without errors. Could it be that there was a network error? But that would 
                probably result in a curl error and break early. :/ Odd!
13:48 < ixxie> chiiba: I actually didn't notice this, but the DO plugin for NixOps is broken in other ways at the moment
13:48 < ixxie> chiiba: I am working with DigitalKiwi to upgrade the plugin now to the latest NixOps architecture
13:50 < ixxie> chiiba: we discuss this in #nixops although I think DigitalKiwi fell asleep :D
13:50 < ixxie> chiiba: I will add your paste into my PR draft here https://github.com/Kiwi/nixops-digitalocean/pull/1
13:50 < {^_^}> Kiwi/nixops-digitalocean#1 (by ixxie, 3 days ago, open): WIP
13:51 < chiiba> Alright, cool! Is there a package I could pin or a patch I could apply to fix this? Or should I just wait for a fix? :D
13:52 < ixxie> chiiba: honestly I don't know which versions of the plugin and nixops are needed to make this work
13:52 < ixxie> chiiba: do you have a github username I can refer you by in the PR?
13:52 < chiiba> Yes, kshaa

[ixxie]

Started work on type hints to leverage mypy to figure out this issue.

Used black to format everything in accordance with NixOps style standards.

[ixxie]

Removed travis.yml since release.nix is gone, so this is not needed anymore.

[kshaa]

It seems that the issue with nixos-infect and it having issues downloading and executing https://nixos.org/nix/install is related to nixos.org moving to Netlify. Here's the announcement .

Here is the problem illustrated:

root@landing-stage:~# curl https://nixos.org/nix/install 
root@landing-stage:~# echo $?
0
root@landing-stage:~# curl https://nixos.org/nix/install -I
HTTP/1.1 301 Moved Permanently
Content-Length: 0
Date: Sat, 04 Jul 2020 11:03:59 GMT
Location: https://releases.nixos.org/nix/nix-2.3.6/install
Server: Netlify
Via: 1.1 e5b93012e2bfb81dc9846f43efd610a6.cloudfront.net (CloudFront)
X-Amz-Cf-Id: rB741Q78wroqSHlA6HF-e7mgb2upHmyII7AoD3srBeaaQUayuUAlBA==
X-Amz-Cf-Pop: FRA2-C2
X-Cache: Miss from cloudfront
Age: 988
Connection: keep-alive
X-NF-Request-ID: 95afe588-9f84-4e26-a190-30548f2d61a5-196678

root@landing-stage:~# curl -L https://nixos.org/nix/install
#!/bin/sh

# This script installs the Nix package manager on your system by
# downloading a binary distribution and running its installer script
# (which in turn creates and populates /nix).

[...]

TL;DR - NixOS/nixops::v1.7::data/nixos-infect should be patched to use the -L curl flag. P.S. elitak/nixos-infect already patched it two weeks ago.

[ixxie]

Made good headway in typing droplet.py.

The following still stumps me:

nixops_digitalocean/backends/droplet.py:145: error: Item "ResourceState[ResourceDefinition]" of "Optional[ResourceState[ResourceDefinition]]" has no attribute "public_key"
nixops_digitalocean/backends/droplet.py:145: error: Item "None" of "Optional[ResourceState[ResourceDefinition]]" has no attribute "public_key"
nixops_digitalocean/backends/droplet.py:152: error: Item "ResourceState[ResourceDefinition]" of "Optional[ResourceState[ResourceDefinition]]" has no attribute "private_key"
nixops_digitalocean/backends/droplet.py:152: error: Item "None" of "Optional[ResourceState[ResourceDefinition]]" has no attribute "private_key"
nixops_digitalocean/backends/droplet.py:160: error: Module has no attribute "ssh_keypair"
nixops_digitalocean/backends/droplet.py:198: error: "ResourceState[ResourceDefinition]" has no attribute "public_key"

[ixxie]

Testing with the following sample:

{
  resources.sshKeyPairs.ssh-key = {};

  test1 = { config, pkgs, lib, ... }: {
    deployment = {
        targetEnv = "droplet";
        droplet = {
            name = "test1";
            region = "fra1";
            size = "1gb";
            authToken = builtins.readFile ./secrets/DO_token.txt;
        };
    };

    # System packages installed
    environment.systemPackages = with pkgs; [
      openssh openssl vim tree gitAndTools.git 
    ];

    # key-based access only
    services.openssh = {
      enable = true;
      challengeResponseAuthentication = false;
      passwordAuthentication = false;
    };

    users.users.ixxie = {
      isNormalUser = true;
      group = "wheel";
      openssh.authorizedKeys.keys = lib.splitString "\\n" (builtins.getEnv "authkeys") ++ [
        "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOm2JiPs6geaZ+coOju+kpUIbaJkLOnydTGcPc+K4V5ksqkqDW2i2fPjZdV3U8Eihv+wUmyYkj5SU+Q75JYy1/0oKwWQi2SX9EqrSsK/JOryex8FmqwhKwm7+afrryILCOJyhhNGeKOm04stxY50UDSrCmOSpyX15PZnMPB6BRuWdiWi3jvGwja2+lFwtKlIJuYooBFCAE7R7buqHgduhvtoLWTh8sLRiKDo9vP7s63qyXmvCx7tY06lSD3V65rRBd6SjA8mqHQZN9RL0RgJry65HVMIE2BapniLeUJi2L32hvttstvkj2PMA0Obm+bxlimKSSXZkTRPoxC/p3tWy7 ixxie@meso"
      ];
    };
    users.mutableUsers = false;

  };
}

But deployment fails with the following error:

[ixxie@meso:~/sparklet/repos/nixops-digitalocean]$ nixops deploy -d sparklet
Traceback (most recent call last):
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/bin/nixops", line 33, in <module>
    sys.exit(load_entry_point('nixops', 'console_scripts', 'nixops')())
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/__main__.py", line 710, in main
    args.op(args)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 638, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate,
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in deploy
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1420, in run_with_notify
    f()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in <lambda>
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1220, in _deploy
    self.evaluate_active(include, exclude, kill_obsolete)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1161, in evaluate_active
    self.evaluate()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 520, in evaluate
    defn = _create_definition(name, cfg, cfg["targetEnv"])
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1749, in _create_definition
    return cls(name, nixops.resources.ResourceEval(config))
  File "/home/ixxie/sparklet/repos/nixops-digitalocean/nixops_digitalocean/backends/droplet.py", line 62, in __init__
    super().__init__(name, config)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/backends/__init__.py", line 53, in __init__
    super().__init__(name, config)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/resources/__init__.py", line 50, in __init__
    self.config = config_type(**config)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/util.py", line 167, in __init__
    setattr(self, key, _transform_value(key, value))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/util.py", line 155, in _transform_value
    typeguard.check_type(key, value, ann)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/lib/python3.7/site-packages/typeguard/__init__.py", line 622, in check_type
    format(argname, qualified_name(expected_type), qualified_name(value)))
TypeError: type of region must be str; got NoneType instead
(nixops-digitalocean-8Il532fe-py3.7) 

[ixxie]

Now stuck with some mysterious error:

[ixxie@meso:~/sparklet/repos/nixops-digitalocean]$ nixops deploy -d sparklet
Traceback (most recent call last):
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/bin/nixops", line 33, in <module>
    sys.exit(load_entry_point('nixops', 'console_scripts', 'nixops')())
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/__main__.py", line 710, in main
    args.op(args)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/script_defs.py", line 638, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate,
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in deploy
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1420, in run_with_notify
    f()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1431, in <lambda>
    self.run_with_notify("deploy", lambda: self._deploy(**kwargs))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1220, in _deploy
    self.evaluate_active(include, exclude, kill_obsolete)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1161, in evaluate_active
    self.evaluate()
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 527, in evaluate
    name, config["resources"][res_type][name], res_type
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/deployment.py", line 1749, in _create_definition
    return cls(name, nixops.resources.ResourceEval(config))
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/resources/ssh_keypair.py", line 21, in __init__
    super().__init__(name, config)
  File "/home/ixxie/.cache/pypoetry/virtualenvs/nixops-digitalocean-8Il532fe-py3.7/src/nixops/nixops/resources/__init__.py", line 45, in __init__
    if not issubclass(config_type, ResourceOptions):
TypeError: issubclass() arg 1 must be a class
(nixops-digitalocean-8Il532fe-py3.7) 

[ixxie]

Closing in favor of https://github.com/Kiwi/nixops-digitalocean/pull/2