Unable to destroy a Hetzner deployment

[danbst]

I want to remove deployment, but can't do that.

$ nixops destroy -d fleet
warning: are you sure you want to destroy Hetzner machine ‘jolly.roger’? (y/N) y
jolly.roger> rebooting machine ‘jolly.roger’ into rescue system
Traceback (most recent call last):
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/bin/..nixops-wrapped-wrapped", line 941, in <module>
    args.op()
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/bin/..nixops-wrapped-wrapped", line 395, in op_destroy
    wipe=args.wipe)
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1060, in destroy_resources
    self._destroy_resources(include, exclude, wipe)
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1054, in _destroy_resources
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.resources.values(), worker_fun=worker)
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/lib/python2.7/site-packages/nixops/parallel.py", line 41, in thread_fun
    result_queue.put((worker_fun(t), None))
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1047, in worker
    if m.destroy(wipe=wipe): self.delete_resource(m)
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 731, in destroy
    return self._destroy(server, wipe)
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 698, in _destroy
    self.reboot_rescue(bootstrap=False, hard=True)
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 344, in reboot_rescue
    server = self._get_server_by_ip(self.main_ipv4)
  File "/nix/store/hhmqn7cg6h1w2ajxb5hacyva1rk192q3-nixops-1.5pre0_abcdef/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 152, in _get_server_by_ip
    return robot.servers.get(ip)
  File "/nix/store/ylycj3msl47qhrida7k51wrfml73k85v-python2.7-hetzner-0.7.4/lib/python2.7/site-packages/hetzner/robot.py", line 274, in get
    return Server(self.conn, self.conn.get('/server/{0}'.format(ip)))
  File "/nix/store/ylycj3msl47qhrida7k51wrfml73k85v-python2.7-hetzner-0.7.4/lib/python2.7/site-packages/hetzner/robot.py", line 260, in <lambda>
    get = lambda s, p: s.request('GET', p)
  File "/nix/store/ylycj3msl47qhrida7k51wrfml73k85v-python2.7-hetzner-0.7.4/lib/python2.7/site-packages/hetzner/robot.py", line 258, in request
    raise RobotError(err, response.status)
hetzner.RobotError

Seems like my admin robot account is not working

$ sqlite3 ~/.nixops/deployments.nixops "select * from ResourceAttrs where name like '%robot%'"
10|hetzner.robotUser|#558...jQ
10|hetzner.robotPass|TqG#;...zJ,eDngAA#-

$ curl -u "#558...jQ:TqG...AA#-" https://robot-ws.your-server.de/server/$IP | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    71  100    71    0     0     76      0 --:--:-- --:--:-- --:--:--    76
{
    "error": {
        "code": "UNAUTHORIZED",
        "message": "Unauthorized",
        "status": 401
    }
}

I'm on https://github.com/NixOS/nixops/commit/c92f864fb13720d66363873c200b4b59c28d94bb

[domenkozar]

You sure you have correct credentials? If you can't query robot api with credentials, I don't know how nixops could do better.

Am I missing something?

[danbst]

I think, that admin robot account credentials were OK some time ago, but then I got a better from hetzner team, where they blocked that account as suspisious (I can find out more details, If needed).

Sometimes I have now broken account and deployment resources are binded to it. I've create new deployment with same server as workaround, but can't drop previous

[nh2]

As written in https://github.com/NixOS/nixops/issues/627#issuecomment-293582267, this is probably because the user/password in the state do not get updated properly when changed in the machine definition in nix.

It appears that create() has a special case for this:

https://github.com/NixOS/nixops/blob/da336200c15d2aa145b0ddf29d5d5db5702d4f4f/nixops/backends/hetzner.py#L602-L614

but destroy() doesn't.

[nh2]

I also found that destroying a Hetzner when admin accounts are used doesn't work well.

There are 2 issues:

1. It asks for the robot user even if it's defined in the file
2. It tries to delete the admin account (which cannot work when using admin accounts in nixops because only the "super-admin" account can do that)

1. Asking for robot user

nixops destroy -d hetzner-reboot-test

Traceback (most recent call last):
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/bin/.nixops-wrapped", line 991, in <module>
    args.op()
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/bin/.nixops-wrapped", line 429, in op_destroy
    wipe=args.wipe)
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1139, in destroy_resources
    self.run_with_notify('destroy', lambda: self._destroy_resources(include, exclude, wipe))
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1044, in run_with_notify
    f()
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1139, in <lambda>
    self.run_with_notify('destroy', lambda: self._destroy_resources(include, exclude, wipe))
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1133, in _destroy_resources
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.resources.values(), worker_fun=worker)
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/parallel.py", line 44, in thread_fun
    result_queue.put((worker_fun(t), None, t.name))
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1126, in worker
    if m.destroy(wipe=wipe): self.delete_resource(m)
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 812, in destroy
    server = self._get_server_from_main_robot(self.main_ipv4)
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 167, in _get_server_from_main_robot
    (robot_user, robot_pass) = self._get_robot_user_and_pass(defn=defn)
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 158, in _get_robot_user_and_pass
    " ‘{0}’".format(self.name))
Exception: please either set ‘deployment.hetzner.robotUser’ or $HETZNER_ROBOT_USER for machine ‘builder’

This is despite having set

    deployment.hetzner.createSubAccount = false; # because we use 2-factor auth
    deployment.hetzner.robotUser = "....stuff....";

2. Incorrectly trying to delete the admin user

HETZNER_ROBOT_USER="#111111+AAAAA" HETZNER_ROBOT_PASS="$HETZNER_ROBOT_PASS" /usr/bin/time ./ops destroy -d hetzner-reboot-test

warning: are you sure you want to destroy Hetzner machine ‘builder’? (y/N) y
builder> rebooting machine ‘builder’ (46.4.89.8) into rescue system
builder> sending hard reset to robot... done.
builder> waiting for rescue system...[down]............................................................................[up]
builder> unsetting server name... done.
builder> removing admin account... Traceback (most recent call last):
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/bin/.nixops-wrapped", line 991, in <module>
    args.op()
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/bin/.nixops-wrapped", line 429, in op_destroy
    wipe=args.wipe)
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1139, in destroy_resources
    self.run_with_notify('destroy', lambda: self._destroy_resources(include, exclude, wipe))
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1044, in run_with_notify
    f()
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1139, in <lambda>
    self.run_with_notify('destroy', lambda: self._destroy_resources(include, exclude, wipe))
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1133, in _destroy_resources
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.resources.values(), worker_fun=worker)
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/parallel.py", line 44, in thread_fun
    result_queue.put((worker_fun(t), None, t.name))
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1126, in worker
    if m.destroy(wipe=wipe): self.delete_resource(m)
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 822, in destroy
    return self._destroy(server, wipe)
  File "/nix/store/kpcls34xivbfg8k0f5fhnd5firkg6a40-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 800, in _destroy
    server.admin.delete()
  File "/nix/store/pihvnr9fdgraqrkxwnjick72d54vcd3j-python2.7-hetzner-0.7.5-custom/lib/python2.7/site-packages/hetzner/server.py", line 425, in admin
    self._admin_account = AdminAccount(self)
  File "/nix/store/pihvnr9fdgraqrkxwnjick72d54vcd3j-python2.7-hetzner-0.7.5-custom/lib/python2.7/site-packages/hetzner/server.py", line 167, in __init__
    self.update_info()
  File "/nix/store/pihvnr9fdgraqrkxwnjick72d54vcd3j-python2.7-hetzner-0.7.5-custom/lib/python2.7/site-packages/hetzner/server.py", line 173, in update_info
    self._scraper.login()
  File "/nix/store/pihvnr9fdgraqrkxwnjick72d54vcd3j-python2.7-hetzner-0.7.5-custom/lib/python2.7/site-packages/hetzner/robot.py", line 95, in login
    " page".format(response.status))
hetzner.WebRobotError: Invalid status code 302 while visiting login page
Command exited with non-zero status 1
0.79user 0.16system 1:25.68elapsed 1%CPU (0avgtext+0avgdata 71640maxresident)k
32inputs+336outputs (0major+57052minor)pagefaults 0swaps

It shouldn't try to do removing admin account when createSubAccount = false.

This exception being thrown means that it won't execute the lines following this error, which actually remove the deployment from the nixops database.