system.stateVersion differs from default value when not setting explicitely

[datafoo]

I am using NixOps to deploy a server. Before the first nixops deploy, the server NixOS version was 19.09 and NixOps successfully noticed that:

$ nixops --version
NixOps 1.7
$ nixops show-physical -d mydeployment
{
  server = { config, lib, pkgs, ... }: {
    config = {
      boot.kernelModules = [];
      networking = {
        extraHosts = "127.0.0.1 applicationServer-encrypted\n";
        firewall.trustedInterfaces = [];
        vpnPublicKey = "ssh-ed25519 […] NixOps VPN key of server";
      };
      system.stateVersion = ( lib.mkDefault "19.09" );
    };
    imports = [
      {
        config.users.extraUsers.root.openssh.authorizedKeys.keys = [
          "ssh-ed25519 […] NixOps client key for server"
        ];
      }
    ];
  };
}

The value is indeed 19.09:

$ nixops info -d mydeployment
[…]
Nix path: -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixos-19.09-small.tar.gz
[…]
$ nixops show-option -d mydeployment server system.stateVersion
"19.09"

However, when I use the nixos-20.03-small channel, the value is changed to 20.03:

$ nixops info -d mydeployment
[…]
Nix path: -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixos-20.03-small.tar.gz
[…]
$ nixops show-option -d mydeployment server system.stateVersion
"20.03"
$ nixops show-physical -d mydeployment
{
  server = { config, lib, pkgs, ... }: {
    config = {
      boot.kernelModules = [];
      networking = {
        extraHosts = "127.0.0.1 applicationServer-encrypted\n";
        firewall.trustedInterfaces = [];
        vpnPublicKey = "ssh-ed25519 […] NixOps VPN key of server";
      };
      system.stateVersion = ( lib.mkDefault "19.09" );
    };
    imports = [
      {
        config.users.extraUsers.root.openssh.authorizedKeys.keys = [
          "ssh-ed25519 […] NixOps client key for server"
        ];
      }
    ];
  };
}

I do not set system.stateVersion in my deployment files. Why doesn’t the value stay at 19.09?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/system-stateversion-with-nixops/7094/4

[jhillyerd]

We should also document if/when to include system.stateVersion in the Nixops manual, as it's not part of the examples and is non-obvious.

[datafoo]

Any idea, anyone?

[ghost]

Just to add to this, the default value is overriding my forced value.

nixops --version
NixOps 1.7

I have in default imports:

system.stateVersion = lib.mkForce "20.09";

nix info
...
Nix path: -I nixpkgs=https://nixos.org/channels/nixos-20.09/nixexprs.tar.xz

nixops show-option -d default server system.stateVersion
"20.09"

And yet nixops show-physical shows system.stateVersion = ( lib.mkDefault "21.03" ); in all the servers, and indeed sets that state version when deploying.