Update for recent azure-mgmt-* libraries

[bendlas]

I've recently reverted an update to the azure-mgmt-* libraries in nixpkgs, because it broke nixops. I've submitted a PR with the updates, but nixops still doesn't play with the newer libraries

from https://github.com/NixOS/nixpkgs/pull/52550

Traceback (most recent call last):
  File "/nix/store/jsnrx8ggppa9q7iiry88s763y0p2qhb5-nixops-1.6.1pre2706_d5ad09c/bin/..nixops-wrapped-wrapped", line 5, in <module>
    from nixops import deployment
  File "/nix/store/jsnrx8ggppa9q7iiry88s763y0p2qhb5-nixops-1.6.1pre2706_d5ad09c/lib/python2.7/site-packages/nixops/deployment.py", line 1275, in <module>
    _load_modules_from("backends")
  File "/nix/store/jsnrx8ggppa9q7iiry88s763y0p2qhb5-nixops-1.6.1pre2706_d5ad09c/lib/python2.7/site-packages/nixops/deployment.py", line 1273, in _load_modules_from
    importlib.import_module("nixops." + dir + "." + module[:-3])
  File "/nix/store/w30zkhdjf6qp4ppx9xzffsh9dic31aq8-python-2.7.15/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/nix/store/jsnrx8ggppa9q7iiry88s763y0p2qhb5-nixops-1.6.1pre2706_d5ad09c/lib/python2.7/site-packages/nixops/backends/azure_vm.py", line 23, in <module>
    from nixops.azure_common import ResourceDefinition, ResourceState, ResId, normalize_location
  File "/nix/store/jsnrx8ggppa9q7iiry88s763y0p2qhb5-nixops-1.6.1pre2706_d5ad09c/lib/python2.7/site-packages/nixops/azure_common.py", line 19, in <module>
    from azure.mgmt.network import NetworkResourceProviderClient
ImportError: cannot import name NetworkResourceProviderClient

[teto]

I don't think the pinning done in nixpkgs works when using ./dev-shell that uses the local release.nix. Can't we finally use the same derivation and just run nix-shell -A nixops instead ?
I just tried only to end up with

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///home/teto/nixops
Installing collected packages: nixops
  Running setup.py develop for nixops
    Complete output from command /nix/store/ih03q5vi6kj278l4jk4kh41lqyagzzdi-python-2.7.15/bin/python2.7 -c "import setuptools, tokenize;__file__='/home/teto/nixops/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/run/user/1000/tmp.0Ny9Q1tV83:
    /nix/store/cf5r9m4nagdcw16vfbiyxv6x1cl27ds6-python2.7-bootstrapped-pip-19.0.3/lib/python2.7/site-packages/setuptools/dist.py:484: UserWarning: The version specified ('@version@') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
      "details." % self.metadata.version
    running develop
    running egg_info
    writing nixops.egg-info/PKG-INFO
    writing top-level names to nixops.egg-info/top_level.txt
    writing dependency_links to nixops.egg-info/dependency_links.txt
    writing manifest file 'nixops.egg-info/SOURCES.txt'
    running build_ext
    Creating /run/user/1000/tmp.0Ny9Q1tV83/lib/python2.7/site-packages/site.py
    Creating /run/user/1000/tmp.0Ny9Q1tV83/lib/python2.7/site-packages/nixops.egg-link (link to .)
    Adding nixops -version- to easy-install.pth file
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/teto/nixops/setup.py", line 30, in <module>
        cmdclass={'test': TestCommand}
      File "/nix/store/ih03q5vi6kj278l4jk4kh41lqyagzzdi-python-2.7.15/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/nix/store/ih03q5vi6kj278l4jk4kh41lqyagzzdi-python-2.7.15/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/nix/store/ih03q5vi6kj278l4jk4kh41lqyagzzdi-python-2.7.15/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/nix/store/cf5r9m4nagdcw16vfbiyxv6x1cl27ds6-python2.7-bootstrapped-pip-19.0.3/lib/python2.7/site-packages/setuptools/command/develop.py", line 38, in run
        self.install_for_development()
      File "/nix/store/cf5r9m4nagdcw16vfbiyxv6x1cl27ds6-python2.7-bootstrapped-pip-19.0.3/lib/python2.7/site-packages/setuptools/command/develop.py", line 156, in install_for_development
        self.process_distribution(None, self.dist, not self.no_deps)
      File "/nix/store/cf5r9m4nagdcw16vfbiyxv6x1cl27ds6-python2.7-bootstrapped-pip-19.0.3/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 732, in process_distribution
        self.install_egg_scripts(dist)
      File "/nix/store/cf5r9m4nagdcw16vfbiyxv6x1cl27ds6-python2.7-bootstrapped-pip-19.0.3/lib/python2.7/site-packages/setuptools/command/develop.py", line 191, in install_egg_scripts
        script_text = strm.read()
      File "/nix/store/ih03q5vi6kj278l4jk4kh41lqyagzzdi-python-2.7.15/lib/python2.7/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2845: ordinal not in range(128)

[teto]

I use libvirt so I removed all the azure dependencies from release.nix as a temporary workaround.

or rather change release.nix line to propagatedBuildInputs = pkgs.nixops.pythonPath;.

[flokli]

Apart from the python2 problem,

nixops' release.nix also imports python2Packages.azure-* from <nixpkgs> by default, currently breaking Travis CI as well: https://github.com/NixOS/nixops/pull/1006#issuecomment-470304447

[asymmetric]

Does this mean nixops with recent versions of <nixpkgs> is broken?

I'm trying running

❯ env NIX_PATH="nixpkgs=https://github.com/nixos/nixpkgs/archive/release-19.03.tar.gz" nix-shell release.nix -A build.x86_64-linux --exclude tarball

and hitting this or a related problem.

Right now I'm forced to run it against release-18.09.

[AmineChikhaoui]

Currently in nixpkgs the azure-* libs are pinned to a working version (https://github.com/NixOS/nixpkgs/tree/master/pkgs/tools/package-management/nixops) so the latest release is still working but it's kinda of a release blocker if we want to release a new version. So we need an update to the implementation to take care of the API updates in azure-* libs if we want to move forward. I'm not sure if I'll get time to do that so if anyone is willing to step up and fix it that would really be great.

[asymmetric]

@AmineChikhaoui My opinion is that considering that:

• the pinned version is 4 years old
• a lot has changed in that library in the meantime (just tried updating it)
• this issue is blocking

We should consider removing Azure support temporarily, until someone feels responsible for fixing it.

[asymmetric]

See also discussion here.

[AmineChikhaoui]

@asymmetric I agree, removing the backend might be a temporary solution until we fix it. I'm attempting the update this evening, just to get an idea on how much is needed. Then if it's too much work/complicated, I think we'll have to just remove it until someone fix it.

[asymmetric]

Support for Azure has been disabled for now.

[Nekroze]

Even with it being "disabled" I was unable to build nixops (because of an import error on a subpackage to the azure libs that I guess no longer exists) using 19.03 nixpkgs, had to use 18.09 instead.

[asymmetric]

I've been building it with 19.03 with no issues, with the following shell.nix:

{
  pkgsSrc ? fetchGit {
    url = https://github.com/nixos/nixpkgs-channels;
    ref = "nixos-19.03";
    rev = "2f1eacc949fe31f34d7af4ebfd97c01a02ceee16";
  },
  nixopsSrc ? fetchTarball {
    url = https://github.com/nixos/nixops/archive/cbbecd24185c3a1704a778f1410a0d63b692892d.tar.gz;
    sha256 = "1fpx7glc14v7mhp86ah010isbbjj0fk6iraszsj6i0b0cznc3yi4";
  } + "/release.nix"
}:

let
  pkgs = (import pkgsSrc {});
  nixops = (import nixopsSrc { nixpkgs = pkgs.path; }).build.x86_64-linux;
in
  pkgs.mkShell {
    name = "nixops-env";
    buildInputs = [ nixops ];
    NIX_PATH="nixpkgs=${pkgs.path}";
  }