search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.39k stars 14.34k forks source link

ansible: Module maven_artifact.py missing dependency #25869

Open fmthoma opened 7 years ago

fmthoma commented 7 years ago

Issue description

Using maven_artifact in an ansible playbook results leads to the following error:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: No module named lxml
fatal: [*** -> localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_3H5i_j/ansible_module_maven_artifact.py\", line 24, in <module>\n    from lxml import etree\nImportError: No module named lxml\n", "module_stdout": "", "msg": "MODULE FAILURE"}

Steps to reproduce

Technical details

Debugging info

Disclaimer: I'm not familiar with Python, so I may get some details wrong.

What I found out so far:

FRidh commented 7 years ago

The maven_artifact.py file is run as as a script. That means it would have to be wrapped with wrapPythonProgramsIn. However, it might be the case that it is also used as a Python module and thus has to be importable. With our current Python infra, we can't support both, because that would result in the same issue as we have with hplip.

fmthoma commented 7 years ago

@FRidh In this case, it's not executable (although it does have a shebang), and hence not touched by wrapPythonProgramsIn. And actually, I'm not sure if it's run as a script after all, there seems to be some custom logic how these modules are loaded (including parsing the actual PYTHONPATH, see lib/python2.7/site-packages/ansible/executor/module_common.py in $out).

So I think in this case your proposal in #24263 of patching the scripts would have to be extended to non-executable modules, or offer an API for the ansible nixpkgs module to patch these files directly.

And for my understanding, one more question: Why was $PYTHONPATH abandoned in the first place?

FRidh commented 7 years ago

If we were to use PYTHONPATH, then we would have had to use set instead of prefix or suffix in order to prevent it from leaking PYTHONPATH to subprocesses. The problem with set is that one cannot extend PYTHONPATH anymore, breaking a Python feature. It would also broke our checkPhase.

BlessJah commented 7 years ago

@FRidh #28171 can be worked around by providing python27.withPackages environment passed as ansible_python_interpreter. I've tested that on Linux, I've also checked that script can be used as shebang interpreter on MacOS 10.13. @copumpkin pointed that it may break MacOS 10.11 and prior.

stale[bot] commented 4 years ago

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

  1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
  2. Ask on the NixOS Discourse.
  3. Ask on the #nixos channel on irc.freenode.net.
cdepillabout commented 2 years ago

I found this issue from https://github.com/NixOS/nixpkgs/issues/28171 when faced with the problem that Ansible doesn't seem to see the boto or boto3 libraries, so we get errors when trying to use the aws_ec2 inventory, the sts_assume_role module, and the ec2 module.

I was able to work around this by forcing the PYTHONPATH to be set in the ansible derivation. Here's something similar to our workaround:

let
  nixpkgs-stable-src = builtins.fetchTarball {
    # release-21.11 as of 2022-02-14
    url = "https://github.com/NixOS/nixpkgs/archive/30d3d79b7d3607d56546dd2a6b49e156ba0ec634.tar.gz";
    sha256 = "0x5j9q1vi00c6kavnjlrwl3yy1xs60c34pkygm49dld2sgws7n0a";
  };

  pkgsStable = import nixpkgs-stable-src { };

  ourPythonPackagesForAnsible = pkgsStable.python39Packages.override
    (oldAttrs: {
      overrides = pkgsStable.lib.composeManyExtensions [
        (oldAttrs.overrides or (_: _: { }))
        (pfinal: pprev: {
          ansible = pprev.ansible.overridePythonAttrs (old: {
            propagatedBuildInputs = (old.propagatedBuildInputs or [ ]) ++ [ pfinal.boto pfinal.boto3 ];
            makeWrapperArgs = (old.makeWrapperArgs or []) ++ [ "--prefix PYTHONPATH : $PYTHONPATH" ];
          });
        })
      ];
    });

  # Note that we are using `ansible` here, and not `ansible-core`, since it doesn't appear that `ansible-core` has support
  # by default for the AWS-related modules and inventories.
  ourAnsible =
    (ourPythonPackagesForAnsible.toPythonApplication ourPythonPackagesForAnsible.ansible);
in
ourAnsible

I'm not very familiar with Python or Ansible, but I got the idea for this from apache-airflow, which seems to be doing something similar:

https://github.com/NixOS/nixpkgs/blob/f7b03e45821aa6427e46f5973aae4753b2f2255d/pkgs/development/python-modules/apache-airflow/default.nix#L212-L215

Also note that this is using 21.11. I'm not sure if anything would need to be changed for a more recent Nixpkgs.

teto commented 1 year ago

it's annoying that ansible doesn't pick up the python in PATH but the unwrapped one. What I use as a quick hack:

      devShells.${system}.default =
        let
          pyEnv = pkgs.python3.withPackages (p: [
            # p.ansible
            p.ansible-core
            p.boto3
          ]);
        in

        pkgs.mkShell {
          name = "whyonearthwouldyouuseansiblewhennixexists";
          buildInputs = [
            pkgs.jq
            pkgs.nodejs
            pyEnv
          ];

          shellHook = ''
            # this is a hack because for some reason ansible doesn't pick the python in
            # path and thus doesn't find boto3
            cat <<EOF > ansible.cfg
            [defaults]
            interpreter_python=${pyEnv.interpreter}
            EOF
          '';
        };

Seems like one can set an environment variable too https://docs.ansible.com/ansible/latest/reference_appendices/config.html#envvar-ANSIBLE_PYTHON_INTERPRETER .

Note that if you set the interpreter in a playbook, it's not interpreter_python but ansible_python_interpreter (just why ? so you could sell consultancy services xD ?)