A nix-shell with awscli and Keras results in awscli not working (reporting a SyntaxError on command invocation)

[CMCDragonkai]

Issue description

It appears that awscli is not working when in the same shell as Keras.

Steps to reproduce

nix-shell -p awscli python35.pkgs.Keras

Once you're in, run aws s3.

It reports:

> aws s3
Traceback (most recent call last):
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/bin/.aws-wrapped", line 28, in <module>
    sys.exit(main())
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/bin/.aws-wrapped", line 24, in main
    return awscli.clidriver.main()
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/lib/python2.7/site-packages/awscli/clidriver.py", line 57, in main
    driver = create_clidriver()
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/lib/python2.7/site-packages/awscli/clidriver.py", line 68, in create_clidriver
    event_hooks=emitter)
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/lib/python2.7/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/lib/python2.7/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/lib/python2.7/site-packages/awscli/handlers.py", line 27, in <module>
    from awscli.customizations.cloudformation import initialize as cloudformation_init
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/lib/python2.7/site-packages/awscli/customizations/cloudformation/__init__.py", line 13, in <module>
    from awscli.customizations.cloudformation.package import PackageCommand
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/lib/python2.7/site-packages/awscli/customizations/cloudformation/package.py", line 22, in <module>
    from awscli.customizations.cloudformation.artifact_exporter import Template
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/lib/python2.7/site-packages/awscli/customizations/cloudformation/artifact_exporter.py", line 26, in <module>
    from awscli.customizations.cloudformation.yamlhelper import yaml_dump, \
  File "/nix/store/q6p0414mhxxjs3npigmfznyj6qb41lmx-awscli-1.14.22/lib/python2.7/site-packages/awscli/customizations/cloudformation/yamlhelper.py", line 14, in <module>
    import yaml
  File "/nix/store/4v8jgw3r3ksz04zpyb7wad5p7r2zy610-python3.5-PyYAML-3.12/lib/python3.5/site-packages/yaml/__init__.py", line 284
    class YAMLObject(metaclass=YAMLObjectMetaclass):
                              ^
SyntaxError: invalid syntax

But if you do this instead:

nix-shell -p python35.pkgs.Keras awscli

It ends up working! However I fear that this would make Keras fail because it would then find the python 2.7 PyYAML instead of the Python 3 YAML.

It's because of the PYTHONPATH order.

I think this behaviour is really unexpected, can these dependencies be made more isolated? Also is this an aspect of nix-shell, and would not be something happens if you were to do nix-env?

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the results.

 - system: `"x86_64-linux"`
 - host os: `Linux 4.15.7, Form Bay ML, noversion`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 1.11.16`
 - nixpkgs: `/nix/nixpkgs`

[CMCDragonkai]

This problem appears to be related (in the ordering of buildInputs): https://github.com/NixOS/nixpkgs/issues/38369#issuecomment-379935922

Both are Python related!

[CMCDragonkai]

This isn't really a Nix problem, but a problem with upstream Python packages that have a resource/naming conflict. Should really be solved in the Python ecosystem. But we should report this on the manual about this edge case.

[Zebradil]

I had the same issue after installing it alone: nix-env -i awscli. It turned out that I had PYTHONPATH set (and it was pointing to a site-packages dir for python3).