Closed cmagina closed 8 years ago
johnsca
commented
8 years ago Can you confirm that the HTTPS_PROXY env var is set on the system (juju run --unit hdfs-master/0 'echo $HTTPS_PROXY')? If that's set as it should be, pip should honor it. Perhaps there's an issue passing env through to the subprocess.
Also, I assume that you are not setting the resources_mirror config option.
cmagina
commented
8 years ago Its not set. I checked the environments/
johnsca
commented
8 years ago You should also be able to use juju set-env http-proxy ... to set it for an existing deployment, though I haven't used that myself so I'm not 100% certain on that.
cmagina
commented
8 years ago Ok, so I got the config right and the environment variables are set now. This is giving me a new error when jujuresources tries to download pathlib, jujubigdata and java-installer via pip.
2015-11-03 23:03:37 INFO juju-log Fetching pathlib 2015-11-03 23:03:40 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:40 INFO worker.uniter.jujuc server.go:158 running hook tool "juju-log" ["-l" "INFO" "Fetching java-installer"] 2015-11-03 23:03:40 DEBUG worker.uniter.jujuc server.go:159 hook context id "apache-hadoop-compute-slave/0-install-6234698512339578973"; dir "/var/lib/juju/agents/unit-apache-hadoop-compute-slave-0/charm" 2015-11-03 23:03:40 INFO juju-log Fetching java-installer 2015-11-03 23:03:40 INFO worker.uniter.jujuc server.go:158 running hook tool "juju-log" ["-l" "INFO" "Fetching jujubigdata"] 2015-11-03 23:03:40 DEBUG worker.uniter.jujuc server.go:159 hook context id "apache-hadoop-compute-slave/0-install-6234698512339578973"; dir "/var/lib/juju/agents/unit-apache-hadoop-compute-slave-0/charm" 2015-11-03 23:03:40 INFO juju-log Fetching jujubigdata 2015-11-03 23:03:42 DEBUG juju.worker.leadership tracker.go:138 apache-hadoop-compute-slave/0 renewing lease for apache-hadoop-compute-slave leadership 2015-11-03 23:03:42 DEBUG juju.worker.leadership tracker.go:165 checking apache-hadoop-compute-slave/0 for apache-hadoop-compute-slave leadership 2015-11-03 23:03:42 DEBUG juju.worker.leadership tracker.go:180 apache-hadoop-compute-slave/0 confirmed for apache-hadoop-compute-slave leadership until 2015-11-03 23:04:42.094156214 +0000 UTC 2015-11-03 23:03:42 INFO juju.worker.leadership tracker.go:182 apache-hadoop-compute-slave/0 will renew apache-hadoop-compute-slave leadership at 2015-11-03 23:04:12.094156214 +0000 UTC 2015-11-03 23:03:48 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:48 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:48 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:48 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:49 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:49 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:49 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:49 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:50 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:50 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:50 INFO install Error fetching index https://pypi.python.org/simple/: 501 (Not Implemented) 2015-11-03 23:03:50 INFO worker.uniter.jujuc server.go:158 running hook tool "juju-log" ["-l" "WARNING" "Failed to fetch resources: pathlib, jujubigdata, java-installer"] 2015-11-03 23:03:50 DEBUG worker.uniter.jujuc server.go:159 hook context id "apache-hadoop-compute-slave/0-install-6234698512339578973"; dir "/var/lib/juju/agents/unit-apache-hadoop-compute-slave-0/charm" 2015-11-03 23:03:50 WARNING juju-log Failed to fetch resources: pathlib, jujubigdata, java-installer 2015-11-03 23:03:50 INFO worker.uniter.jujuc server.go:158 running hook tool "status-set" ["blocked" "Unable to fetch required resources: pathlib, jujubigdata, java-installer"] 2015-11-03 23:03:50 DEBUG worker.uniter.jujuc server.go:159 hook context id "apache-hadoop-compute-slave/0-install-6234698512339578973"; dir "/var/lib/juju/agents/unit-apache-hadoop-compute-slave-0/charm" 2015-11-03 23:03:50 DEBUG juju.worker.uniter.context context.go:290 [WORKLOAD-STATUS] blocked: Unable to fetch required resources: pathlib, jujubigdata, java-installer
cmagina
commented
8 years ago I used juju run to see if pip could successfully install the packages and it can:
juju run --unit apache-hadoop-compute-slave/0 'pip install pathlib' Downloading/unpacking pathlib Running setup.py (path:/tmp/pip_build_root/pathlib/setup.py) egg_info for package pathlib
Installing collected packages: pathlib Running setup.py install for pathlib
Successfully installed pathlib Cleaning up...
kwmonroe
commented
8 years ago Hi Craig - this looks like issue #11, with a potential fix provided in PR #12. We'll bump the priority to get this tested and pushed as soon as we verify it doesn't break non-restricted environments. Thanks for the report! I'll leave this open until an updated jujuresources has been released.
kwmonroe
commented
8 years ago This has been resolved with PR #15, and is available in jujuresources >= 0.3.0. All of the the Apache-based big data charms in the ~bigdata-dev namespace have been updated to use this library:
https://jujucharms.com/u/bigdata-dev/#charms
This will make its way from -dev to the recommended namespace during the next promulgation window. Thanks for the report!
The apache-hadoop charms use jujuresources to install some resources from pypi. In a locked down environment, it will fail even if there is a proxy configured via juju's environments.yaml that would allow it through.