dependency on `google-compute-engine`

[indraniel]

I've noticed an issue when installing dsub on my laptop. When I ran the example mentioned in the README:

dsub \
   --provider local \
   --logging /tmp/dsub-test/logging/ \
   --output OUT=/tmp/dsub-test/output/out.txt \
   --command 'echo "Hello World" > "${OUT}"' \
   --wait

I got an error like so:

Waiting for: echo--me--181205-185159-848773.
  echo--me--181205-185159-848773: FAILURE
['2018-12-05 18:52:04 I: Copying the logs before cleanup']
JobExecutionError: One or more jobs finished with status FAILURE or CANCELED during wait.
echo--me--181205-185159-848773

When I inspected the relevant log file, I noticed the following:

$ cat /tmp/dsub-local/echo--me--181205-185159-848773/task/runner-log.txt
2018-12-05 18:51:59 I: Localizing inputs.
2018-12-05 18:51:59 I: Checking image userid.
2018-12-05 18:52:01 I: Running Docker image.
322bc9fc4bcc7fd531f3cfecdf1ce919217b383bcf3ca0c6d4f4647fa1561da9
2018-12-05 18:52:02 I: Docker exit code 0.
2018-12-05 18:52:02 I: Ensure host user (1000:1001) owns Docker-written data
2018-12-05 18:52:03 I: Copying outputs.
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil", line 22, in <module>
    gsutil.RunMain()
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py", line 116, in RunMain
    import gslib.__main__
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 39, in <module>
    import boto
  File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/boto/boto/__init__.py", line 1216, in <module>
    boto.plugin.load_plugins(config)
  File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/boto/boto/plugin.py", line 93, in load_plugins
    _import_module(file)
  File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/boto/boto/plugin.py", line 75, in _import_module
    return imp.load_module(name, file, filename, data)
  File "/usr/lib/python2.7/dist-packages/google_compute_engine/boto/compute_auth.py", line 19, in <module>
    from google_compute_engine import logger
ImportError: No module named google_compute_engine
2018-12-05 18:52:04 E: Error on or near line 45; /tmp/dsub-local/echo--me--181205-185159-848773/task/runner.sh exiting with status 1
2018-12-05 18:52:04 I: Copying the logs before cleanup

Upon running pip install google-compute-engine into my virtualenv, the issue went away. Should installing the python library google-compute-engine be mentioned in the documentation, or added into the setup.py?

[mbookman]

Hi @indraniel!

The issue isn't a direct dependency of dsub on this particular library, so I would not change setup.py. The local provider here is calling gsutil to do some file copies and for some reason your gsutil install has some issues.

There are a number of issues that have been seen with this problem with the gsutil install:

https://github.com/travis-ci/travis-ci/issues/7940 https://github.com/travis-ci/travis-ci/issues/8182

I think the root of this is addressed here:

https://github.com/GoogleCloudPlatform/gsutil/issues/459

Boto automatically looks in certain locations (/etc/boto.cfg being one of them) and picks up those config files if present That config file tells Boto it should load the google_compute_engine package from somewhere outside the virtual environmen

do you have an /etc/boto.cfg?

Thanks!

[indraniel]

Yes, I do have an /etc/boto.cfg file. I was installing dsub in a virtualenv on a Google Compute Instance VM. That makes more sense now. Thanks again for the explanation @mbookman !

[indraniel]

I'll go ahead and close this issue for now.