error: can't copy 'version.ini': doesn't exist or not a regular file

[juju4]

Issue with pip install on HEAD

    copying grr_response_core/artifacts/flow_templates/processes.yaml -> /usr/local/share/env-grr/grr_response_core/artifacts/flow_templates\n    copying grr_response_core/artifacts/flow_templates/windows.yaml -> /usr/local/share/env-grr/grr_response_core/artifacts/flow_templates\n    copying grr_response_core/artifacts/flow_templates/cron.yaml -> /usr/local/share/env-grr/grr_response_core/artifacts/flow_templates\n    copying grr_response_core/artifacts/flow_templates/grr_actions.yaml -> /usr/local/share/env-grr/grr_response_core/artifacts/flow_templates\n    copying grr_response_core/artifacts/flow_templates/disk.yaml -> /usr/local/share/env-grr/grr_response_core/artifacts/flow_templates\n    copying grr_response_core/artifacts/flow_templates/darwin.yaml -> /usr/local/share/env-grr/grr_response_core/artifacts/flow_templates\n    warning: install_data: setup script did not provide a directory for 'version.ini' -- installing right in '/usr/local/share/env-grr'\n    \n    error: can't copy 'version.ini': doesn't exist or not a regular file\n    ----------------------------------------\nERROR: Command errored out with exit status 1: /usr/local/share/env-grr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/tmp/pip-install-a2nOZN/grr-response-core/grr/core/setup.py'\"'\"'; __file__='\"'\"'/tmp/pip-install-a2nOZN/grr-response-core/grr/core/setup.py'\"'\"';f=getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__);code=f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' install --record /tmp/pip-record-8cqovZ/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/share/env-grr/include/site/python2.7/grr-response-core Check the logs for full command output.\n"}

https://travis-ci.org/juju4/ansible-grr/jobs/579014806#L2381 (Ubuntu 18.04, pip install from google github HEAD with python2)

similar with python3 on centos7 + ubuntu 18.04 and slightly updated code (https://github.com/google/grr/compare/master...juju4:devel)

    copying grr_response_core/artifacts/flow_templates/disk.yaml -> /usr/local/share/env-grr/grr_response_core/artifacts/flow_templates\n    copying grr_response_core/artifacts/flow_templates/darwin.yaml -> /usr/local/share/env-grr/grr_response_core/artifacts/flow_templates\n    warning: install_data: setup script did not provide a directory for 'version.ini' -- installing right in '/usr/local/share/env-grr'\n    \n    error: can't copy 'version.ini': doesn't exist or not a regular file\n    ----------------------------------------\nERROR: Command errored out with exit status 1: /usr/local/share/env-grr/bin/python36 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/tmp/pip-install-0d6sbs6d/grr-response-core/grr/core/setup.py'\"'\"'; __file__='\"'\"'/tmp/pip-install-0d6sbs6d/grr-response-core/grr/core/setup.py'\"'\"';f=getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__);code=f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' install --record /tmp/pip-record-9w9l_8cy/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/share/env-grr/include/site/python3.6/grr-response-core Check the logs for full command output.\n"}

https://travis-ci.org/juju4/ansible-grr/jobs/579014808#L1659 https://travis-ci.org/juju4/ansible-grr/jobs/579014810#L2388

Thanks

[mbushkov]

@juju4 , quick question - this is not a regression, right? I.e. the setup you reference is not something that was working before and got broken due to recent GRR changes, correct?

I checked the travis CI. Seems like you're trying to install HEAD GRR version through PIP by doing this:

ok: [localhost] => {
    "grr_pip_name": [
        "mysqlclient", 
        "git+https://github.com/chipsec/chipsec.git@1.4.0#egg=chipsec", 
        "git+https://github.com/juju4/grr.git@devel#egg=grr-response-core&subdirectory=grr/core", 
        "git+https://github.com/juju4/grr.git@devel#egg=grr-response-client&subdirectory=grr/client", 
        "git+https://github.com/juju4/grr.git@devel#egg=grr-response-server&subdirectory=grr/server", 
        "git+https://github.com/juju4/grr.git@devel#egg=grr-response-proto&subdirectory=grr/proto", 
        "git+https://github.com/juju4/grr.git@devel#egg=grr-response-client-builder&subdirectory=grr/client_builder", 
        "git+https://github.com/juju4/grr.git@devel#egg=grr-api-client&subdirectory=api_client/python"
    ]
}

Unfortunately, this setup method is not supported by GRR at the moment. All our PIP packages depend on a version.ini file at the root of GRR repository. If you checkout just a part of the repository (and this is what pip would do when provided with a URL like git+https://github.com/juju4/grr.git@devel#egg=grr-response-core&subdirectory=grr/core), then version.ini file wouldn't be found by setup.py code. Note that this issue doesn't happen with "proper" (i.e. - sdist-packed) PIP packages, because sdist commands copy the version.ini into every package's archive.

If you want to install Grr from source, then the only supported way at the moment is this: https://grr-doc.readthedocs.io/en/latest/installing-grr-server/from-source.html

[juju4]

On devel-python3, I tried to use a fork as I had issues with cryptography and chipsec versions.

Above error also happens on devel-HEAD suite (https://travis-ci.org/juju4/ansible-grr/jobs/579014806) which use google grr HEAD repository.

Can you confirm that python3 is supported or if ongoing work?

I reviewe the from-source docs and adapt from there. https://grr-doc.readthedocs.io/en/latest/installing-grr-server/from-source.html is not up to date/working. It asked for travis/install_protobuf.sh linux which does not exist anymore in https://github.com/google/grr/tree/master/travis. It seems to be included in install.sh now and does not help when reproducing install.sh in ansible https://travis-ci.org/juju4/ansible-grr/jobs/579923386#L2397

ERROR: No matching distribution found for grr-response-proto==3.3.0post6 (from grr-response-core==3.3.0.post6)"]

from

failed: [localhost] (item=grr/core) => {"ansible_loop_var": "item", "changed": true, "cmd": ["/usr/local/share/env-grr/bin/pip", "install", "-e", "grr/core", "--progress-bar", "off"], "delta": "0:00:02.485717", "end": "2019-09-02 18:30:01.531041", "item": "grr/core", "msg": "non-zero return code", "rc": 1, "start": "2019-09-02 18:29:59.045324", "stderr": "DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support\n  ERROR: Could not find a version that satisfies the requirement grr-response-proto==3.3.0post6 (from grr-response-core==3.3.0.post6) (from versions: 3.2.1.post4, 3.2.1.post7, 3.2.2.post0, 3.2.3.post0, 3.2.3.post1, 3.2.3.post2, 3.2.4.post0, 3.2.4.post2, 3.2.4.post3, 3.2.4.post4, 3.2.4.post5, 3.2.4.post6, 3.2.4.post9, 3.3.0.post0, 3.3.0.post2, 3.3.0.post3, 3.3.0.post4)\nERROR: No matching distribution found for grr-response-proto==3.3.0post6 (from grr-response-core==3.3.0.post6)", "stderr_lines": ["DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support", "  ERROR: Could not find a version that satisfies the requirement grr-response-proto==3.3.0post6 (from grr-response-core==3.3.0.post6) (from versions: 3.2.1.post4, 3.2.1.post7, 3.2.2.post0, 3.2.3.post0, 3.2.3.post1, 3.2.3.post2, 3.2.4.post0, 3.2.4.post2, 3.2.4.post3, 3.2.4.post4, 3.2.4.post5, 3.2.4.post6, 3.2.4.post9, 3.3.0.post0, 3.3.0.post2, 3.3.0.post3, 3.3.0.post4)", "ERROR: No matching distribution found for grr-response-proto==3.3.0post6 (from grr-response-core==3.3.0.post6)"], "stdout": "Obtaining file:///root/grr/grr/core\nCollecting biplist==1.0.3 (from grr-response-core==3.3.0.post6)\n  Downloading https://files.pythonhosted.org/packages/3e/56/2db170a498c9c6545cda16e93c2f2ef9302da44802787b45a8a520d01bdb/biplist-1.0.3.tar.gz\nCollecting configparser==3.5.0 (from grr-response-core==3.3.0.post6)\n  Downloading https://files.pythonhosted.org/packages/7c/69/c2ce7e91c89dc073eb1aa74c0621c3eefbffe8216b3f9af9d3885265c01c/configparser-3.5.0.tar.gz\nCollecting cryptography==2.4.2 (from grr-response-core==3.3.0.post6)\n  Downloading https://files.pythonhosted.org/packages/7f/ba/383b51cc26e3141c689ce988814385c7659f5ba01c4b5f2de38233010b5f/cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl (2.1MB)\nCollecting distro==1.4.0 (from grr-response-core==3.3.0.post6)\n  Downloading https://files.pythonhosted.org/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl\nCollecting fleetspeak==0.1.3 (from grr-response-core==3.3.0.post6)\n  Downloading https://files.pythonhosted.org/packages/18/c5/3ac205a0a3d33034e33f7ced4865fd5096005622654504ce9ee3cd8f2908/fleetspeak-0.1.3.zip (67kB)\nCollecting future==0.17.0 (from grr-response-core==3.3.0.post6)\n  Downloading https://files.pythonhosted.org/packages/85/aa/ba2e24dcb889d7e98733f87515d80b3512418b80ba79d82d2ddcd43fadf3/future-0.17.0.tar.gz (827kB)\nCollecting grr-response-proto==3.3.0post6 (from grr-response-core==3.3.0.post6)", "stdout_lines": ["Obtaining file:///root/grr/grr/core", "Collecting biplist==1.0.3 (from grr-response-core==3.3.0.post6)", "  Downloading https://files.pythonhosted.org/packages/3e/56/2db170a498c9c6545cda16e93c2f2ef9302da44802787b45a8a520d01bdb/biplist-1.0.3.tar.gz", "Collecting configparser==3.5.0 (from grr-response-core==3.3.0.post6)", "  Downloading https://files.pythonhosted.org/packages/7c/69/c2ce7e91c89dc073eb1aa74c0621c3eefbffe8216b3f9af9d3885265c01c/configparser-3.5.0.tar.gz", "Collecting cryptography==2.4.2 (from grr-response-core==3.3.0.post6)", "  Downloading https://files.pythonhosted.org/packages/7f/ba/383b51cc26e3141c689ce988814385c7659f5ba01c4b5f2de38233010b5f/cryptography-2.4.2-cp27-cp27mu-manylinux1_x86_64.whl (2.1MB)", "Collecting distro==1.4.0 (from grr-response-core==3.3.0.post6)", "  Downloading https://files.pythonhosted.org/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl", "Collecting fleetspeak==0.1.3 (from grr-response-core==3.3.0.post6)", "  Downloading https://files.pythonhosted.org/packages/18/c5/3ac205a0a3d33034e33f7ced4865fd5096005622654504ce9ee3cd8f2908/fleetspeak-0.1.3.zip (67kB)", "Collecting future==0.17.0 (from grr-response-core==3.3.0.post6)", "  Downloading https://files.pythonhosted.org/packages/85/aa/ba2e24dcb889d7e98733f87515d80b3512418b80ba79d82d2ddcd43fadf3/future-0.17.0.tar.gz (827kB)", "Collecting grr-response-proto==3.3.0post6 (from grr-response-core==3.3.0.post6)"]}

deb package works fine on xenial and bionic.

[mbushkov]

Thanks for pointing out install_protobuf.sh issue in our docs. install_protobuf.sh is indeed obsolete and not needed. I updated the docs and also added a section on Python 3 support. Please see it here: https://grr-doc.readthedocs.io/en/latest/installing-grr-server/from-source.html#installing-python-3-grr-version-from-source

By now GRR codebase should be fully Python 3 compatible: unit and end-to-end tests are passing in a properly setup Python 3 GRR environment. However, the first Python 3 GRR release is expected in Dec. 2019. Until that’s done, Python 3 support should be treated as experimental: unforeseen packaging issues not covered by our tests are possible.

The issue here - https://travis-ci.org/juju4/ansible-grr/jobs/579923386#L2397 seems to be caused by an issue with protobufs compilation. grr/proto/setup.py should automatically install grpcio-tools package into the virtualenv.

I can see this happening in the log:

Collecting grpcio-tools==1.17.1

However later there's this:

Installing collected packages: enum34, futures, grpcio, grpcio-tools
Successfully installed enum34-1.1.6 futures-3.3.0 grpcio-1.23.0 grpcio-tools-1.17.1
/usr/bin/python: No module named grpc_tools
Compiling /root/grr/grr/proto/grr_response_proto/deprecated.proto with (cwd: /root/grr/grr/proto): python -m grpc_tools.protoc --python_out /root/grr/grr/proto --proto_path=/root/grr/grr/proto /root/grr/grr/proto/grr_response_proto/deprecated.proto

It seems that the Python binary used to compile the protos doesn't come from the virtualenv, but instead from /usr/bin. Is virtualenv properly activated with usr/local/share/env-grr/bin/activate.sh or do you just run /usr/local/share/env-grr/bin/pip? If the virtualenv is not properly activated, PATH environment variable won't be modified and running "python" from within grr/proto/setup.py would trigger system's Python instead of the virtualenv's one.