Open AnatoliyNelyubin opened 2 years ago
in the installed version google-cloud-logging-2.3.1 the attribute shouldn't be missing (see source) I am thinking that there might be another library around that is making google.cloud.logging a namespace package, can you try installing in a new virtualenv and verify if you get this same error?
Thank's for your answer! I am afraid that the issues started appearing even on the installation phase. On a completely fresh Ubuntu 22.04 install -r requirements.txt
does not properly install all packages:
Collecting typing-extensions>=4.0.1
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
typing-extensions>=4.0.1 from https://files.pythonhosted.org/packages/ed/d6/2afc375a8d55b8be879d6b4986d4f69f01115e795e36827fd3a40166028b/typing_extensions-4.3.0-py3-none-any.whl#sha256=25642c956049920a5aa49edcdd6ab1e06d7e5d467fc00e0506c44ac86fbfca02 (from azure-core==1.24.2->-r requirements.txt (line 95))
After forcefully installing
pip install https://files.pythonhosted.org/packages/ed/d6/2afc375a8d55b8be879d6b4986d4f69f01115e795e36827fd3a40166028b/typing_extensions-4.3.0-py3-none-any.whl#sha256=25642c956049920a5aa49edcdd6ab1e06d7e5d467fc00e0506c44ac86fbfca02
there is another error:
Collecting googleapis-common-protos[grpc]<2.0.0dev,>=1.56.0
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
googleapis-common-protos[grpc]<2.0.0dev,>=1.56.0 from https://files.pythonhosted.org/packages/e2/fd/d9efa2085bd762fba3a637eb3e36d76d72eb6e083d170aeaca65a75f1f9c/googleapis_common_protos-1.56.4-py2.py3-none-any.whl#sha256=8eb2cbc91b69feaf23e32452a7ae60e791e09967d81d4fcc7fc388182d1bd394 (from grpc-google-iam-v1==0.12.4->-r requirements.txt (line 421))
after installing:
pip install https://files.pythonhosted.org/packages/e2/fd/d9efa2085bd762fba3a637eb3e36d76d72eb6e083d170aeaca65a75f1f9c/googleapis_common_protos-1.56.4-py2.py3-none-any.whl#sha256=8eb2cbc91b69feaf23e32452a7ae60e791e09967d81d4fcc7fc388182d1bd394
Collecting googleapis-common-protos==1.56.4
Downloading googleapis_common_protos-1.56.4-py2.py3-none-any.whl (211 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.7/211.7 KB 4.6 MB/s eta 0:00:00
Collecting protobuf<5.0.0dev,>=3.15.0
Downloading protobuf-4.21.5-cp37-abi3-manylinux2014_x86_64.whl (408 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 408.4/408.4 KB 8.4 MB/s eta 0:00:00
Installing collected packages: protobuf, googleapis-common-protos
Successfully installed googleapis-common-protos-1.56.4 protobuf-4.21.5
This error appears:
python -c 'import google.cloud.logging; client = google.cloud.logging.Client()'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'google.cloud.logging'
I noticed that the package gax-google-logging-v2 breaks the config.
(forensic) anatolii@siftworkstation: ~/forensic/dftimewolf
$ pip list |grep logging
gax-google-logging-v2 0.8.3
google-cloud-logging 2.3.1
grpc-google-logging-v2 0.8.1
(forensic) anatolii@siftworkstation: ~/forensic/dftimewolf
$ python -c 'import google.cloud.logging; client = google.cloud.logging.Client()'
Traceback (most recent call last):
File "<string>", line 1, in <module>
AttributeError: module 'google.cloud.logging' has no attribute 'Client'
(forensic) anatolii@siftworkstation: ~/forensic/dftimewolf
$ pip uninstall gax-google-logging-v2
Found existing installation: gax-google-logging-v2 0.8.3
Uninstalling gax-google-logging-v2-0.8.3:
Would remove:
/home/anatolii/forensic/lib/python3.9/site-packages/gax_google_logging_v2-0.8.3-nspkg.pth
/home/anatolii/forensic/lib/python3.9/site-packages/gax_google_logging_v2-0.8.3.dist-info/*
/home/anatolii/forensic/lib/python3.9/site-packages/google/cloud/logging/v2/*
Proceed (y/n)? y
Successfully uninstalled gax-google-logging-v2-0.8.3
(forensic) anatolii@siftworkstation: ~/forensic/dftimewolf
$ python -c 'import google.cloud.logging; client = google.cloud.logging.Client()'
(forensic) anatolii@siftworkstation: ~/forensic/dftimewolf
Is this package really necessary? What is using it?
A bit late to the party here, apologies - we moved away from pip and pipenv in favor of poetry, this wasn't reflected in the docs (https://github.com/log2timeline/dftimewolf/pull/675). Please use poetry to install dftimewolf, otherwise you migth run into dependency conflicts.
Still doesn't work, though used poetry for proper installation:
$ dftimewolf gcp_logging_collect koz-bad 'timestamp>="2022-11-28T19:00:00"'
Messages
[ dftimewolf ] Debug log: /tmp/dftimewolf-run-20221128_195510_ximdl1al.log
[ dftimewolf ] An unknown error occurred in module GCPLogsCollector: module 'google.cloud.logging' has no attribute 'Client'
[ dftimewolf ] Critical error found. Aborting.
$ python3 -c 'import google.cloud.logging; client = google.cloud.logging.Client()'
Traceback (most recent call last):
File "<string>", line 1, in <module>
AttributeError: module 'google.cloud.logging' has no attribute 'Client'
$ pip uninstall gax-google-logging-v2
Found existing installation: gax-google-logging-v2 0.8.3
Uninstalling gax-google-logging-v2-0.8.3:
Would remove:
/home/anatolii/.cache/pypoetry/virtualenvs/dftimewolf-pBwI8mxo-py3.10/lib/python3.10/site-packages/gax_google_logging_v2-0.8.3-py3.10-nspkg.pth
/home/anatolii/.cache/pypoetry/virtualenvs/dftimewolf-pBwI8mxo-py3.10/lib/python3.10/site-packages/gax_google_logging_v2-0.8.3.dist-info/*
/home/anatolii/.cache/pypoetry/virtualenvs/dftimewolf-pBwI8mxo-py3.10/lib/python3.10/site-packages/google/cloud/logging/v2/*
Proceed (Y/n)? Y
Successfully uninstalled gax-google-logging-v2-0.8.3
$ python3 -c 'import google.cloud.logging; client = google.cloud.logging.Client()'
$ dftimewolf gcp_logging_collect koz-bad 'timestamp>="2022-11-28T19:00:00"'
Messages
[ dftimewolf ] Debug log: /tmp/dftimewolf-run-20221128_200635_shjq4jkr.log
[ GCPLogsCollector ] Downloaded logs to /tmp/tmp69hu7a6h.jsonl
$ wc -l /tmp/tmp69hu7a6h.jsonl
26593 /tmp/tmp69hu7a6h.jsonl
Still some troubles with the package gax-google-logging-v2-0.8.3
I dug into this a bit more, and it seems that gax-google-logging-v2-0.8.3
is a third or fourth party dependency of Turbinia. They're currently changing the API client to drop all GCP requirements, which should alleviate this requirement and hopefully fix this. I should check whether the Turbinia recipes fail if this lib isn't installed.
Turbinia depdends on psq
, which has been unmaintained for the past 3 years, and which in turns depends on gax-google-logging-v2
, which has been unmaintained for even longer (2016?)
The best workaround for now is to pip uninstall ax-google-logging-v2
like @AnatoliyNelyubin recommended. Still investigating whether that will break turbinia modules.
After clean installation of all requirements
pip install -r requirements.txt
in a new python environment:GCP logging in this venv stops working: