[kubernetes] ERROR __main__.py L237: Initialization failure: module 'collections' has no attribute 'Callable'

BafDyce commented 2 years ago

Describe the bug

Running scout kubernetes against an OpenShift cluster running on AWS (self-managed), results in the following error:

(virtualenv-develop) [bob@XXX ScoutSuite]$ scout kubernetes --debug
2022-09-27 10:21:46 XXX scout[3734] INFO Launching Scout
2022-09-27 10:21:46 XXX scout[3734] INFO Authenticating to cloud provider
2022-09-27 10:21:47 XXX scout[3734] ERROR __main__.py L237: Initialization failure: module 'collections' has no attribute 'Callable'
Traceback (most recent call last):
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/ScoutSuite/__main__.py", line 237, in _run
    cloud_provider = get_provider(provider=provider,
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/ScoutSuite/providers/__init__.py", line 37, in get_provider
    provider_object = get_provider_object(provider)
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/ScoutSuite/providers/__init__.py", line 11, in get_provider_object
    provider_module = __import__(f'ScoutSuite.providers.{provider}.provider', fromlist=[provider_class])
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/ScoutSuite/providers/kubernetes/provider.py", line 7, in <module>
    from ScoutSuite.providers.kubernetes.resources.workload import Workload
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/ScoutSuite/providers/kubernetes/resources/workload.py", line 1, in <module>
    from ScoutSuite.providers.kubernetes.resources.base import KubernetesCompositeResources, KubernetesResourcesWithFacade
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/ScoutSuite/providers/kubernetes/resources/base.py", line 3, in <module>
    from ScoutSuite.providers.kubernetes.facade import KubernetesFacade
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/ScoutSuite/providers/kubernetes/facade/__init__.py", line 2, in <module>
    from ScoutSuite.providers.gcp.facade.base import GCPFacade
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/ScoutSuite/providers/gcp/facade/base.py", line 5, in <module>
    from ScoutSuite.providers.gcp.facade.basefacade import GCPBaseFacade
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/ScoutSuite/providers/gcp/facade/basefacade.py", line 9, in <module>
    from googleapiclient import http
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/googleapiclient/http.py", line 53, in <module>
    from googleapiclient import _auth
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/googleapiclient/_auth.py", line 34, in <module>
    import oauth2client.client
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/oauth2client/client.py", line 39, in <module>
    from oauth2client import transport
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/oauth2client/transport.py", line 285, in <module>
    _CACHED_HTTP = httplib2.Http(MemoryCache())
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/httplib2shim/__init__.py", line 117, in __init__
    pool = self._make_pool(proxy_info=proxy_info)
  File "/home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/httplib2shim/__init__.py", line 45, in _default_make_pool
    if isinstance(proxy_info, collections.Callable):
AttributeError: module 'collections' has no attribute 'Callable'

To Reproduce

The ~/.kube/config file is valid and both oc and kubectl work fine against the cluster
I also tried with the --cluster-provider eks option, which leads to the same error
I tried with both, existing and removed ~/.aws/{config,credentials}
ScoutSuite version: Latest develop branch (cf44bebc428cd40fa40acac72e31d8a63cb42eaf) with fresh virtualenv as instructed at https://research.nccgroup.com/2022/09/22/tool-release-project-kubescout-adding-kubernetes-support-to-scout-suite/

Additional context

$ python --version
Python 3.10.7
$ pip --version
pip 22.0.4 from /home/bob/tools/ScoutSuite/virtualenv-develop/lib/python3.10/site-packages/pip (python 3.10)
$ pip freeze
aliyun-python-sdk-actiontrail==2.2.0
aliyun-python-sdk-core==2.13.36
aliyun-python-sdk-ecs==4.24.25
aliyun-python-sdk-kms==2.16.0
aliyun-python-sdk-ocs==0.0.4
aliyun-python-sdk-ram==3.2.0
aliyun-python-sdk-rds==2.7.1
aliyun-python-sdk-sts==3.1.0
aliyun-python-sdk-vpc==3.0.25
asyncio-throttle==0.1.1
autocommand==2.2.1
azure-common==1.1.28
azure-core==1.25.1
azure-identity==1.5.0
azure-mgmt-authorization==1.0.0
azure-mgmt-compute==18.2.0
azure-mgmt-core==1.3.2
azure-mgmt-keyvault==8.0.0
azure-mgmt-monitor==2.0.0
azure-mgmt-network==17.1.0
azure-mgmt-rdbms==8.0.0
azure-mgmt-redis==12.0.0
azure-mgmt-resource==15.0.0
azure-mgmt-security==1.0.0
azure-mgmt-sql==1.0.0
azure-mgmt-storage==16.0.0
azure-mgmt-web==1.0.0
boto3==1.24.81
botocore==1.27.81
cachetools==5.2.0
certifi==2022.9.24
cffi==1.15.1
charset-normalizer==2.1.1
cheroot==8.6.0
CherryPy==18.8.0
cherrypy-cors==1.6
circuitbreaker==1.4.0
coloredlogs==10.0
crcmod==1.7
cryptography==37.0.2
google-api-core==1.33.1
google-api-python-client==2.62.0
google-auth==2.11.1
google-auth-httplib2==0.1.0
google-cloud-appengine-logging==1.1.4
google-cloud-audit-log==0.2.4
google-cloud-container==2.12.0
google-cloud-core==2.3.2
google-cloud-iam==2.8.2
google-cloud-kms==1.3.0
google-cloud-logging==3.2.2
google-cloud-monitoring==1.1.0
google-cloud-resource-manager==1.6.1
google-cloud-storage==2.5.0
google-crc32c==1.5.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.4
grpc-google-iam-v1==0.12.4
grpcio==1.49.1
grpcio-status==1.48.2
httpagentparser==1.9.3
httplib2==0.20.4
httplib2shim==0.0.3
humanfriendly==10.0
idna==3.4
inflect==6.0.0
isodate==0.6.1
jaraco.classes==3.2.3
jaraco.collections==3.5.2
jaraco.context==4.1.2
jaraco.functools==3.5.2
jaraco.text==3.9.1
jmespath==0.10.0
kubernetes==24.2.0
more-itertools==8.14.0
msal==1.19.0
msal-extensions==0.3.1
msgraph-core==0.2.2
msrest==0.7.1
netaddr==0.8.0
oauth2client==4.1.3
oauthlib==3.2.1
oci==2.84.0
oss2==2.16.0
policyuniverse==1.5.0.20220613
portalocker==2.5.1
portend==3.1.0
proto-plus==1.22.1
protobuf==3.20.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pycryptodome==3.15.0
pydantic==1.10.2
PyJWT==2.5.0
pyOpenSSL==22.0.0
pyparsing==3.0.9
python-dateutil==2.8.0
pytz==2022.2.1
PyYAML==6.0
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.9
s3transfer==0.6.0
ScoutSuite @ file:///home/bob/tools/ScoutSuite
six==1.16.0
sqlitedict==2.0.0
tempora==5.0.2
types-cryptography==3.3.23
typing_extensions==4.3.0
uritemplate==4.1.1
urllib3==1.26.12
websocket-client==1.4.1
zc.lockfile==2.0

superpoussin22 commented 2 years ago

I found this : https://github.com/GoogleCloudPlatform/httplib2shim/pull/16/files

liyun-li commented 2 years ago

Thanks for the debugging information! I will work out a pull request.

liyun-li commented 2 years ago

@BafDyce: I just had some time to check out the debug log. I think downgrading Python to 3.7 or something should work for now. It might take some time for this to be resolved.

MartinPetkov commented 2 years ago

For what it's worth, I manually applied the change in https://github.com/nccgroup/ScoutSuite/issues/1463#issuecomment-1265486944 and scout kubernetes worked against a GKE cluster.

liyun-li commented 2 years ago

Folks, try pulling the develop branch. The PR is merged. If it fixes your bug, please close the issue.

superpoussin22 commented 2 years ago

Folks, try pulling the develop branch. The PR is merged. If it fixes your bug, please close the issue.

hello, same issue , the patch is needed https://github.com/GoogleCloudPlatform/httplib2shim/pull/16/files but this is not on ScoutSuite side , perhaps we must stay away from Python 3.10 until they correct

liyun-li commented 2 years ago

https://github.com/GoogleCloudPlatform/httplib2shim The library is going out of maintenance Nov. 1. It's best for ScoutSuite to move away from it, too.

BafDyce commented 2 years ago

Folks, try pulling the develop branch. The PR is merged. If it fixes your bug, please close the issue.

I'll try as soon as I get my hands on a GKE cluster again. Nevertheless, thanks for the quick fix & feedback.

liyun-li commented 7 months ago

Closing because I think this is resolved

nccgroup / ScoutSuite

[kubernetes] ERROR main.py L237: Initialization failure: module 'collections' has no attribute 'Callable' #1463

nccgroup / ScoutSuite

[kubernetes] ERROR __main__.py L237: Initialization failure: module 'collections' has no attribute 'Callable' #1463

[kubernetes] ERROR main.py L237: Initialization failure: module 'collections' has no attribute 'Callable' #1463