Closed oliviermtl closed 4 months ago
Hi @oliviermtl could you please share the dependencies you have installed in your environment?
Hi @oliviermtl, it could also be because the compliance flag is --compliance
and not -c
. Let us know if that is the issue, thanks for using Prowler! π
Hi @oliviermtl, it could also be because the compliance flag is
--compliance
and not-c
. Let us know if that is the issue, thanks for using Prowler! π
it works fine locally with -c
Hi @oliviermtl could you please share the dependencies you have installed in your environment?
Successfully installed Flask-3.0.3 Jinja2-3.1.4 MarkupSafe-2.1.5 PyJWT-2.8.0 Werkzeug-3.0.3 XlsxWriter-3.2.0 about-time-4.2.1 aiohttp-3.9.5 aiosignal-1.3.1 alive-progress-3.1.5 anyio-4.4.0 attrs-23.2.0 awsipranges-0.3.3 azure-common-1.1.28 azure-core-1.30.1 azure-identity-1.16.0 azure-keyvault-keys-4.9.0 azure-mgmt-applicationinsights-4.0.0 azure-mgmt-authorization-4.0.0 azure-mgmt-compute-31.0.0 azure-mgmt-containerservice-30.0.0 azure-mgmt-core-1.4.0 azure-mgmt-cosmosdb-9.5.0 azure-mgmt-keyvault-10.3.0 azure-mgmt-monitor-6.0.2 azure-mgmt-network-25.4.0 azure-mgmt-rdbms-10.1.0 azure-mgmt-resource-23.1.1 azure-mgmt-security-7.0.0 azure-mgmt-sql-3.0.1 azure-mgmt-storage-21.1.0 azure-mgmt-subscription-3.1.1 azure-mgmt-web-7.2.0 azure-storage-blob-12.20.0 blinker-1.8.2 boto3-1.34.109 botocore-1.34.113 cachetools-5.3.3 cffi-1.16.0 charset-normalizer-3.3.2 click-8.1.7 click-plugins-1.1.1 colorama-0.4.6 cryptography-42.0.7 dash-2.17.0 dash-bootstrap-components-1.6.0 dash-core-components-2.0.0 dash-html-components-2.0.0 dash-table-5.0.0 deprecated-1.2.14 detect-secrets-1.5.0 dnspython-2.6.1 email-validator-2.1.1 frozenlist-1.4.1 google-api-core-2.19.0 google-api-python-client-2.130.0 google-auth-2.29.0 google-auth-httplib2-0.2.0 googleapis-common-protos-1.63.0 grapheme-0.6.0 h11-0.14.0 h2-4.1.0 hpack-4.0.0 httpcore-1.0.5 httplib2-0.22.0 httpx-0.27.0 hyperframe-6.0.1 idna-3.7 importlib-metadata-7.1.0 isodate-0.6.1 itsdangerous-2.2.0 jsonschema-4.22.0 jsonschema-specifications-2023.12.1 kubernetes-29.0.0 microsoft-kiota-abstractions-1.3.3 microsoft-kiota-authentication-azure-1.0.0 microsoft-kiota-http-1.3.1 microsoft-kiota-serialization-form-0.1.0 microsoft-kiota-serialization-json-1.2.0 microsoft-kiota-serialization-multipart-0.1.0 microsoft-kiota-serialization-text-1.0.0 msal-1.28.0 msal-extensions-1.1.0 msgraph-core-1.0.0 msgraph-sdk-1.4.0 msrest-0.7.1 multidict-6.0.5 nest-asyncio-1.6.0 numpy-1.26.4 oauthlib-3.2.2 opentelemetry-api-1.25.0 opentelemetry-sdk-1.25.0 opentelemetry-semantic-conventions-0.46b0 packaging-24.0 pandas-2.2.2 pendulum-3.0.0 plotly-5.22.0 portalocker-2.8.2 proto-plus-1.23.0 protobuf-4.25.3 prowler-4.2.1 py-ocsf-models-0.1.1 pyasn1-0.6.0 pyasn1-modules-0.4.0 pycparser-2.22 pydantic-1.10.15 pyparsing-3.1.2 pytz-2024.1 referencing-0.35.1 requests-2.32.3 requests-file-2.1.0 requests-oauthlib-2.0.0 retrying-1.3.4 rpds-py-0.18.1 rsa-4.9 schema-0.7.7 shodan-1.31.0 slack-sdk-3.27.2 sniffio-1.3.1 std-uritemplate-0.0.57 tabulate-0.9.0 tenacity-8.3.0 time-machine-2.14.1 tldextract-5.1.2 typing-extensions-4.12.0 tzdata-2024.1 tzlocal-5.2 uritemplate-4.1.1 websocket-client-1.8.0 wrapt-1.16.0 yarl-1.9.4 zipp-3.19.0
Solved by using python 3.11 on Amazon-Linux-2023.
This is the BuildSpec from my cloudformation template :
BuildSpec: |
version: 0.2
phases:
install:
runtime-versions:
python: 3.11
commands:
- echo "Installing Prowler and dependencies..."
- pip3 install detect-secrets alive_progress colorama jsonschema tabulate pydantic==1.10.15 azure-identity azure.mgmt==4.0.0 schema slack_sdk pytz tzlocal msgraph-core==1.0.0 msgraph-sdk==1.4.0 py_ocsf_models
- pip3 install --upgrade google-api-python-client
- yum -y install jq
- curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
- unzip awscliv2.zip
- ./aws/install --update
- git clone --depth 1 --branch 4.2.1 https://github.com/prowler-cloud/prowler.git
That's great news @oliviermtl π
Where did you grab the cloudformation templace? Is it something built by you or present in this repository? If it is the latter it'd be great to include that code into the repository. We can do that or help you to do that.
Thanks!
The template was already here when I took over the position so I cannot confirm where it comes from. Happy to push a PR with the template.
Awesome! You can include it under contrib/aws
. Thanks!
~/Code/prowler cloudformation-template > git push --set-upstream origin cloudformation-template
remote: Permission to prowler-cloud/prowler.git denied to oliviermtl.
fatal: unable to access 'https://github.com/prowler-cloud/prowler/': The requested URL returned error: 403
What am I missing here ? I git clone, followed all the dev guide, created a branch
You cannot contribute directly to the repository, you have to create a fork, push the code to a branch in your forked repository and then open the pull request from your forked branch to Prowler master
.
ok, you might to want to include this information at the top of the developer guide.
ok, you might to want to include this information at the top of the developer guide.
Sure, good catch!
Hi @oliviermtl, we've just included that in our documentation, thanks!
Hi @oliviermtl are you finally planning to open up a PR with the CodeBuild template? Thanks!
Hi @oliviermtl I'm closing this issue. Please open the PR you mention once is ready, we are looking forward to have that!
Thanks for using Prowler π
Steps to Reproduce
./prowler.py --output-formats {csv,json-asff,json-ocsf,html} -c cis_2.0_aws -w arn:aws:dynamodb:ap-southeast-1:xxxxxxxxxxx:table/prowler_allow_list
Expected behavior
run correctly
Actual Result with Screenshots or Logs
[Container] 2024/05/31 07:03:52.594491 Running command ./prowler.py --output-formats {csv,json-asff,json-ocsf,html} -c cis_2.0_aws -w arn:aws:dynamodb:ap-southeast-1:xxxxxxxxxxx:table/prowler_allow_list Traceback (most recent call last): File "/codebuild/output/src513582655/src/prowler/./prowler.py", line 5, in
from prowler.main import prowler
File "/codebuild/output/src513582655/src/prowler/prowler/main.py", line 10, in
from prowler.lib.check.check import (
File "/codebuild/output/src513582655/src/prowler/prowler/lib/check/check.py", line 18, in
from prowler.lib.check.compliance_models import load_compliance_framework
File "/codebuild/output/src513582655/src/prowler/prowler/lib/check/compliance_models.py", line 80, in
class CIS_Requirement_Attribute(BaseModel):
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 202, in new
complete_model_class(
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 539, in complete_model_class
schema = cls.get_pydantic_core_schema(cls, handler)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/main.py", line 626, in get_pydantic_core_schema
return handler(source)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in call
schema = self._handler(source_type)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 502, in generate_schema
schema = self._generate_schema_inner(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 753, in _generate_schema_inner
return self._model_schema(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 580, in _model_schema
{k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 580, in
{k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 916, in _generate_md_field_schema
common_field = self._common_field_schema(name, field_info, decorators)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 1081, in _common_field_schema
schema = self._apply_annotations(
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 1820, in _apply_annotations
schema = get_inner_schema(source_type)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in call
schema = self._handler(source_type)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 1801, in inner_handler
schema = self._generate_schema_inner(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 758, in _generate_schema_inner
return self.match_type(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 844, in match_type
return self._unknown_type_schema(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 405, in _unknown_type_schema
raise PydanticSchemaGenerationError(
pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'prowler.lib.check.compliance_models.CIS_Requirement_Attribute_Profile'>. Set
arbitrary_types_allowed=True
in the model_config to ignore this error or implement__get_pydantic_core_schema__
on your type to fully support it.If you got this error by calling handler() within
__get_pydantic_core_schema__
then you likely need to callhandler.generate_schema(<some type>)
since we do not call__get_pydantic_core_schema__
on<some type>
otherwise to avoid infinite recursion.How did you install Prowler?
Cloning the repository from github.com (git clone)
Environment Resource
Cloudformation template
OS used
Amazon Linux 2
Prowler version
4.2.1
Pip version
21.1.2
Context
I am using the same cloudformation template that I was using with prowler v2, except for python 3.6->3.9