search

Checkmarx / kics

Find security vulnerabilities, compliance issues, and infrastructure misconfigurations early in the development cycle of your infrastructure-as-code with KICS by Checkmarx.
https://kics.io
Apache License 2.0
2.08k stars 308 forks source link

Process in CodeBuild hangs #5055

Open ALutchko opened 2 years ago

ALutchko commented 2 years ago

Expected Behavior

it runs

Actual Behavior

it hangs for more than 10 mins, no error messages

Steps to Reproduce the Problem

Run the following commands in CodeBuild:

[Container] 2022/03/25 18:39:00 Running command echo "kics_img_name $kics_img_name"
kics_img_name checkmarx/kics:v1.5.4

[Container] 2022/03/25 18:39:17 Running command docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation
Unable to find image 'checkmarx/kics:v1.5.4' locally
v1.5.4: Pulling from checkmarx/kics
Digest: sha256:628ed3b084e6ace14838b2772bf261187a606066d7bb31ed77087beb66ed0847
Status: Downloaded newer image for checkmarx/kics:v1.5.4
6:39PM DBG Could not find string flag ci
6:39PM DBG console.scan()
6:39PM DBG console.scan()

                   .0MO.                                    
                   OMMMx                                    
                   ;NMX;                                    
                    ...           ...              ....     
WMMMd     cWMMM0.  KMMMO      ;xKWMMMMNOc.     ,xXMMMMMWXkc.
WMMMd   .0MMMN:    KMMMO    :XMMMMMMMMMMMWl   xMMMMMWMMMMMMl
WMMMd  lWMMMO.     KMMMO   xMMMMKc...'lXMk   ,MMMMx   .;dXx 
WMMMd.0MMMX;       KMMMO  cMMMMd        '    'MMMMNl'       
WMMMNWMMMMl        KMMMO  0MMMN               oMMMMMMMXkl.  
WMMMMMMMMMMo       KMMMO  0MMMX                .ckKWMMMMMM0.
WMMMMWokMMMMk      KMMMO  oMMMMc              .     .:OMMMM0
WMMMK.  dMMMM0.    KMMMO   KMMMMx'    ,kNc   :WOc.    .NMMMX
WMMMd    cWMMMX.   KMMMO    kMMMMMWXNMMMMMd .WMMMMWKO0NMMMMl
WMMMd     ,NMMMN,  KMMMO     'xNMMMMMMMNx,   .l0WMMMMMMMWk, 
xkkk:      ,kkkkx  okkkl        ;xKXKx;          ;dOKKkc    

Scanning with Keeping Infrastructure as Code Secure v1.5.4

6:39PM INF Scanning with Keeping Infrastructure as Code Secure v1.5.4
6:39PM DBG storage.NewMemoryStorage()
6:39PM DBG Looking for queries in executable path and in current work directory
6:39PM DBG helpers.GetDefaultQueryPath()
6:39PM DBG helpers.GetExecutableDirectory()
6:39PM DBG Queries found in /app/bin/assets/queries
6:39PM DBG source.NewFilesystemSource()
6:39PM DBG engine.NewInspector()
6:39PM DBG Custom library not provided. Loading embedded library instead
6:39PM DBG Custom library not provided. Loading embedded library instead
6:39PM DBG Could not open embedded library data for cloudFormation platform
6:39PM INF Inspector initialized, number of queries=496
6:39PM INF Query execution timeout=1m0s
6:39PM DBG provider.NewFileSystemSourceProvider()
6:39PM DBG parser.NewBuilder()
6:39PM DBG resolver.Add()
6:39PM DBG resolver.Build()

here it hangs, you have to stop CodeBuild manually. Could you help, please?

ALutchko commented 2 years ago

ping :( If I can get more verbose debug please let me know.

rafaela-soares commented 2 years ago

Hello, @ALutchko!

Thank you so much for reaching us 😊

Can you give us details about your project, please?

ALutchko commented 2 years ago

Hello @rafaela-soares, What kind of information do you need? We're building fully on AWS/CloudFormation.

rafaela-soares commented 2 years ago

Hi, @ALutchko! Can you provide us with the project that you are running against KICS, please?

ALutchko commented 2 years ago

I am really sorry but no, it is not OpenSource. If you provide me with more details what kind of information is needed then most likely I'll be able to provide it.

Buildspec for CodeBuild looks like:

      - ecr_domain="$CURR_ACC_ID.dkr.ecr.$REGION.amazonaws.com"
      - echo "ecr_domain $ecr_domain"
      - kics_img_name=$(grep checkmarx/kics ../ecr_images.txt)
      - aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $CURR_ACC_ID.dkr.ecr.$REGION.amazonaws.com/checkmarx
      - echo "kics_img_name $kics_img_name"
      - docker pull $ecr_domain/$kics_img_name
      - cd ../..  # repo root
      - pwd  # check we're in the correct place
      - docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation

The repo root has a few directories which contain CloudFormation yamls. Above "path" is literally the word "path", no substitution in this message.

rafaela-soares commented 2 years ago

I understand 😊 Let's try to figure out what is happening.

ALutchko commented 2 years ago

@rafaela-soares , thank you very much for the quick response.

rafaela-soares commented 2 years ago
ALutchko commented 2 years ago

The result (I've added mem just before):


[Container] 2022/04/12 09:57:32 Running command free -h
              total        used        free      shared  buff/cache   available
Mem:          3.6Gi       709Mi       403Mi       0.0Ki       2.5Gi       2.7Gi
Swap:            0B          0B          0B
[Container] 2022/04/12 09:57:32 Running command docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation --profiling MEM
Unable to find image 'checkmarx/kics:v1.5.4' locally
v1.5.4: Pulling from checkmarx/kics
Digest: sha256:628ed3b084e6ace14838b2772bf261187a606066d7bb31ed77087beb66ed0847
Status: Downloaded newer image for checkmarx/kics:v1.5.4
9:57AM DBG Could not find string flag ci
9:57AM DBG console.scan()
9:57AM DBG console.scan()

                   .0MO.                                    
                   OMMMx                                    
                   ;NMX;                                    
                    ...           ...              ....     
WMMMd     cWMMM0.  KMMMO      ;xKWMMMMNOc.     ,xXMMMMMWXkc.
WMMMd   .0MMMN:    KMMMO    :XMMMMMMMMMMMWl   xMMMMMWMMMMMMl
WMMMd  lWMMMO.     KMMMO   xMMMMKc...'lXMk   ,MMMMx   .;dXx 
WMMMd.0MMMX;       KMMMO  cMMMMd        '    'MMMMNl'       
WMMMNWMMMMl        KMMMO  0MMMN               oMMMMMMMXkl.  
WMMMMMMMMMMo       KMMMO  0MMMX                .ckKWMMMMMM0.
WMMMMWokMMMMk      KMMMO  oMMMMc              .     .:OMMMM0
WMMMK.  dMMMM0.    KMMMO   KMMMMx'    ,kNc   :WOc.    .NMMMX
WMMMd    cWMMMX.   KMMMO    kMMMMMWXNMMMMMd .WMMMMWKO0NMMMMl
WMMMd     ,NMMMN,  KMMMO     'xNMMMMMMMNx,   .l0WMMMMMMMWk, 
xkkk:      ,kkkkx  okkkl        ;xKXKx;          ;dOKKkc    

Scanning with Keeping Infrastructure as Code Secure v1.5.4

9:57AM INF Scanning with Keeping Infrastructure as Code Secure v1.5.4
9:57AM DBG storage.NewMemoryStorage()
9:57AM DBG Looking for queries in executable path and in current work directory
9:57AM DBG helpers.GetDefaultQueryPath()
9:57AM DBG helpers.GetExecutableDirectory()
9:57AM DBG Queries found in /app/bin/assets/queries
9:57AM DBG source.NewFilesystemSource()
9:57AM DBG engine.NewInspector()
9:57AM DBG Started MEM profiling for get_queries
9:57AM DBG Custom library not provided. Loading embedded library instead
9:57AM DBG Custom library not provided. Loading embedded library instead
9:57AM DBG Could not open embedded library data for cloudFormation platform
9:58AM DBG Stopped MEM profiling for get_queries
9:58AM INF Total MEM usage for get_queries: 226.13MB
9:58AM INF Inspector initialized, number of queries=496
9:58AM INF Query execution timeout=1m0s
9:58AM DBG provider.NewFileSystemSourceProvider()
9:58AM DBG parser.NewBuilder()
9:58AM DBG resolver.Add()
9:58AM DBG resolver.Build()
9:58AM DBG Started MEM profiling for prepare_sources

it hanged in this state for 15 mins then I stopped the build.

rafaela-soares commented 2 years ago

Hello again, @ALutchko!

I think I finally reproduced the issue!

With 3 GB memory, 2 vCPUs:

With 7 GB memory, 4 vCPUs:

P.S. I used this "project": https://github.com/rafaela-soares/codebuild-kics-scan

We will try to understand why there is a difference in the scan time and fix it. There is a strange behaviour when using the flag --type. We apologize for it.

rafaela-soares commented 2 years ago

@ALutchko, can you try with 7 GB memory, 4 vCPUs and let us know if the build succeeded, please?

ALutchko commented 2 years ago

Have just tried, waited 10 mins, the same result: Scanning with Keeping Infrastructure as Code Secure v1.5.4

7:39PM INF Scanning with Keeping Infrastructure as Code Secure v1.5.4
7:39PM DBG storage.NewMemoryStorage()
7:39PM DBG Looking for queries in executable path and in current work directory
7:39PM DBG helpers.GetDefaultQueryPath()
7:39PM DBG helpers.GetExecutableDirectory()
7:39PM DBG Queries found in /app/bin/assets/queries
7:39PM DBG source.NewFilesystemSource()
7:39PM DBG engine.NewInspector()
7:39PM DBG Started MEM profiling for get_queries
7:39PM DBG Custom library not provided. Loading embedded library instead
7:39PM DBG Custom library not provided. Loading embedded library instead
7:39PM DBG Could not open embedded library data for cloudFormation platform
7:40PM DBG Stopped MEM profiling for get_queries
7:40PM INF Total MEM usage for get_queries: 252.36MB
7:40PM INF Inspector initialized, number of queries=496
7:40PM INF Query execution timeout=1m0s
7:40PM DBG provider.NewFileSystemSourceProvider()
7:40PM DBG parser.NewBuilder()
7:40PM DBG resolver.Add()
7:40PM DBG resolver.Build()
7:40PM DBG Started MEM profiling for prepare_sources

What will be the application behavior if I just remove --type CloudFormation? It will scan for all possible types and try to guess?

rafaela-soares commented 2 years ago

Yes. Can you try without the flag --type, please?

ALutchko commented 2 years ago

The same. Switched off after 27 min from start. Also, I've updated to 1.5.5

9:25AM INF Scanning with Keeping Infrastructure as Code Secure v1.5.5
9:25AM DBG storage.NewMemoryStorage()
9:25AM DBG Looking for queries in executable path and in current work directory
9:25AM DBG helpers.GetDefaultQueryPath()
9:25AM DBG helpers.GetExecutableDirectory()
9:25AM DBG Queries found in /app/bin/assets/queries
9:25AM DBG source.NewFilesystemSource()
9:25AM DBG engine.NewInspector()
9:25AM DBG Started MEM profiling for get_queries
9:25AM DBG Custom library not provided. Loading embedded library instead
9:25AM DBG Custom library not provided. Loading embedded library instead
9:25AM DBG Could not open embedded library data for cloudFormation platform
9:25AM DBG Stopped MEM profiling for get_queries
9:25AM INF Total MEM usage for get_queries: 260.02MB
9:25AM INF Inspector initialized, number of queries=496
9:25AM INF Query execution timeout=1m0s
9:25AM DBG provider.NewFileSystemSourceProvider()
9:25AM DBG parser.NewBuilder()
9:25AM DBG resolver.Add()
9:25AM DBG resolver.Build()
9:25AM DBG Started MEM profiling for prepare_sources
9:35AM DBG Stopped MEM profiling for prepare_sources
9:35AM INF Total MEM usage for prepare_sources: 1.48GB
9:35AM DBG Started MEM profiling for start_scan
9:35AM DBG service.StartScan()
9:35AM DBG service.StartScan()
9:35AM DBG engine.Inspect()
rafaela-soares commented 2 years ago

So it is not related to the behaviour of the flag --type.

Can you try to use 15 GB memory, 8 vCPUs, please?

rafaela-soares commented 2 years ago

@ALutchko, can you send us only the IaC code to kics@checkmarx.com? Only for debugging purposes to understand what is happening.

ALutchko commented 2 years ago

Large: just for test purpose, because using such size in real workflow won't make sense due to price. Tested, switched off after 20 min from start

10:21AM INF Scanning with Keeping Infrastructure as Code Secure v1.5.5
10:21AM DBG storage.NewMemoryStorage()
10:21AM DBG Looking for queries in executable path and in current work directory
10:21AM DBG helpers.GetDefaultQueryPath()
10:21AM DBG helpers.GetExecutableDirectory()
10:21AM DBG Queries found in /app/bin/assets/queries
10:21AM DBG source.NewFilesystemSource()
10:21AM DBG engine.NewInspector()
10:21AM DBG Started MEM profiling for get_queries
10:21AM DBG Custom library not provided. Loading embedded library instead
10:21AM DBG Custom library not provided. Loading embedded library instead
10:21AM DBG Could not open embedded library data for cloudFormation platform
10:21AM DBG Stopped MEM profiling for get_queries
10:21AM INF Total MEM usage for get_queries: 225.68MB
10:21AM INF Inspector initialized, number of queries=496
10:21AM INF Query execution timeout=1m0s
10:21AM DBG provider.NewFileSystemSourceProvider()
10:21AM DBG parser.NewBuilder()
10:21AM DBG resolver.Add()
10:21AM DBG resolver.Build()
10:21AM DBG Started MEM profiling for prepare_sources
10:30AM DBG Stopped MEM profiling for prepare_sources
10:30AM INF Total MEM usage for prepare_sources: 1.48GB
10:30AM DBG Started MEM profiling for start_scan
10:30AM DBG service.StartScan()
10:30AM DBG service.StartScan()
10:30AM DBG engine.Inspect()

I'll try to send something.

ALutchko commented 2 years ago

Sent.

rafaela-soares commented 2 years ago

Hello again, @ALutchko! Thank you so much for sending it.

We ran KICS against your folder and the scan was quick.

ALutchko commented 2 years ago

Locally it works, even on VM having 1GB ram so memory isn't the issue here. Looks strange but timestamps are in GMT, not my local time. 

r@l $ docker run -v /home/ans/clsm-ci-cd:/path checkmarx/kics scan  --ignore-on-exit all -p /path -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --profiling MEM
2:19PM DBG Could not find string flag ci
2:19PM DBG console.scan()
2:19PM DBG console.scan()

   [ logo ]

Scanning with Keeping Infrastructure as Code Secure v1.5.5

2:19PM INF Scanning with Keeping Infrastructure as Code Secure v1.5.5
2:19PM DBG storage.NewMemoryStorage()
2:19PM DBG Looking for queries in executable path and in current work directory
2:19PM DBG helpers.GetDefaultQueryPath()
2:19PM DBG helpers.GetExecutableDirectory()
2:19PM DBG Queries found in /app/bin/assets/queries
2:19PM DBG source.NewFilesystemSource()
2:19PM DBG engine.NewInspector()
2:19PM DBG Started MEM profiling for get_queries
2:19PM DBG Custom library not provided. Loading embedded library instead
2:19PM DBG Custom library not provided. Loading embedded library instead
2:19PM DBG Could not open embedded library data for cloudFormation platform
2:20PM DBG Stopped MEM profiling for get_queries
2:20PM INF Total MEM usage for get_queries: 273.14MB
2:20PM INF Inspector initialized, number of queries=496
2:20PM INF Query execution timeout=1m0s
2:20PM DBG provider.NewFileSystemSourceProvider()
2:20PM DBG parser.NewBuilder()
2:20PM DBG resolver.Add()
2:20PM DBG resolver.Build()
2:20PM DBG Started MEM profiling for prepare_sources
2:20PM DBG Stopped MEM profiling for prepare_sources
2:20PM INF Total MEM usage for prepare_sources: 273.14MB
2:20PM DBG Started MEM profiling for start_scan
2:20PM DBG service.StartScan()
2:20PM DBG service.StartScan()
2:20PM DBG engine.Inspect()
2:20PM DBG match: true :: 6.256660416485939
2:20PM DBG match: true :: 5.037401197654112
2:20PM DBG engine.Inspect()

2:22PM DBG Stopped MEM profiling for start_scan
2:22PM INF Total MEM usage for start_scan: 403.90MB
2:22PM DBG model.CreateSummary()
2:22PM DBG HTTP POST to descriptions endpoint
2:22PM DBG HTTP Status: 200 OK 253.227496ms
2:22PM DBG console.resolveOutputs()
2:22PM DBG helpers.PrintResult()
Files scanned: 38
Parsed files: 38
Queries loaded: 496
Queries failed to execute: 0

------------------------------------

EC2 Not EBS Optimized, Severity: INFO, Results: 1
Description: It's considered a best practice for an EC2 instance to use an EBS optimized instance. This provides the best 
....
Results Summary:
HIGH: 67
MEDIUM: 38
LOW: 76
INFO: 1
TOTAL: 182

2:22PM INF Files scanned: 38
2:22PM INF Lines scanned: 8594
2:22PM INF Parsed files: 38
2:22PM INF Lines parsed: 8594
2:22PM INF Queries loaded: 496
2:22PM INF Queries failed to execute: 0
2:22PM INF Inspector stopped
2:22PM DBG console.printOutput()
2:22PM DBG Output formats provided [junit]
2:22PM DBG helpers.GenerateReport()
2:22PM DBG Started MEM profiling for generate_report
2:22PM INF Results saved to file /path/junit-kics-report.xml fileName=junit-kics-report.xml
Results saved to file /path/junit-kics-report.xml
2:22PM DBG Stopped MEM profiling for generate_report
2:22PM INF Total MEM usage for generate_report: 403.90MB
Scan duration: 3m46.054786187s
2:22PM INF Scan duration: 3m46.054786187s

The repo root contains files like below:

aws-ci-cd # (the dir in the archive)
...
kics.config
README.md

Just in case:

$ cat kics.config

log-file: true
log-level: DEBUG
#log-path: .
silent: false
type:
  - CloudFormation
# Ansible, AzureResourceManager, Buildah, CloudFormation, DockerCompose, Dockerfile,
# GRPC, GoogleDeploymentManager, Kubernetes, OpenAPI, Terraform
verbose: true

But the same config exists locally.

Looks like the issue is in relations with CodeBuild.

ALutchko commented 2 years ago

I've double-checked, "PrivilegedMode" set to "true" for this CodeBuild

rafaela-soares commented 2 years ago

Yes, it seems that is some CodeBuild issue...

You can specify with directories/files you want to scan. Maybe it will help in the CodeBuild.

For example: docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path/<IaCFolderOrFileName>,/path/<IaCFolderOrFileName2> -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation

Let me know if it helps 🤞

ALutchko commented 2 years ago

This scan completed in 1.5 mins:

[Container] 2022/04/13 14:59:58 Running command path_to_check="/path/aws-ci-cd/infra-auto/infra-auto.v2.yml"
[Container] 2022/04/13 14:59:58 Running command docker run -v $(pwd):/path $kics_img_name scan --ignore-on-exit all -p "$path_to_check" -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --profiling MEM

So, imho it makes sense to add debug to kics before and after each file and/or dir to check which one breaks test. What do you think?

rafaela-soares commented 2 years ago

You can try. But as you showed, the scan is quick locally. I do not know why CodeBuild is crashing 😢

ALutchko commented 2 years ago

I cannot try, it should be done from inside of KICS :) You have that loop inside which iterated over dirs/files :)

rafaela-soares commented 2 years ago

Unfortunately, we can not do much if we do not have access to the entire project.

Regarding the part of the project that you sent us, we did not find anything that breaks or crashes. We tested it locally, and it was quick. And we tested it in the CodeBuild (in a private repo, of course), and it was fast too (even with 3 GB memory, 2 vCPUs).

rafaela-soares commented 2 years ago

Furthermore, as you showed, your entire project took ~3/4 min locally. So, it seems that it is not a problem from the KICS side.

ALutchko commented 2 years ago

I don't blame "it's on your side" but this is the only way I can think about to debug. It's impossible to try files one by one in CodeBuild env :(

ALutchko commented 2 years ago

Maybe the issue can be escalated to the main team?

rafaela-soares commented 2 years ago

The main team is already aware of this issue.

rafaela-soares commented 2 years ago

Did you try the suggestion of pointing only the IaC files in the scan?

docker run -v $(pwd):/path $kics_img_name scan --no-progress --ignore-on-exit all -p /path/aws-ci-cd/infra-auto,/path/aws-ci-cd/infra-manual,/path/aws-ci-cd/pipelines -o /path --report-formats junit --output-name kics-report --log-level=DEBUG --type CloudFormation

We would be very glad to help you 😊 I will talk to the team tomorrow about introducing more debug log messages.

ALutchko commented 2 years ago

Even worse: stopped after 20 mins. Maybe there's a typo or need spaces between comas? And I see this hang-in-silence regardless of "--log-level=DEBUG" 👎👎👎 obraz

ALutchko commented 2 years ago

This completes in 2 mins obraz

rafaela-soares commented 2 years ago

There is no typo and no need for spaces between comas:

[Container] 2022/04/14 07:50:38 Running command docker run -v $PWD:/path checkmarx/kics scan --no-progress --ignore-on-exit all -p /path/aws-ci-cd/infra-auto,/path/aws-ci-cd/infra-manual,/path/aws-ci-cd/pipelines -o /path --report-formats junit --output-name kics-report -v --log-level DEBUG --profiling MEM --type CloudFormation
31 | Unable to find image 'checkmarx/kics:latest' locally
32 | latest: Pulling from checkmarx/kics
33 | 40e059520d19: Pulling fs layer
34 | c45c8fc16a24: Pulling fs layer
35 | d839455a985f: Pulling fs layer
36 | eb9b894d736d: Pulling fs layer
37 | 52f6c41608e2: Pulling fs layer
38 | 4f4fb700ef54: Pulling fs layer
39 | eb9b894d736d: Waiting
40 | 52f6c41608e2: Waiting
41 | 4f4fb700ef54: Waiting
42 | 40e059520d19: Verifying Checksum
43 | 40e059520d19: Download complete
44 | 40e059520d19: Pull complete
45 | d839455a985f: Verifying Checksum
46 | d839455a985f: Download complete
47 | eb9b894d736d: Verifying Checksum
48 | eb9b894d736d: Download complete
49 | 4f4fb700ef54: Verifying Checksum
50 | 4f4fb700ef54: Download complete
51 | 52f6c41608e2: Verifying Checksum
52 | 52f6c41608e2: Download complete
53 | c45c8fc16a24: Verifying Checksum
54 | c45c8fc16a24: Download complete
55 | c45c8fc16a24: Pull complete
56 | d839455a985f: Pull complete
57 | eb9b894d736d: Pull complete
58 | 52f6c41608e2: Pull complete
59 | 4f4fb700ef54: Pull complete
60 | Digest: sha256:024bc81f6ba68fdac9a194a52e88d2bc341b3a65467cd451faa160a53257ee74
61 | Status: Downloaded newer image for checkmarx/kics:latest
62 | 7:50AM DBG Could not find string flag ci
63 | 7:50AM DBG console.scan()
64 | 7:50AM WRN Any kics.config file will be ignored, please use --config if kics.config is wanted
65 | 7:50AM DBG console.scan()
66 | 7:50AM WRN Any kics.config file will be ignored, please use --config if kics.config is wanted
67 |  
68 | .0MO.
69 | OMMMx
70 | ;NMX;
71 | ...           ...              ....
72 | WMMMd     cWMMM0.  KMMMO      ;xKWMMMMNOc.     ,xXMMMMMWXkc.
73 | WMMMd   .0MMMN:    KMMMO    :XMMMMMMMMMMMWl   xMMMMMWMMMMMMl
74 | WMMMd  lWMMMO.     KMMMO   xMMMMKc...'lXMk   ,MMMMx   .;dXx
75 | WMMMd.0MMMX;       KMMMO  cMMMMd        '    'MMMMNl'
76 | WMMMNWMMMMl        KMMMO  0MMMN               oMMMMMMMXkl.
77 | WMMMMMMMMMMo       KMMMO  0MMMX                .ckKWMMMMMM0.
78 | WMMMMWokMMMMk      KMMMO  oMMMMc              .     .:OMMMM0
79 | WMMMK.  dMMMM0.    KMMMO   KMMMMx'    ,kNc   :WOc.    .NMMMX
80 | WMMMd    cWMMMX.   KMMMO    kMMMMMWXNMMMMMd .WMMMMWKO0NMMMMl
81 | WMMMd     ,NMMMN,  KMMMO     'xNMMMMMMMNx,   .l0WMMMMMMMWk,
82 | xkkk:      ,kkkkx  okkkl        ;xKXKx;          ;dOKKkc
83 |  
84 |  
85 | Scanning with Keeping Infrastructure as Code Secure v1.5.5
86 |  
87 |  
88 | 7:50AM INF Scanning with Keeping Infrastructure as Code Secure v1.5.5
89 | 7:50AM DBG storage.NewMemoryStorage()
90 | 7:50AM DBG Looking for queries in executable path and in current work directory
91 | 7:50AM DBG helpers.GetDefaultQueryPath()
92 | 7:50AM DBG helpers.GetExecutableDirectory()
93 | 7:50AM DBG Queries found in /app/bin/assets/queries
94 | 7:50AM DBG source.NewFilesystemSource()
95 | 7:50AM DBG engine.NewInspector()
96 | 7:50AM DBG Started MEM profiling for get_queries
97 | 7:50AM DBG Custom library not provided. Loading embedded library instead
98 | 7:50AM DBG Custom library not provided. Loading embedded library instead
99 | 7:50AM DBG Could not open embedded library data for cloudFormation platform
100 | 7:51AM DBG Stopped MEM profiling for get_queries
101 | 7:51AM INF Total MEM usage for get_queries: 257.19MB
102 | 7:51AM INF Inspector initialized, number of queries=496
103 | 7:51AM INF Query execution timeout=1m0s
104 | 7:51AM DBG provider.NewFileSystemSourceProvider()
105 | 7:51AM DBG parser.NewBuilder()
106 | 7:51AM DBG resolver.Add()
107 | 7:51AM DBG resolver.Build()
108 | 7:51AM DBG Started MEM profiling for prepare_sources
109 | 7:51AM DBG Stopped MEM profiling for prepare_sources
110 | 7:51AM INF Total MEM usage for prepare_sources: 257.19MB
111 | 7:51AM DBG Started MEM profiling for start_scan
112 | 7:51AM DBG service.StartScan()
113 | 7:51AM DBG engine.Inspect()
114 | 7:51AM DBG service.StartScan()
115 | 7:51AM DBG engine.Inspect()
116 |  
117 | 7:52AM DBG Stopped MEM profiling for start_scan
118 | 7:52AM INF Total MEM usage for start_scan: 336.61MB
119 | 7:52AM DBG model.CreateSummary()
120 | 7:52AM DBG HTTP POST to descriptions endpoint
121 | 7:52AM DBG HTTP Status: 200 OK 204.0446ms
122 | 7:52AM DBG console.resolveOutputs()
123 | 7:52AM DBG helpers.PrintResult()
124 | Files scanned: 7
125 | Parsed files: 7
126 | Queries loaded: 496
127 | Queries failed to execute: 0
128 |  
129 | ------------------------------------
130 |  
131 | EC2 Not EBS Optimized, Severity: INFO, Results: 1
132 | Description: It's considered a best practice for an EC2 instance to use an EBS optimized instance. This provides the best performance for your EBS volumes by minimizing contention between Amazon EBS I/O and other traffic from your instance
133 | Platform: CloudFormation
134 |  
135 | [1]: ../../path/aws-ci-cd/infra-auto/infra-auto.v2.yml:543

...
nunoocx commented 2 years ago

Hi @ALutchko, great. Can you check the other paths also one by one so you can pinpoint the problematic path and who knows the problematic file. We will also consider you suggestion of adding more debug when processing files. Let us know what other debugging info you are missing so we can consider it as well in our next soon-to-be release! Thanks

ALutchko commented 2 years ago

@nunoocx

"Let us know what other debugging info you are missing"

In this case most likely we have a problematic file but it's impossible to see which file it is. At the moment the only way is to check all files narrowing down using -p, which is really time- and resource-consuming. It would be very helpful to see in the debug mode which file was passed and which just opened for checking then something happened. So, a message like "opening somedir/fileXYZ" will resolve this problem very easy.

Trying to narrow down meanwhile.

nunoocx commented 2 years ago

@ALutchko, we are on that ;) We know that it is not a processing issue because locally this runs well for you and in CodeBuild you get to the Inspect phase - which means KICS reaches to the querying stage. But will add more debug messages for further investigation.

We suspect that it may be some CodeBuild configuration when using Docker/Containers that by default limits resources, regardless of the memory you configure CodeBuild... did you look to something related with this lead?

rafaela-soares commented 2 years ago

Hello again, @ALutchko 🙂

We launched the new KICS version with more log messages. I hope it helps!

I would like to ask you for more information if you do not mind:

rafaela-soares commented 2 years ago

Hello @ALutchko,

Can you give us an update, please?

ALutchko commented 2 years ago

@rafaela-soares Sorry, I had no time to do that, I don't use the tool since that. Maybe later. Thanks!

rafaela-soares commented 2 years ago

Thanks, @ALutchko!