Multiple custom policies not all scanning

[tmorse9]

• terrascan version: v1.12.0
• Operating System: MacOS BigSur

Description

I am working on creating custom policies to meet my specific requirements. I have been able to write several policies that meet my needs. I am finding that if I have all my rego and json files in a single directory that only one policy gets scanned. If I separate the policies (rego and json files) in their own directory I can specify -p for each I am able to scan each policy. Is there any way to get terrascan to run multiple custom policies at once from a single directory rather than needing to define -p X number of times?

What I Did

My run command that works but requires each dir to be defined:

terrascan scan -p ../../../../tf-sec-policies/security_groups/allowedPorts/ -p ../../../../tf-sec-policies/security_groups/portRange/ -p ../../../../tf-sec-policies/security_groups/sourceDest/  -p ../../../../tf-sec-policies/security_groups/rulesInMainResource/ -o json

What I would like to be able to run:

terrascan scan -p ../../../../tf-sec-policies/security_groups

My current directory structure that I can get to work looks like this:

security_groups:
-> allowedPorts
    -> json file
    -> rego file
-> portRange
    -> json file
    -> rego file

I have tried combining all the files into a single directory with no sub-directories and that is when only a single scan is executed. I

• if I have multiple json policies pointing to a single rego only one runs
• if I have multiple json and rego files in a directory only one runs
• if I separate the json and rego pairs into their own sub-directories and call the parent directory only 1 runs

Summary is there a way to get -p to be recursive and run multiple policies? Also I would be looking to run both the standard provided polices and my custom policies but when -p is defined only those policies run

I see the following info when I run against the one main directory with sub directories underneath:

2021-11-04T16:59:18.397-0400    debug   runtime/executor.go:184 using policy path [../../../../tf-sec-policies/security_groups/]
2021-11-04T16:59:18.397-0400    debug   opa/engine.go:133   no metadata files were found{dir 15 0 ../../../../tf-sec-policies/security_groups/ <nil>}
2021-11-04T16:59:18.399-0400    debug   opa/engine.go:219   loaded 1 Rego rules from 4 rego files (4 metadata files).

note the 4 found and 1 run seems strange. I get the same output if all the files are in the same directory as well

security_groups:
-> rego file 1
-> json file 1
-> rego file 2
-> json file 2
-> rego file 3
-> json file 3

Any guidance would be really appreciated!

[cesar-rodriguez]

Hi @tmorse9,

Thanks for opening this issue. While we look into this, a workaround is to place your custom policies in the appropriate directory under $HOME/.terrascan/pkg/policies/opa/rego and running Terrascan without the -p flag. That's the default directory where Terrascan looks for policies.

[tmorse9]

@cesar-rodriguez You info helped me find the underlying issue. I tried moving this to the proper directories that everything get scanned from. I continued to see the problem where only the first policy is being scanned, of my custom policies. With that context I determined that in the examples provided here https://runterrascan.io/docs/policies/policies/ the deprecated field ruleReferenceId which in the provided policies is listed as reference_id is required for terrascan to see each json as an individual rather than only pulling the first rule. Once I added that field with a unique value all my policies were being scanned both in the standard dir path and in the -p under a single dir specified once.

My current command now works as: all 4 policies are now being scanned!

terrascan scan -o json -p ../../../../tf-sec-policies/test/

My end json looks like this:

{
    "name": "ValidateSourceIsNotCidr",
    "file": "sourceDestCheck.rego",
    "policy_type": "aws",
    "resource_type": "aws_security_group_rule",
    "template_args": {
        "name": "ValidateSourceIsNotCidr",
        "resourceType": "aws_security_group_rule",
    "ruleType": "\"ingress\"",
        "defaultValue": "notFound",
        "prefix": "",
        "suffix": ""
    },
    "severity": "HIGH",
    "description": "\n The SG rule ingress contains a CIDR address. Flagging for review. \n If you would like to avoid reviews please use SG IDs or self routing rules",
    "category": "NETWORK_SECURITY",
    "reference_id": "FAN.AWS.SG.Networking.HIGH.5",
    "version": 1,
    "id": "FAN_AWS_5"
}