Graph Engine - InternalExecutionError

[ppillai7777]

Ran the graph engine scan on our application source as shown below and report shows the InternalExecutionError rules. sfdx scanner:run:dfa --format=csv --outfile=CodeAnalyzerDFA9.csv --target="./" --projectdir="./" --category="Security" --sfgejvmargs "-Xmx20g" --pathexplimit -1 --rule-thread-count 2 --rule-thread-timeout 300000

The output of the CSV file: "Problem","Severity","Source File","Source Line","Source Column","Source Type","Source Method","Sink File","Sink Line","Sink Column","Rule","Description","URL","Category","Engine" "1","3","C:\Users\PJPillai\git\salesforce-connector\connector\force-app\main\default\classes\controller\DocumentController.cls","147","29","DocumentController","queryDocument","","","","InternalExecutionError","Path evaluation timed out after 300000 ms","https://forcedotcom.github.io/sfdx-scanner/en/v3.x/salesforce-graph-engine/rules/#ApexFlsViolationRule","InternalExecutionError","sfge" "2","3","C:\Users\PJPillai\git\salesforce-connector\connector\force-app\main\default\classes\controller\DocumentController.cls","114","29","DocumentController","queryDocuments","","","","InternalExecutionError","Path evaluation timed out after 300000 ms","https://forcedotcom.github.io/sfdx-scanner/en/v3.x/salesforce-graph-engine/rules/#ApexFlsViolationRule","InternalExecutionError","sfge" "3","3","C:\Users\PJPillai\git\salesforce-connector\connector\force-app\main\default\classes\controller\DocumentController.cls","202","29","DocumentController","queryRecords","","","","InternalExecutionError","Path evaluation timed out after 300000 ms","https://forcedotcom.github.io/sfdx-scanner/en/v3.x/salesforce-graph-engine/rules/#ApexFlsViolationRule","InternalExecutionError","sfge" "4","3","C:\Users\PJPillai\git\salesforce-connector\connector\force-app\main\default\classes\controller\LookupController.cls","78","32","LookupController","search","","","","InternalExecutionError","Graph Engine identified your source and sink, but you must manually verify that you have a sanitizer in this path. Then, add an engine directive to skip the path. Next, create a Github issue for the Code Analyzer team that includes the error and stack trace. After we fix this issue, check the Code Analyzer release notes for more info. Error and stacktrace: UnexpectedException: ArrayLoadExpression{properties={FirstChild=true, BeginLine=133, DefiningType_CaseSafe=lookupcontroller, LastChild=false, DefiningType=LookupController, EndLine=133, childIdx=0, BeginColumn=46}}: com.salesforce.graph.symbols.PathScopeVisitor.afterVisit(PathScopeVisitor.java:805);com.salesforce.graph.symbols.DefaultSymbolProviderVertexVisitor.afterVisit(DefaultSymbolProviderVertexVisitor.java:737);com.salesforce.graph.vertex.ArrayLoadExpressionVertex.afterVisit(ArrayLoadExpressionVertex.java:58);com.salesforce.graph.ops.expander.ApexPathExpander.performAfterVisit(ApexPathExpander.java:577);com.salesforce.graph.ops.expander.ApexPathExpander.visit(ApexPathExpander.java:536);com.salesforce.graph.ops.expander.ApexPathExpander.visit(ApexPathExpander.java:523)","https://forcedotcom.github.io/sfdx-scanner/en/v3.x/salesforce-graph-engine/rules/#ApexFlsViolationRule","InternalExecutionError","sfge" "5","3","C:\Users\PJPillai\git\salesforce-connector\connector\force-app\main\default\classes\controller\SyncController.cls","79","34","SyncController","createObjectStore","","","","InternalExecutionError","Graph Engine identified your source and sink, but you must manually verify that you have a sanitizer in this path. Then, add an engine directive to skip the path. Next, create a Github issue for the Code Analyzer team that includes the error and stack trace. After we fix this issue, check the Code Analyzer release notes for more info. Error and stacktrace: UnexpectedException: SObjectType is not available in DescribeSObjectResult: ApexStandardValue{apexType='Schema.DescribeSObjectResult'} ApexValue(DescribeSObjectResult) {status=INITIALIZED, declarationVertex=null, valueVertex=null, resolvedValues={}, returnedFrom=ApexStandardValue{apexType='Schema.SObjectField'} ApexValue(SObjectField) {status=INITIALIZED, declarationVertex=null, valueVertex=null, resolvedValues={}, returnedFrom=ApexValue(ApexFieldDescribeMapValue) {status=INITIALIZED, declarationVertex=null, valueVertex=null, resolvedValues={}, returnedFrom=ApexStandardValue{apexType='Schema.DescribeSObjectResult'} ApexValue(DescribeSObjectResult) {status=INITIALIZED, declarationVertex=null, valueVertex=null, resolvedValues={}, returnedFrom=SObjectType{associatedObjectType='null'} ApexStandardValue{apexType='Schema.SObjectType'} ApexValue(SObjectType) {status=INDETERMINANT, declarationVertex=SObjectType{associatedObjectType='null'} ApexStandardValue{apexType='Schema.SObjectType'} ApexValue(SObjectType) {status=INDETERMINANT, declarationVertex=VariableDeclaration{properties={FirstChild=false, BeginLine=25, Type=SObjectType, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=25, Name_CaseSafe=sotype, childIdx=1, BeginColumn=21, Name=soType}}, valueVertex=MethodCallExpressionVertex{fullMethodName=os.getSObjectType, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[os], BeginLine=25, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=25, Name_CaseSafe=os, childIdx=0, BeginColumn=30, ReferenceType=METHOD, Name=os}}}, chainedNames=[os], properties={FirstChild=true, FullMethodName=os.getSObjectType, BeginLine=25, FullMethodName_CaseSafe=os.getsobjecttype, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=25, MethodName_CaseSafe=getsobjecttype, childIdx=0, BeginColumn=33, MethodName=getSObjectType}}, resolvedValues={}, returnedFrom=null, invocableExpression=null, method=null}, valueVertex=MethodCallExpressionVertex{fullMethodName=os.getSObjectType, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[os], BeginLine=25, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=25, Name_CaseSafe=os, childIdx=0, BeginColumn=30, ReferenceType=METHOD, Name=os}}}, chainedNames=[os], properties={FirstChild=true, FullMethodName=os.getSObjectType, BeginLine=25, FullMethodName_CaseSafe=os.getsobjecttype, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=25, MethodName_CaseSafe=getsobjecttype, childIdx=0, BeginColumn=33, MethodName=getSObjectType}}, resolvedValues={}, returnedFrom=null, invocableExpression=null, method=null}, invocableExpression=MethodCallExpressionVertex{fullMethodName=soType.getDescribe, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[soType], BeginLine=26, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=26, Name_CaseSafe=sotype, childIdx=0, BeginColumn=53, ReferenceType=METHOD, Name=soType}}}, chainedNames=[soType], properties={FirstChild=true, FullMethodName=soType.getDescribe, BeginLine=26, FullMethodName_CaseSafe=sotype.getdescribe, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=26, MethodName_CaseSafe=getdescribe, childIdx=0, BeginColumn=60, MethodName=getDescribe}}, method=Method{properties={FirstChild=false, BeginLine=2, IsStandard=true, DefiningType_CaseSafe=schema.sobjecttype, LastChild=false, DefiningType=Schema.SObjectType, Constructor=false, EndLine=2, Name_CaseSafe=getdescribe, childIdx=1, ReturnType=Schema.DescribeSObjectResult, Name=getDescribe, Arity=0, ReturnType_CaseSafe=schema.describesobjectresult, BeginColumn=38}}}, invocableExpression=MethodCallExpressionVertex{fullMethodName=fields.getMap, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[fields], BeginLine=26, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=26, Name_CaseSafe=fields, childIdx=0, BeginColumn=74, ReferenceType=METHOD, Name=fields}}}, chainedNames=[fields], properties={FirstChild=true, FullMethodName=fields.getMap, BeginLine=26, FullMethodName_CaseSafe=fields.getmap, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=26, MethodName_CaseSafe=getmap, childIdx=0, BeginColumn=81, MethodName=getMap}}, method=null}, invocableExpression=MethodCallExpressionVertex{fullMethodName=fieldMap.get, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[fieldMap], BeginLine=32, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=32, Name_CaseSafe=fieldmap, childIdx=0, BeginColumn=39, ReferenceType=METHOD, Name=fieldMap}}}, chainedNames=[fieldMap], properties={FirstChild=true, FullMethodName=fieldMap.get, BeginLine=32, FullMethodName_CaseSafe=fieldmap.get, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=32, MethodName_CaseSafe=get, childIdx=0, BeginColumn=48, MethodName=get}}, method=null}, invocableExpression=MethodCallExpressionVertex{fullMethodName=sof.getDescribe, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[sof], BeginLine=38, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=38, Name_CaseSafe=sof, childIdx=0, BeginColumn=21, ReferenceType=METHOD, Name=sof}}}, chainedNames=[sof], properties={FirstChild=true, FullMethodName=sof.getDescribe, BeginLine=38, FullMethodName_CaseSafe=sof.getdescribe, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=38, MethodName_CaseSafe=getdescribe, childIdx=0, BeginColumn=25, MethodName=getDescribe}}, method=Method{properties={FirstChild=false, BeginLine=2, IsStandard=true, DefiningType_CaseSafe=schema.sobjectfield, LastChild=false, DefiningType=Schema.SObjectField, Constructor=false, EndLine=2, Name_CaseSafe=getdescribe, childIdx=1, ReturnType=Schema.DescribeFieldResult, Name=getDescribe, Arity=0, ReturnType_CaseSafe=schema.describefieldresult, BeginColumn=36}}}: com.salesforce.rules.fls.apex.operations.FlsValidationRepresentation.setObject(FlsValidationRepresentation.java:86);com.salesforce.rules.fls.apex.operations.SchemaBasedValidationAnalyzer.convert(SchemaBasedValidationAnalyzer.java:183);com.salesforce.rules.fls.apex.operations.SchemaBasedValidationAnalyzer.checkForValidation(SchemaBasedValidationAnalyzer.java:77);com.salesforce.rules.fls.apex.operations.FlsValidationCentral.checkSchemaBasedFlsValidation(FlsValidationCentral.java:71);com.salesforce.rules.fls.apex.AbstractFlsVisitor.afterVisit(AbstractFlsVisitor.java:67);com.salesforce.graph.vertex.StandardConditionVertex$Negative.afterVisit(StandardConditionVertex.java:82)","https://forcedotcom.github.io/sfdx-scanner/en/v3.x/salesforce-graph-engine/rules/#ApexFlsViolationRule","InternalExecutionError","sfge" "6","3","C:\Users\PJPillai\git\salesforce-connector\connector\force-app\main\default\classes\controller\SyncController.cls","91","24","SyncController","syncSingleClass","","","","InternalExecutionError","Graph Engine identified your source and sink, but you must manually verify that you have a sanitizer in this path. Then, add an engine directive to skip the path. Next, create a Github issue for the Code Analyzer team that includes the error and stack trace. After we fix this issue, check the Code Analyzer release notes for more info. Error and stacktrace: StackOverflowError: null: java.base/java.util.TreeMap.containsKey(TreeMap.java:233);com.salesforce.graph.symbols.PathScopeVisitor.getApexValue(PathScopeVisitor.java:337);com.salesforce.graph.symbols.PathScopeVisitor.getApexValue(PathScopeVisitor.java:352);com.salesforce.graph.symbols.PathScopeVisitor.getApexValue(PathScopeVisitor.java:352);com.salesforce.graph.symbols.PathScopeVisitor.getApexValue(PathScopeVisitor.java:352);com.salesforce.graph.symbols.PathScopeVisitor.getApexValue(PathScopeVisitor.java:352)","https://forcedotcom.github.io/sfdx-scanner/en/v3.x/salesforce-graph-engine/rules/#ApexFlsViolationRule","InternalExecutionError","sfge" "7","3","C:\Users\PJPillai\git\salesforce-connector\connector\force-app\main\default\classes\controller\SyncController.cls","111","24","SyncController","updateSingleClass","","","","InternalExecutionError","Graph Engine identified your source and sink, but you must manually verify that you have a sanitizer in this path. Then, add an engine directive to skip the path. Next, create a Github issue for the Code Analyzer team that includes the error and stack trace. After we fix this issue, check the Code Analyzer release notes for more info. Error and stacktrace: UnexpectedException: SObjectType is not available in DescribeSObjectResult: ApexStandardValue{apexType='Schema.DescribeSObjectResult'} ApexValue(DescribeSObjectResult) {status=INITIALIZED, declarationVertex=null, valueVertex=null, resolvedValues={}, returnedFrom=ApexStandardValue{apexType='Schema.SObjectField'} ApexValue(SObjectField) {status=INITIALIZED, declarationVertex=null, valueVertex=null, resolvedValues={}, returnedFrom=ApexValue(ApexFieldDescribeMapValue) {status=INITIALIZED, declarationVertex=null, valueVertex=null, resolvedValues={}, returnedFrom=ApexStandardValue{apexType='Schema.DescribeSObjectResult'} ApexValue(DescribeSObjectResult) {status=INITIALIZED, declarationVertex=null, valueVertex=null, resolvedValues={}, returnedFrom=SObjectType{associatedObjectType='null'} ApexStandardValue{apexType='Schema.SObjectType'} ApexValue(SObjectType) {status=INDETERMINANT, declarationVertex=SObjectType{associatedObjectType='null'} ApexStandardValue{apexType='Schema.SObjectType'} ApexValue(SObjectType) {status=INDETERMINANT, declarationVertex=VariableDeclaration{properties={FirstChild=false, BeginLine=25, Type=SObjectType, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=25, Name_CaseSafe=sotype, childIdx=1, BeginColumn=21, Name=soType}}, valueVertex=MethodCallExpressionVertex{fullMethodName=os.getSObjectType, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[os], BeginLine=25, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=25, Name_CaseSafe=os, childIdx=0, BeginColumn=30, ReferenceType=METHOD, Name=os}}}, chainedNames=[os], properties={FirstChild=true, FullMethodName=os.getSObjectType, BeginLine=25, FullMethodName_CaseSafe=os.getsobjecttype, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=25, MethodName_CaseSafe=getsobjecttype, childIdx=0, BeginColumn=33, MethodName=getSObjectType}}, resolvedValues={}, returnedFrom=null, invocableExpression=null, method=null}, valueVertex=MethodCallExpressionVertex{fullMethodName=os.getSObjectType, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[os], BeginLine=25, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=25, Name_CaseSafe=os, childIdx=0, BeginColumn=30, ReferenceType=METHOD, Name=os}}}, chainedNames=[os], properties={FirstChild=true, FullMethodName=os.getSObjectType, BeginLine=25, FullMethodName_CaseSafe=os.getsobjecttype, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=25, MethodName_CaseSafe=getsobjecttype, childIdx=0, BeginColumn=33, MethodName=getSObjectType}}, resolvedValues={}, returnedFrom=null, invocableExpression=null, method=null}, invocableExpression=MethodCallExpressionVertex{fullMethodName=soType.getDescribe, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[soType], BeginLine=26, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=26, Name_CaseSafe=sotype, childIdx=0, BeginColumn=53, ReferenceType=METHOD, Name=soType}}}, chainedNames=[soType], properties={FirstChild=true, FullMethodName=soType.getDescribe, BeginLine=26, FullMethodName_CaseSafe=sotype.getdescribe, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=26, MethodName_CaseSafe=getdescribe, childIdx=0, BeginColumn=60, MethodName=getDescribe}}, method=Method{properties={FirstChild=false, BeginLine=2, IsStandard=true, DefiningType_CaseSafe=schema.sobjecttype, LastChild=false, DefiningType=Schema.SObjectType, Constructor=false, EndLine=2, Name_CaseSafe=getdescribe, childIdx=1, ReturnType=Schema.DescribeSObjectResult, Name=getDescribe, Arity=0, ReturnType_CaseSafe=schema.describesobjectresult, BeginColumn=38}}}, invocableExpression=MethodCallExpressionVertex{fullMethodName=fields.getMap, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[fields], BeginLine=26, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=26, Name_CaseSafe=fields, childIdx=0, BeginColumn=74, ReferenceType=METHOD, Name=fields}}}, chainedNames=[fields], properties={FirstChild=true, FullMethodName=fields.getMap, BeginLine=26, FullMethodName_CaseSafe=fields.getmap, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=26, MethodName_CaseSafe=getmap, childIdx=0, BeginColumn=81, MethodName=getMap}}, method=null}, invocableExpression=MethodCallExpressionVertex{fullMethodName=fieldMap.get, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[fieldMap], BeginLine=32, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=32, Name_CaseSafe=fieldmap, childIdx=0, BeginColumn=39, ReferenceType=METHOD, Name=fieldMap}}}, chainedNames=[fieldMap], properties={FirstChild=true, FullMethodName=fieldMap.get, BeginLine=32, FullMethodName_CaseSafe=fieldmap.get, DefiningType_CaseSafe=flsvalidation, LastChild=false, DefiningType=FLSValidation, EndLine=32, MethodName_CaseSafe=get, childIdx=0, BeginColumn=48, MethodName=get}}, method=null}, invocableExpression=MethodCallExpressionVertex{fullMethodName=sof.getDescribe, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[sof], BeginLine=38, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=38, Name_CaseSafe=sof, childIdx=0, BeginColumn=21, ReferenceType=METHOD, Name=sof}}}, chainedNames=[sof], properties={FirstChild=true, FullMethodName=sof.getDescribe, BeginLine=38, FullMethodName_CaseSafe=sof.getdescribe, DefiningType_CaseSafe=flsvalidation, LastChild=true, DefiningType=FLSValidation, EndLine=38, MethodName_CaseSafe=getdescribe, childIdx=0, BeginColumn=25, MethodName=getDescribe}}, method=Method{properties={FirstChild=false, BeginLine=2, IsStandard=true, DefiningType_CaseSafe=schema.sobjectfield, LastChild=false, DefiningType=Schema.SObjectField, Constructor=false, EndLine=2, Name_CaseSafe=getdescribe, childIdx=1, ReturnType=Schema.DescribeFieldResult, Name=getDescribe, Arity=0, ReturnType_CaseSafe=schema.describefieldresult, BeginColumn=36}}}: com.salesforce.rules.fls.apex.operations.FlsValidationRepresentation.setObject(FlsValidationRepresentation.java:86);com.salesforce.rules.fls.apex.operations.SchemaBasedValidationAnalyzer.convert(SchemaBasedValidationAnalyzer.java:183);com.salesforce.rules.fls.apex.operations.SchemaBasedValidationAnalyzer.checkForValidation(SchemaBasedValidationAnalyzer.java:77);com.salesforce.rules.fls.apex.operations.FlsValidationCentral.checkSchemaBasedFlsValidation(FlsValidationCentral.java:71);com.salesforce.rules.fls.apex.AbstractFlsVisitor.afterVisit(AbstractFlsVisitor.java:67);com.salesforce.graph.vertex.StandardConditionVertex$Negative.afterVisit(StandardConditionVertex.java:82)","https://forcedotcom.github.io/sfdx-scanner/en/v3.x/salesforce-graph-engine/rules/#ApexFlsViolationRule","InternalExecutionError","sfge"

[jfeingold35]

@ppillai7777 , we're going to need the following additional information:

1. Which specific part of the output here are you seeking help with?
2. What is the code that produces the output you're seeing?

[ppillai7777]

1, There two type of issue. a. Path evaluation timed out after 300000 ms I set the --rule-thread-timeout 300000 in the command line option. How much more I need to increase the timeout that I don't see this issue anymore. b. The second issue is the following. There are four issues with this description. This is more of scanner issue than application. Graph Engine identified your source and sink, but you must manually verify that you have a sanitizer in this path. Then, add an engine directive to skip the path. Next, create a Github issue for the Code Analyzer team that includes the error and stack trace. After we fix this issue, check the Code Analyzer release notes for more info. Error and stacktrace: UnexpectedException: ArrayLoadExpression{properties={FirstChild=true, BeginLine=133, DefiningType_CaseSafe=lookupcontroller, LastChild=false, DefiningType=LookupController, EndLine=133, childIdx=0, BeginColumn=46}}: com.salesforce.graph.symbols.PathScopeVisitor.afterVisit(PathScopeVisitor.java:805);com.salesforce.graph.symbols.DefaultSymbolProviderVertexVisitor.afterVisit(DefaultSymbolProviderVertexVisitor.java:737);com.salesforce.graph.vertex.ArrayLoadExpressionVertex.afterVisit(ArrayLoadExpressionVertex.java:58);com.salesforce.graph.ops.expander.ApexPathExpander.performAfterVisit(ApexPathExpander.java:577);com.salesforce.graph.ops.expander.ApexPathExpander.visit(ApexPathExpander.java:536);com.salesforce.graph.ops.expander.ApexPathExpander.visit(ApexPathExpander.java:523)

2. I ran the following command against our Apex code in the Application. sfdx scanner:run:dfa --format=csv --outfile=CodeAnalyzerDFA9.csv --target="./" --projectdir="./" --category="Security" --sfgejvmargs "-Xmx20g" --pathexplimit -1 --rule-thread-count 2 --rule-thread-timeout 300000

[jfeingold35]

Regarding the timeout: Every codebase is different, so I can't say for certain. That said, 300,000 is smaller than the default value of 900,000, so that might be your problem. Try using 1800000 (1,800,000 ms, or 30 minutes) and see if that helps. Regarding the other issue: The error message mentions a vertex with DefiningType=LookupController and BeginLine=133. So whatever the problem is, it's happening at line 133 of LookupController.cls. Can you post the code at line 133?

[ppillai7777]

OK I will run the scan with 30 minutes timeout. Here is code from LookupController.cls:

[jfeingold35]

@ppillai7777 , where is fieldDefs declared? And if you assign fieldDefs.get(obj.type) to its own variable (e.g., foo) and then reference that variable (foo[0] at 133 and foo[1] at 134) does that resolve the error?

[ppillai7777]

The fieldDefs is defined in the same method as shown below: Map<String, List> fieldDefs = new Map<String, List>();

There is code prior to this for loop populates data into this map. I could try change code as you mentioned above but the current code is valid Apex code.

[jfeingold35]

@ppillai7777 , I'm not disputing that the code you currently have is valid apex. But clearly the Graph Engine is struggling to properly process it. This seems like it might be a duplicate of #973 , and if that's the case then the same workaround could be used here.

[ppillai7777]

I'm still getting Path evaluation timed out after timeout was set to 1800000 ms. I started another scan with timeout set to 1 hour.

[ppillai7777]

What about the other three issues with Graph Engine? Are those known issues as well?

[ppillai7777]

With timeout set to 1 hour, I got the OutOfMemory error:

PS C:\Users\PJPillai\git\salesforce-connector\connector> sfdx scanner:run:dfa --format=csv --outfile=CodeAnalyzerDFA12.csv --target="./" --projectdir="./" --category="Security" --sfgejvmargs "-Xmx20g" --pathexplimit -1 --rule-thread-count 2 --rule-thread-timeout 3600000 WARNING: We're continually improving Salesforce Code Analyzer. Tell us what you think! Give feedback at https://research.net/r/SalesforceCA (node:5448) Warning: Deprecated config name: apiVersion. Please use org-api-version instead. (Usenode --trace-warnings ...to show where the warning was created) Analyzing with Salesforce Graph Engine. See C:\Users\PJPillai\.sfdx-scanner\sfge.log for details.... Error ERROR running scanner:run:dfa: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError, Caused by: java.lang.OutOfMemoryError

[ppillai7777]

I modified the LookupController.cls as per your suggestion, ran the scan and the workaround did not work. Here is the description: Graph Engine identified your source and sink, but you must manually verify that you have a sanitizer in this path. Then, add an engine directive to skip the path. Next, create a Github issue for the Code Analyzer team that includes the error and stack trace. After we fix this issue, check the Code Analyzer release notes for more info. Error and stacktrace: UnexpectedException: ArrayLoadExpression{properties={FirstChild=true, BeginLine=136, DefiningType_CaseSafe=lookupcontroller, LastChild=false, DefiningType=LookupController, EndLine=136, childIdx=0, BeginColumn=36}}: com.salesforce.graph.symbols.PathScopeVisitor.afterVisit(PathScopeVisitor.java:805);com.salesforce.graph.symbols.DefaultSymbolProviderVertexVisitor.afterVisit(DefaultSymbolProviderVertexVisitor.java:737);com.salesforce.graph.vertex.ArrayLoadExpressionVertex.afterVisit(ArrayLoadExpressionVertex.java:58);com.salesforce.graph.ops.expander.ApexPathExpander.performAfterVisit(ApexPathExpander.java:577);com.salesforce.graph.ops.expander.ApexPathExpander.visit(ApexPathExpander.java:536);com.salesforce.graph.ops.expander.ApexPathExpander.visit(ApexPathExpander.java:523)

[jfeingold35]

@ppillai7777 , try a larger heap size. If that still doesn't work, then we can explore other options.

And what are the other three issues? If the issues all have different error messages, can I ask that you please log them separately? That will make tracking and fixing them significantly easier for us.

[ppillai7777]

I tried the 25GB heap size so now the scan completes without OOM error however the workaround do not work. Here is the error: Graph Engine identified your source and sink, but you must manually verify that you have a sanitizer in this path. Then, add an engine directive to skip the path. Next, create a Github issue for the Code Analyzer team that includes the error and stack trace. After we fix this issue, check the Code Analyzer release notes for more info. Error and stacktrace: UnexpectedException: ArrayLoadExpression{properties={FirstChild=true, BeginLine=136, DefiningType_CaseSafe=lookupcontroller, LastChild=false, DefiningType=LookupController, EndLine=136, childIdx=0, BeginColumn=36}}: com.salesforce.graph.symbols.PathScopeVisitor.afterVisit(PathScopeVisitor.java:805);com.salesforce.graph.symbols.DefaultSymbolProviderVertexVisitor.afterVisit(DefaultSymbolProviderVertexVisitor.java:737);com.salesforce.graph.vertex.ArrayLoadExpressionVertex.afterVisit(ArrayLoadExpressionVertex.java:58);com.salesforce.graph.ops.expander.ApexPathExpander.performAfterVisit(ApexPathExpander.java:577);com.salesforce.graph.ops.expander.ApexPathExpander.visit(ApexPathExpander.java:536);com.salesforce.graph.ops.expander.ApexPathExpander.visit(ApexPathExpander.java:523)

Here is a command: sfdx scanner:run:dfa --format=csv --outfile=CodeAnalyzerDFA9.csv --target="./" --projectdir="./" --category="Security" --sfgejvmargs "-Xmx25g" --pathexplimit -1 --rule-thread-count 2 --rule-thread-timeout 3600000

[jfeingold35]

Okay. Sounds like you've resolved the issue with the timeouts and OOM errors, so seeing as the only remaining error appears to be a duplicate of #973 , I'm going to close this issue as a duplicate of that one.

[ppillai7777]

No timeout issues not solved solved even with --rule-thread-timeout 3600000 (1 hour). I will open another issue. This issue is also not solved. The workaround you gave do not work. See the error description above.

[jfeingold35]

@ppillai7777 , you do not need to log a new issue for the timeouts.

[jfeingold35]

@ppillai7777 , to clarify, I will reopen this issue and leave it open for now. However, the timeouts do not require their own issue. I understand that it's frustrating, but you may just have to increase the time limit until the execution succeeds. It's possible that time might be a few hours. For this reason, we recommend that users with particularly complex codebases run the Graph Engine rules on a nightly build instead of on every commit. And as for the ArrayLoadException, I recognize that the workaround hasn't worked. But the substance of your issue is the same as that of an existing issue, so there's no need to have two issues for the same root cause open.

[ppillai7777]

@jfeingold35 Thank you! I've installed the latest verion 3.17.0. Where can I get the nightly build? I don't see the link from the main page. Regarding the timeout, I've set it to 6 hours and started the scan. Apart from the error above I do have 3 additional issues. Error description is different. Should I open separate issues for those? Also, I'm attaching the log file from the last run. Please take a look. sfge-10-05-2023-1.log.gz

[jfeingold35]

@ppillai7777 , I think there's been a miscommunication. We don't have a nightly build. We recommend that users run the DFA rules as part of their own nightly build. If you've got three different errors with three different messages, then yeah please log different issues for each one.

[ppillai7777]

@jfeingold35 Thank you for the clarification on nightly build.

With the heap size set to 25GB and timeout set to 6 hours, I'm getting Java Out of memory again. See attached screenshot. No log file generated for this scan.

I can try adding 5 GB more but that would be the max on my laptop. Not sure why we need so much memory.

This issue is for tracking OOM, Timeout and UnexpectedException: ArrayLoadExpression{properties={FirstChild=true, BeginLine=136, DefiningType_CaseSafe=lookupcontroller, LastChild=false, DefiningType=LookupController, EndLine=136, childIdx=0, BeginColumn=36}}:

I will create separate issues for the three issues mentioned above.

[jfeingold35]

@ppillai7777 , what version of the scanner are you running? You've never actually indicated this. Please run sfdx plugins and tell me what version of the scanner it says you have.

[ppillai7777]

@jfeingold35 I was using 3.16.0 but I updated to 3.17.0. I did mention that here yesterday - https://github.com/forcedotcom/sfdx-scanner/issues/1201#issuecomment-1749773367

PS C:\Users\PJPillai> sfdx plugins @salesforce/sfdx-scanner 3.17.0

[jfeingold35]

@ppillai7777 , If you're seeing memory issues and timeouts after multiple hours, it sounds like the code you're trying to scan is exceptionally complicated. If there's the timeout is happening with a specific entrypoint, we recommend scanning that entrypoint with its own independent run via the method-level targeting syntax (path/to/myfile.cls#mymethod). Doing this may also relieve some of the pressure on memory. Finally, please see this comment on Issue 1186. It outlines some refactors you can try that can break one entrypoint into two smaller ones.

[ppillai7777]

@jfeingold35 After 7 hours of running the scan and heap size set to 30GB, I still got the OOM. Only one out of three timeout was resolved. I don't think the code in question is exceptionally complicated.

I will try using method level syntax for other two timeouts.

Please see attached log. I see the 4 OOM error in the log. Could you review those stack traces and see if anything else can be done here? sfge-10-05-2023-2.log.gz

[jfeingold35]

@ppillai7777 , well, right away, it seems like there's a gratuitous amount of recursion being noted in the log. Do you have a particularly high amount of recursive calls in your codebase? Also, I'm seeing a lot of this:

2023-10-05 19:16:10 d0e72512-3723-4e2d-8eb0-0f2c4c5a4749 WARN  PathScopeVisitor:401 - TODO: PathScopeVisitor.getApexValue() can currently only support chains of length 2 or lower. keySequence=[System, JSONToken, START_OBJECT]
2023-10-05 19:16:10 d0e72512-3723-4e2d-8eb0-0f2c4c5a4749 WARN  PathScopeVisitor:401 - TODO: PathScopeVisitor.getApexValue() can currently only support chains of length 2 or lower. keySequence=[System, JSONToken, END_OBJECT]

Can you break System.JSONToken.END_OBJECT (for example) into a chain of length 2 or less? (E.g., declare a variable whose value is System.JSONToken and then do foo.END_OBJECT)

[ppillai7777]

Yes, there are recursive calls in the codebase but not much. The code change you are requesting above is because of the limitation in the scanning tool. If so, we are not is position right now to make that change as it requires complete testing.

[jfeingold35]

@ppillai7777 , yes, the requested code change is because of a limitation in the scanner. I'm sorry for that. But as a genuine question, does changing Object o = System.JSONToken.END_OBJECT to

Object t = System.JSONToken;
Object o = t.END_OBJECT;

really require complete testing? Part of the reason we recommend that change is because it's an extremely low-effort and low-risk change.

[johnbelosf]

hi @ppillai7777 we are aware that Graph Engine might face issues with larger codebases, which is why we provided the recommendations that @jfeingold35 set out above. We are continuing to work on optimising the engine for these scenarios so please stay tuned as we continue working through this.

[ppillai7777]

Ok I will try it. I have already spent hours assessing this issue. Every time it errors out with timeout or OOM. The heap size is set to 30GB. Yesterday I ran a scan on single file with timeout set to 6 hours and it still error out with OOM again. I cannot increase the heap size more than 30GB since that is limit on my system.

[ppillai7777]

What about the following TODO? I see ton of these in the log. 2023-10-09 17:02:17 9532f118-6f12-4567-bbfa-0d454d98667f WARN ApexValueBuilder:471 - TODO: Choosing trueValue. ternaryVertex=TernaryExpression{properties={FirstChild=true, BeginLine=74, DefiningType_CaseSafe=requesthelper, LastChild=true, DefiningType=RequestHelper, EndLine=74, childIdx=0, BeginColumn=34}}, trueValue=MethodCallExpressionVertex{fullMethodName=LANGUAGES_MAPPING.get, referenceVertex=LazyVertex{result=ReferenceExpression{properties={FirstChild=true, Names=[LANGUAGES_MAPPING], BeginLine=74, DefiningType_CaseSafe=requesthelper, LastChild=false, DefiningType=RequestHelper, EndLine=74, Name_CaseSafe=languages_mapping, childIdx=0, BeginColumn=54, ReferenceType=METHOD, Name=LANGUAGES_MAPPING}}}, chainedNames=[LANGUAGES_MAPPING], properties={FirstChild=false, FullMethodName=LANGUAGES_MAPPING.get, BeginLine=74, FullMethodName_CaseSafe=languages_mapping.get, DefiningType_CaseSafe=requesthelper, LastChild=false, DefiningType=RequestHelper, EndLine=74, MethodName_CaseSafe=get, childIdx=1, BeginColumn=72, MethodName=get}}, falseValue=VariableExpression{properties={FirstChild=false, BeginLine=74, DefiningType_CaseSafe=requesthelper, LastChild=true, DefiningType=RequestHelper, EndLine=74, Name_CaseSafe=lang, childIdx=2, BeginColumn=84, Name=lang}}

[jrabyIBM]

@jfeingold35 @johnbelosf thanks for your help with this issue. We are faced with the situation that our Salesforce app, which we have been shipping for over 3 years, is now prevented from shipping due to the more stringent security review process. We have no problem with that. We have addressed the security vulnerabilities that were identified, and would like to move forward, as we have customers waiting on our new release (and Salesforce is threatening to remove our app from the app store if we do not provide a new version that addresses the issues that were found).

However we are now not blocked by any actual security vulnerability, but rather by an inability to run this scan tool. And not due to any problem in our code, but due to issues with the scan tool itself, as discussed above. To be clear, if it was a matter of changing a single line of code to work around the scan tool deficiencies, we would do it right away. But if we need to go through our code base and update all lines of code that use multiple levels of object indirection, that would indeed require major surgery, and is not a best practice in software development. It doesn’t make the code any better or safer to break logic into multiple lines of code, due to fragility of the scanning tools. I hope that you’ll agree that something like “Object o = System.JSONToken.END_OBJECT;” is not complex code, and I’m sure that you will find examples like this (and much more complex statements) in any Salesforce app (and indeed, all over Salesforce’s own code base).

We are also facing two other problems with the tool, as discussed in https://github.com/forcedotcom/sfdx-scanner/issues/1216 and https://github.com/forcedotcom/sfdx-scanner/issues/1217 which no one has yet responded to.

We understand that Salesforce are doing their best to resolve these issues with the scanner, and we do appreciate your help. But in the meantime, we have very pressing business commitments that we need to meet. We are hoping that Salesforce can allow us to pass the security review with this incomplete scan outstanding, until such time that Salesforce is able to resolve the problems with the scanning tool.

[jfeingold35]

@jrabyIBM , I understand your frustration, but please note that InternalErrorExceptions are not a blocker for security review. They can be documented in your report the same as any other false positive. If you wouldn't mind, we'd love for you to indicate what led you to believe otherwise, so we can look into the possibility of modifying the relevant language to be more clear on this point.

[jrabyIBM]

@jfeingold35 thanks. It is good to know that InternalErrorExceptions can be documented as a false positive. What led us to think otherwise was the verbiage in the security review process which asks us to either submit a "clean Salesforce Code Analyzer report" or to respond why we didn't use the tool. Maybe a bad assumption on our part, but it sounded like submitting a report that is not "clean" was not one of the options. I'm still not sure if we should submit the report with the failures, or not submit a report at all, and cite the Git issues (including this one) as the reason for not submitting

[jfeingold35]

@jrabyIBM , submit the report with the failures, but document them with any other false positives. Clarifying question: Did the phrase "clean Salesforce Code Analyzer report" give you the impression that the report must contain no violations of any kind? Or were you aware that false positives were acceptable as long as they were sufficiently documented, but unaware that InternalExecutionErrors could also be documented in this fashion?

[jrabyIBM]

Yes, we misunderstood "provide a clean Salesforce Code Analyzer report". -Thanks for clarifying!

[johnbelosf]

No action needed