[BUG] apex.jorje.parser.impl.BaseApexLexer dedupe

[ptrsconsulting]

Have you tried to resolve this issue yourself first?

Yes

Bug Description

When I run the scan with the following parameters: sf scanner run dfa --projectdir force-app/main/default/ --target force-app/main/default/classes/MyClass.cls --sfgejvmargs "-Xmx4g" --format csv --outfile output/dfa_MyClass.csv I get this error:

Analyzing with Salesforce Graph Engine. See /home/pietras/.sfdx-scanner/sfge.log for details.... Error
Error (1):

 Aug 28, 2024 11:23:15 AM apex.jorje.parser.impl.BaseApexLexer dedupe
INFO: Deduped array ApexLexer.DFA22_transition. Found 7075545 shorts which is 13MB not including array overhead. Removed 6418975 shorts which is 12MB not counting array overhead. Took 42 ms.

It looks like same issue as in https://github.com/forcedotcom/sfdx-scanner/issues/1234

Output / Logs

2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass155.cls 2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass156.cls 2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass157.cls 2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass158.cls 2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass159.cls 2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass160.cls 2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass161.cls 2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass162.cls 2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass163.cls 2024-08-28 13:28:34 INFO GraphUtil:207 - Loading file. path=/home/pietras/workspace/scanner/sfdc/force-app/main/default/classes/MyClass164.cls

Steps To Reproduce

1. checkout the code
2. run the command mentioned in the description
3. get the error, no output file, nothing in the logs

Expected Behavior

I would expect the scan to finish with no errors and creating an output file.

Operating System

Ubuntu 22.04, Windows 10

Salesforce CLI Version

@salesforce/cli/2.55.6 linux-x64 node-v20.17.0

Code Analyzer Plugin (@salesforce/sfdx-scanner) Version

@salesforce/sfdx-scanner 4.4.0 (4.4.0)

Java Version

openjdk version "11.0.24" 2024-07-16

Additional Context (Screenshots, Files, etc)

No response

Workaround

No response

Urgency

Moderate

[ptrsconsulting]

Additionally I tried scanning just an empty class

public class DoNothing {

    @InvocableMethod
    public static void DoNothing() {

    }

}

and it still failed with the same error:

Error (1):

 Aug 29, 2024 8:49:03 AM apex.jorje.parser.impl.BaseApexLexer dedupe
INFO: Deduped array ApexLexer.DFA22_transition. Found 7075545 shorts which is 13MB not including array overhead. Removed 6418975 shorts which is 12MB not counting array overhead. Took 16 ms.

[stephen-carter-at-sf]

I'm unable to reproduce with just the DoNothing class in my project directory. The issue must be with another file somewhere in your project.

Can you try to narrow down the issue, but starting with a new blank folder called myProject and just add in the DoNothing.cls to the folder, and then run:

sf scanner run dfa --projectdir myProject --target myProject/DoNothing --sfgejvmargs "-Xmx4g"

and if that passes, add in some more of your class files into your myProject folder... keep repeating until you figure out which file is causing the error. Then please report back here with a copy of that file so that we can reproduce it on our end.

[ptrsconsulting]

Well, there are hundreads of classes in the project, but I'll do my best 😅

[ptrsconsulting]

OK, the results were not conclusive as I got the errors with random situations, same scope, random results (i.e. error/success) so I went another route. We've created a big and powerful AWS instance and run the scan there and it went through successfully with absolutely no errors but I had to grant 16g of RAM just for the process.

So I guess we can close this issue as a problem between the size of the project and the hardware.