Codeql using open-api generator does not take into consideration .openapi-generator-ignore

[mihaileb]

We are using codeql in our Jenkins environment for a java library for a product. The library contains a submodule which uses the openapi-generator maven plugin to generate classes from some specs. We have added an .openapi-generator-ignore file containing the 3 files and locally it does not work. When we run in in the pipeline it sees that this setting is not taken into consideration and we have compilation failure detection due to some files which should not be generated but taken from the generated source folder.

[2024-03-17T21:42:59.680Z] info  codeqlExecuteScan - [2024-03-17 22:42:59] [build-stdout] [INFO] Recompiling the module because of changed dependency.
[2024-03-17T21:42:59.680Z] info  codeqlExecuteScan - [2024-03-17 22:42:59] [build-stdout] [INFO] Compiling 194 source files with javac [debug parameters release 17] to target/classes
[2024-03-17T21:43:00.247Z] info  codeqlExecuteScan - [2024-03-17 22:43:00] [build-stdout] [INFO] -------------------------------------------------------------
[2024-03-17T21:43:00.247Z] error codeqlExecuteScan - [2024-03-17 22:43:00] [build-stdout] [ERROR] COMPILATION ERROR : 
[2024-03-17T21:43:00.247Z] info  codeqlExecuteScan - [2024-03-17 22:43:00] [build-stdout] [INFO] -------------------------------------------------------------
[2024-03-17T21:43:00.247Z] error codeqlExecuteScan - [2024-03-17 22:43:00] [build-stdout] [ERROR] /home/jenkins/agent/workspace/tityService_dim-client-java_main/dim-client-domain-model/generated/src/gen/java/main/com/sap/cp/cf/dim/client/model/DIDDocumentDTOContext.java:[67,58] illegal start of type

Have you encountered such an issue? Thank you, Bruno.

[smowton]

Do you mean that locally it DOES work, and in your Jenkins environment it doesn't? If so, are you able to run a plain old mvn package in your Jenkins environment?

What CodeQL database creation command are you using in your jenkins env? If it's plain old codeql database create -l java db then it should run effectively the same mvn package routine, but perhaps you're using a custom build command? If so does that build command complete successfully without CodeQL but in your Jenkins environment?

Are you able to share a complete log, or to check if your problem also occurs in a github actions environment, or to share a toy example that reproduces the example so we can investigate in more detail?

[mihaileb]

Hello Ton, Yes, locally it works fine. Also command mvn clean install works fine in our Central Build stage in Jenkins. We are code ql scan step in our Security stage: https://www.project-piper.io/steps/codeqlExecuteScan/ However, our project piper is internally managed (by SAP). Also our codeql scan step configuration is something like this:

codeqlExecuteScan: dockerImage: 'piper.int.repositories.cloud.sap/piper/codeql:jdk17' uploadResults: true querySuite: "java-security-extended.qls" buildTool: 'maven' buildCommand: 'mvn package -DskipTests=true'

Now could it be the maven configuration within this dockerImage? As I mentioned, mvn seems to compile generated classes ( even the ones ignored by the .openapi-generator-ignore file)

Thank you, Bruno

[smowton]

What happens if you run mvn package -DskipTests=true inside that docker image without codeql?

[mihaileb]

Hello Ton,

Sorry, I got with something else. This works perfectly fine, we are actually running in the Central build stage this. Only when using the codeql docker image, we are facing this issue.

[smowton]

Can you please post the rest of the log of the failed run?

[mihaileb]

Hello Ton, The issue is I think that using the codeql dependency, it triggers the recompile of the module: codeqlExecuteScan - [2024-04-24 06:28:32] [build-stdout] [INFO] Recompiling the module because of changed dependency. Maybe this is a maven issue? The docker image used has:

09:27:30 [INFO] executing pipeline step 'codeqlExecuteScan' with docker image 'piper.int.repositories.cloud.sap/piper/codeql:jdk17' 09:27:30 [Pipeline] echo 09:27:30 [INFO] executing pipeline step 'detectExecuteScan' with docker image 'maven:3.8.4-openjdk-17' How can I prevent the recompilation of the module?

Thank you, Bruno

[smowton]

This is expected: CodeQL works by executing a clean build and tracing the Java compilation steps that result from that. If the compilation was skipped, CodeQL would not analyse the code being compiled.

There are two alternatives: fix the environment so that the rebuild can successfully execute, or consider using the new Java buildless mode (build-mode: none; see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages#example-using-the-build-mode-option) which does not depend on actually building your code (but will still benefit from querying Maven for your code's dependencies, if at least that can be made to work, though it can function even without this)

[mihaileb]

Hello Ton,

We are using open-generatorapi maven plugin for generating some classes. now, the problem is that locally and on any other build, the .openapi-generator-ignore file is found and used. when we use codeqlscan, it does not find it: codeqlExecuteScan - [2024-05-29 07:54:42] [build-stdout] [INFO] No .openapi-generator-ignore file found. We have this file commited in github. any ideas? Thank you

[smowton]

@mihaileb can you post or link to the full log?

[mihaileb]

Hello Ton, actually I have found the issue. The problem was in our pipeline setup. The .openapi-ignore file was not stashed in the respective stage of the codeql run. Finally solved. Thank you