search

github / codeql

CodeQL: the libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security
https://codeql.github.com
MIT License
7.32k stars 1.46k forks source link

CodeQL run time increased from mins to hours #16448

Open asreehari-splunk opened 1 month ago

asreehari-splunk commented 1 month ago

The code analysis run duration increased from mins to hours from 2.16.4. I've attached the runtime options as pdf for both versions below 2.16.4.pdf 2.16.3.pdf

It was consistently in the minute range before and has only increase since 2.16.4. The exact date seems to be March 12th, 2024 Latest version that is being used is "2.17.1"

mbg commented 1 month ago

Hi @asreehari-splunk 👋

We shipped some updates to our Go support in 2.16.4 which mean that we better understand large Go repositories and analyse more code than we did in previous versions of CodeQL. For large repositories, that can lead to big increases in the time needed for the analysis because we perform a lot more work than before.

We shipped some performance improvements in CodeQL 2.17.0 that should have improved the time again. Can you confirm that you've tried 2.17.0 or later? If so, has that made any difference at all?

asreehari-splunk commented 1 month ago

Hi @mbg ...thank you for the clarification. yeah, looks like we are currently using 2.17.1 and seeing 1h+ run time. here is a screenshot

Screenshot 2024-05-07 at 4 37 10 PM
asreehari-splunk commented 1 month ago

one follow up question. Is there a metric I can look for like # of lines-of-code/files/packages etc that qualifies as large ? to my knowledge there hasn't been a big change in the repo

aibaars commented 1 month ago

@mbg These are two runs at almost the same time, the main difference is the switch from 2.16.3 to 2.16.4 . The source code should be mostly identical. It's the autobuilding phase that is taking an hour more. Was there any change in the autobuilding strategy or is there a some performance problem in the extractor? I also see that CodeQL is running the go tests, these are not needed for analysis so perhaps test running can be skipped to speed things up.

aibaars commented 1 month ago

Looking at those logs, it could be that @mbg is right about 2.16.4 scanning more code:

Spending 6 times longer on 3 times as many files sounds like there may still be a performance issue. It can also be that the test cases for the additional files simply take a long time. Could you try a run without running any tests (a test pull request with a quick-n-dirty tweak to the Makefile should work).

mbg commented 1 month ago

Thanks @aibaars for having a look.

Like I said, we shipped big changes to the Go autobuilder in 2.16.4 which result in better support for repositories with multiple go.mod files. In CodeQL 2.16.3 and earlier, we didn't really support that well and our analysis of such repositories was just a best effort. As @aibaars notes for your repository, that meant we only extracted 155 out of 917 Go files then.

As of 2.16.4 and above, we do support repositories with multiple go.mod files properly and that often explains an increase in extraction/analysis time for such repositories, since we actually extract more code now. As @aibaars notes, that's now 488 out of 917 Go files.

I have had a look over your logs to determine why there is such a big increase in the time needed for this now and why this has not improved with 2.17.0 or above (when we shipped performance improvements to dependency extraction that improved the times for most users).

For mainly historical reasons, we run make if there is a Makefile in the repository before we begin extraction of the source code. Your Makefile in particular seems to build and test all of your code.

When we implemented the changes to the Go autobuilder in 2.16.4, we kept the part that invokes make before extraction to ensure that CodeQL would not suddenly break for repositories which relied on this behaviour. However, it seems that this now gets erroneously invoked for every go.mod file in your repository. I will look into getting this fixed ASAP.

In the meantime, to avoid this issue until it is fixed, you can either revert to 2.16.3 (but fewer Go sources files will get extracted) or switch to a custom build. The latter would involve replacing the autobuild step in your workflow with a step that invokes the right build commands for your repository (possibly just make).

asreehari-splunk commented 1 month ago

Thank you @aibaars and @mbg for the quick follow ups and really appreciate helping me understand the root cause. I will take this back and see what the team has to say. Do we need to change the label on this to something other than a question ?

mbg commented 1 month ago

I have updated the labels, but we are also tracking this internally as well.

mbg commented 3 weeks ago

@asreehari-splunk We merged https://github.com/github/codeql/pull/16704 yesterday which should fix this issue. I also tested it with your repo to confirm this. It will take a little before this trickles down into a CodeQL release, but once it has you should see your CodeQL workflow times improve.

It will still be a little longer than they used to be, because we do extract more code than before as of the changes that we shipped back in 2.17.0, but it shouldn't be nearly as long as the workflows take to run at the moment.

Sorry this took a bit longer to fix than I anticipated - we discovered a chain of other things that needed to be fixed or improved first and some of us were on holidays recently as well.

I will leave this issue open for now and would appreciate an update once you notice a change. Based on current plans, I am expecting this to ship in CodeQL 2.18.0, which should be released in about a month. If you want to give it a go before then, you could use one of our nightly releases by pointing the tools property of the codeql-action/init action at a recent CLI bundle (e.g. this one should work).

asreehari-splunk commented 4 days ago

Thanks Michael. I was away for a couple of weeks. will review this in the coming days

On Wed, Jun 12, 2024, 4:29 PM Michael B. Gale @.***> wrote:

[ External sender. Exercise caution. ]

@asreehari-splunk https://github.com/asreehari-splunk We merged #16704 https://github.com/github/codeql/pull/16704 yesterday which should fix this issue. I also tested it with your repo to confirm this. It will take a little before this trickles down into a CodeQL release, but once it has you should see your CodeQL workflow times improve.

It will still be a little longer than they used to be, because we do extract more code than before as of the changes that we shipped back in 2.17.0, but it shouldn't be nearly as long as the workflows take to run at the moment.

Sorry this took a bit longer to fix than I anticipated - we discovered a chain of other things that needed to be fixed or improved first and some of us were on holidays recently as well.

I will leave this issue open for now and would appreciate an update once you notice a change. Based on current plans, I am expecting this to ship in CodeQL 2.18.0, which should be released in about a month. If you want to give it a go before then, you could use one of our nightly releases https://github.com/dsp-testing/codeql-cli-nightlies/releases by pointing the tools property of the codeql-action/init action at a recent CLI bundle (e.g. this one https://github.com/dsp-testing/codeql-cli-nightlies/releases/download/codeql-bundle-20240612/codeql-bundle.tar.gz should work).

— Reply to this email directly, view it on GitHub https://github.com/github/codeql/issues/16448#issuecomment-2162716307, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARX2CBO6LW4NOLK3TRV4C5DZHASYFAVCNFSM6AAAAABHLVHCBGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSG4YTMMZQG4 . You are receiving this because you were mentioned.Message ID: @.***>