Add CodeQL Workflow for Code Security Analysis

[b4yuan]

Summary

This pull request introduces a CodeQL workflow to enhance the security analysis of this repository.

What is CodeQL

CodeQL is a static analysis tool that helps identify and mitigate security vulnerabilities. It is primarily intra-function but does provide some support for inter-function analysis. By integrating CodeQL into a GitHub Actions workflow, it can proactively identify and address potential issues before they become security threats.

For more information on CodeQL and how to interpret its results, refer to the GitHub documentation and the CodeQL documentation (https://codeql.github.com/ and https://codeql.github.com/docs/).

What this PR does

We added a new CodeQL workflow file (.github/workflows/codeql.yml) that

• Runs on every pull request (functionality to run on every push to main branches is included as a comment for convenience).
• Runs daily.
• Excludes queries with a high false positive rate or low-severity findings.
• Does not display results for git submodules, focusing only on our own codebase.

Validation

To validate the functionality of this workflow, we have run several test scans on the codebase and reviewed the results. The workflow successfully compiles the project, identifies issues, and provides actionable insights while reducing noise by excluding certain queries and third-party code.

Using the workflow results

If this pull request is merged, the CodeQL workflow will be automatically run on every push to the main branch and on every pull request to the main branch. To view the results of these code scans, follow these steps:

1. Under the repository name, click on the Security tab.
2. In the left sidebar, click Code scanning alerts.

Is this a good idea?

We are researchers at Purdue University in the USA. We are studying the potential benefits and costs of using CodeQL on open-source repositories of embedded software.

We wrote up a report of our findings so far. The TL;DR is “CodeQL outperforms the other freely-available static analysis tools, with fairly low false positive rates and lots of real defects”. You can read about the report here: https://arxiv.org/abs/2310.00205

Review of engineering hazards

License: see the license at https://github.com/github/codeql-cli-binaries/blob/main/LICENSE.md:

Here's what you may also do with the Software, but only with an Open Source Codebase and subject to the License Restrictions provisions below:

Perform analysis on the Open Source Codebase.

If the Open Source Codebase is hosted and maintained on GitHub.com, generate CodeQL databases for or during automated analysis, CI, or CD.

False positives: We find that around 20% of errors are false positives, but that these FPs are polarized and only a few rules contribute to most FPs. We find that the top rules contributing to FPs are: cpp/uninitialized-local, cpp/missing-check-scanf, cpp/suspicious-pointer-scaling, cpp/unbounded-write, cpp/constant-comparison, and cpp/inconsistent-null-check. Adding a filter to filter out certain rules that contribute to a high FP rate can be done simply in the workflow file.

[kstribrnAmzn]

Thanks for bringing up this PR! We currently use Coverity for our static analysis tool. I will discuss with the team to see if we would like to take on CodeQL as a supported SAST.

[Machiry]

Thank you @kstribrnAmzn, for the response.

Our run of CodeQL on this repo found several issues. We have raised pull requests for the corresponding issues, i.e., https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1861, https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1860, https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1859 Given that you run Coverity, the above pull requests provide concrete evidence of things missed by Coverity but found by CodeQL.

[kstribrnAmzn]

CodeQL does not appear to be under any form of the MIT license. It has a custom license, provided here. With this in mind, this change would need to be reviewed by our legal department to ensure this software use is acceptable. This may take significant time (likely weeks).

I just want to keep you in the loop regarding the timeline. I will submit this for review to legal.

[b4yuan]

Hey @kstribrnAmzn, any update on this?

[zhuang-hy]

已收到，我会尽快回复！

[kstribrnAmzn]

Thanks for you patience @b4yuan! I did receive the okay to proceed from legal roughly a month ago but this slipped by me. I've left a couple small comments on the PR. If you can address those then I'd be happy to get this PR approved and merged.

My thoughts are to accept your CodeQL integration into this repo, and if we find it valuable - which it sound like we will given it's already finding results - then we'll rework this integration into our CI-CD actions repository for FreeRTOS-wide consumption.

So in short - thanks for this pull request. I think it will really move the quality bar on FreeRTOS libraries.

[kstribrnAmzn]

I'm going to approve and then remove my approval. I would like to see the action run and view the artifacts which get created.

[b4yuan]

@kstribrnAmzn I've gone ahead and made those changes, thanks for the update!

[github-advanced-security[bot]]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[kstribrnAmzn]

I looked through your paper and the results of CodeQL are impressive. Do you have any data on how it compares to SonarCloud or Coverity? I would like to have data to show that this CodeQL performs better than the two tools we use currently. I worry that adding another SAST tool on top of what we have could introduce additional false positives (at worst case) and more Github actions to maintain (at best case).

[Machiry]

@kstribrnAmzn Unfortunately, the mentioned products prohibit performing or publicizing any comparative evaluation. Nonetheless, we have some indirect evidence (Here: https://github.com/aws/aws-iot-device-sdk-embedded-C/pull/1894#issuecomment-1786036115 ) that CodeQL can find issues missed by Coverity on aws-iot-device-sdk.

[kstribrnAmzn]

I've talked about any potential comparison risk here with my team and I believe were safe since we will be using CodeQL and other SAST tooling. Not comparing them.

With this in mind I'll review this and make any necessary tweaks to add this as an optional CI check. Once we known how CodeQL performs, and if it does indeed catch addition bugs, we can upgrade this to a required check.

[b4yuan]

@kstribrnAmzn I've updated the permissions here

[kstribrnAmzn]

Thank you @b4yuan! Tagging @aggarg as he had further ideas on how this should be integrated.

[aggarg]

Thank you for your contribution! CodeQL seems to be a great static code analysis tool. We currently use Coverity for static code analysis. At the moment, we do not intend to increase the number of static analysis tools that we run on each PR to limit the operational burden, which includes keeping the tools up to date, eliminating false positives etc.

Having said that, we certainly want to enable users who want to run CodeQL. We have recently created this file which links all the third-party tools. Would you please host this code in a repo and share the link with so that we can add it to the same file?

In future, if we reconsider the decision about static analysis tool, we will certainly consider your inputs about CodeQL.

Thanks.

[aggarg]

@b4yuan Did you get a chance to read my comment above? Do you plan to host it in share the link with us?

[aggarg]

@b4yuan I am closing this PR. Please feel free to re-open or create a new one if you need anything else from us.