Few questions about semmle-extractor-options

[artem-smotrakov]

Hey team!

1. Is there any documentation about semmle-extractor-options? I see some docs that mention extractor options but I can't find a list of available extractor options (like --expect_errors, etc). I usually look for how semmle-extractor-options is used in the existing CodeQL tests but that's not very convenient and I can probably miss some useful options.

2. I see ${testdir} in some semmle-extractor-options. Is it possible to set custom variables? I tired something like export myvar=... ; codeql test run ... but it throws Attempting to expand unknown variable ... error.

[mbg]

Hi @artem-smotrakov 👋

Thanks for the questions!

1. I have done some digging, but don't think there's any documentation for the options available. I will check with the relevant team to see if they have anything.

2. I don't believe there is a way to set custom variables. The set of variables that are expanded is deliberately limited. What would you be hoping to have custom variables for?

[artem-smotrakov]

Hi @mbg !

I will check with the relevant team to see if they have anything.

That'd be great, thank you!

The set of variables that are expanded is deliberately limited. What would you be hoping to have custom variables for?

I have CodeQL tests that based on header files form a separate codebase. Those are not unit tests but some sort of "integration" tests if that makes sense. To run the tests, I need to have the codebase locally, and I need to let the tests know where the codebase is. I use semmle-extractor-options, it looks like this

// file: test.cpp
#include <path/to/some/header/file/from/the/codebase>
[...]
// semmle-extractor-options: -I${testdir}../../../../codebase/include

I was wondering if I could have a custom variable to make it look a bit better

// file: test.cpp
#include "<path/to/some/header/file/from/the/codebase>"
[...]
// semmle-extractor-options: -I${codebase}/include

It's not a big deal though, the workaround with ${testdir} works fine.

But setting custom variable might be helpful to make tests a bit more customizable.

[mbg]

@artem-smotrakov Thanks for the extra detail, your explanation makes sense. I have checked with the team and there is no way to improve on the solution you already have with this approach. The semmle-extractor-options mechanism is not really designed be used by anything other than some of our own tests, which is why there is no documentation and no convenience features like custom variables.

[artem-smotrakov]

Hi @mbg Thanks for the context!

I see semmle-extractor-options is used in many tests in this repository. From my personal experience, this is quite a useful feature for writing tests using the standard CodeQL test framework. I'd be great if it became an officially supported feature one day. I believe it'd make it easier to contribute queries and tests in this repository as well.

For custom variables, it's definitely more a convenience feature which would not be probably used much.