Open aeisenberg opened 2 years ago
This is meant to be a smoke test and lightly exercise all of the extension's main features.
First stab at a plan (at no point in these steps should there be an error message in a popup). These steps are for the local queries only. Someone on the Sec Exp team can fill in the portion about MRVA.
vsix
file of the build you want to test.qlpack.yml
file in the root of the main folder with these contents:
name: dsp-testing/vscode-manual-testing
version: 0.0.1
dependencies:
codeql/ruby-all: "*"
Create a query.ql
file with the following contents:
/**
* @name 6 block
* @kind problem
* @problem.severity warning
* @id ruby/example/six-block
*/
import ruby
from Block b
where b.getNumberOfStatements() > 5 and b.getNumberOfStatements() <= 6
select b, "This is a 6 block."
codeql/ruby-all
pack is not downloaded and available in your package cache.
a. If you do have compilation errors: Run the command CodeQL: Install Pack Dependencies
b.getNumberOfStatements() > 5 and b.getNumberOfStatements() <= 6
CodeQL: Quick Evaluation
rails/rails
source folder. actioncable/lib/action_cable/engine.rb
is a good oneI tried to focus on only the features that are of high importance and have been problematic in the past. There's much more we can test, but I think there are diminishing returns when we start getting to more obscure features.
We've started a test plan doc for the MRVA side of things. It'll be good to combine forces! It's currently a private doc (for GH employees) so I won't share here (see pinned items on our team channel) but I'm keen to make this information public.
This issue is about creating a simple and easy to follow manual test plan that any release manager will perform before the release.
It might be overkill, but we can write tests semi-formally in a Gherin style.
My initial thoughts are that we just create a simple set of steps that are easy to follow, but are expressed informally. We can formalize later if we have any need.