Useful Semgrep Links

[github-learning-lab[bot]]

This issue collects various links to useful Semgrep resources and documentation in one place so you can reference it if you ever get stuck.

• semgrep.dev - Semgrep's home page
  • @returntocorp/semgrep - Semgrep's source code on GitHub
• Semgrep Registry - Home of Semgrep's free, out-of-the-box rules (that is, security checks), written by r2c and the community.
  • These rules are grouped into "rulesets" that collect related functionality, like rules that check for secrets, target specific languages (e.g. javascript) or frameworks (e.g. django), or even entire vulnerability classes (e.g. xss or insecure transport).
  • @returntocorp/semgrep-rules - The source code of Semgrep's rules on GitHub.
  • Note that the Registry has more rules than what's just in the returntocorp/semgrep-rules repo, as the Registry includes Semgrep rules from other community repos, like NodeJSScan or Go rules by Damian Gryski.
• Semgrep Playground - Write and share Semgrep rules right from your browser, no installation required!

Rule Writing

There's a step by step rule writing tutorial here.

If you go to the Playground, you can also click the "Examples" button to view a number of illustrative built-in examples.

And of course, you can also review the over 1,000 rules in @returntocorp/semgrep-rules.

Docs

Semgrep has pretty extensive docs, which you can view here.

Of note:

• Pattern syntax - All of the ways you can match code within one pattern.
• Rule syntax - All of the ways you can combine Semgrep patterns to form more complex queries. For example, looking for code that matches this AND that, or this but NOT that, etc.

Community

Feel free to join the r2c community Slack to ask questions (we're super responsive!) or reach out to us on Twitter (@r2cdev), or send us an email at support@r2c.dev.

[ishaq201]

Done

[github-learning-lab[bot]]

Getting Started

Alright, first we'll do a few quick things to get you up and running.

At a high level, here's what we're going to do:

Join the r2c Community Slack - There's a channel for this workshop you can ask questions in, and we'll use it to set up notifications when Semgrep finds issues.

Create a free Semgrep App account - This lets us easily manage Semgrep in CI, set up notifications, configure scanning policy, view results over time, and more.

⌨️ Activity: Create a Dashboard Account, Set up Slack Notifications

1. Join the r2c community Slack and the #workshop-2021-owasp-devslop channel.
2. Log in to the Semgrep Dashboard.
3. Set up Slack Notifications.
   1. Go to the Incoming WebHooks page on the Slack App Directory, and in "Post to Channel" choose your name. This way, all notifications are going to be sent to you via direct message.
   2. Copy the "Webhook URL" generated on the next page (it should look like: https://hooks.slack.com/services/...) and go to the Semgrep Integrations page (you may need to click on "Integrations" in the left hand side navbar), create a new integration, select "Slack", provide a name, paste in the webhook url, then save it.
   3. Click the "Test" button, and you should see a message from Semgrep in Slack.
4. Now, on the Semgrep Policies page, click on each policy, go to Settings -> Integrations -> Add, select the Slack notification you set up, and click "Save".

---

Comment on this pull request when you're ready and I'll respond with the next step.

[ishaq201]

Done

[github-learning-lab[bot]]

Great! Now we're going to set up Semgrep scanning every PR via GitHub actions by creating a semgrep.yml.

Though we're going to be using GitHub Actions in this workshop, because Semgrep is nice and portable, easily runnable as a standalone binary or Docker, it's pretty easy to set up Semgrep in pretty much any CI platform under the sun.

See these docs for info about setting up Semgrep in GitLab, Buildkit, CircleCI, or other providers, and see here for more info about Semgrep in CI.

⌨️ Activity: Set up Semgrep in CI

1. On the Projects page, select the "Add CI job to GitHub project" option, and click the "Get started" button.
   1. Semgrep's GitHub App is going to ask for a few, minimal permissions so it can auto-set things up for you (create a PR adding semgrep.yml to repos you want to onboard, etc.).
   2. If you want, you can only add the Semgrep GitHub App to this intro-to-semgrep repo. If you want to add more repos, you select "All repositories" or hand select a few more. You can always update this later via your GitHub profile Installed Applications settings.
2. After you've authorized the Semgrep GitHub App, navigate back to the Projects page, click the "Refresh projects from GitHub button", and then click the "Add CI job" button next to the intro-to-semgrep repo row.
3. On the next page, click the "Commit file" button, then follow the instructions for setting up a GitHub Secret on the intro-to-semgrep repo.
   1. Note that we're adding the Secret to just this repo. If you want to run Semgrep across many of your or your org's repos, you probably want to add this Secret at your profile or org level, so you don't have to add it one repo at a time.
4. A Semgrep scan will automatically start. We'll examine the results later, for now, comment on this PR and let's get writing some rules!

---

Comment on this Pull Request once you've finished onboarding Semgrep (semgrep.yml GitHub Action) to this repo.

[ishaq201]

Done

[github-learning-lab[bot]]

Houston, We Have Scan Results 🚀

Great! I've merged in the Semgrep config you set up into this branch so we can iterate on rules and see the results right in this PR.

After the check suite finishes, you should see a PR comment warning about the use of eval() in the code this PR is adding in eval_test.ts.

And check your notifications in the r2c community Slack, you should see a message from the webhook flagging this issue as well.

[github-learning-lab[bot]]

The Power of Democratizing Static Analysis

One of the key differentiators about Semgrep is how easy it is to write custom rules.

This fundamentally changes how you can leverage static analysis to scale your AppSec program.

Rather than being a black box, one-size-fits-all, "I sure hope the vendor built all the use cases I could ever need," single purpose tool, Semgrep is a Swiss army knife and your imagination is the limit.

Yes, there are over 1,000 out-of-the-box security checks you get for free.

But you can also use Semgrep for:

• Enforcing secure guardrails: "You should be using our internal_auth library for all auth purposes."
• Just-in-time developer education: "Hey there, I see you're doing something dangerous like crypto or parsing XML, here's how we do it in our company: link to internal docs."
• Enforcing org or code base specific business logic: "foo() should always be called before bar(), else it's a bug."
• And much more

As Semgrep rules look just like the code you're targeting (with some helpful abstractions), many developers and engineering orgs can write custom rules as well (or better!) than security teams.

Why have separate tools when developers and the security team can solve their respective problems with the same tool! 🤝

Alright, let's get into it.

---

Writing Our First Rule

Semgrep Playground Overview

For these exercises we're going to be using the Semgrep playground: https://semgrep.dev/editor, as it's a convenient way to iterate on rules right from your browser, without installing anything.

If you'd prefer, you can also write Semgrep rules offline in your IDE of choice. After all, they're just YAML!

This is the rule we're going to start on, open it in a separate browser tab: https://semgrep.dev/s/clintgibler:juice-shop-eval-try.

UI Overview

In the top left, you can select a "Language" for the rule you're currently writing. In this case, we're using "TypeScript," because Juice Shop is mostly in TypeScript.

The "code is" section is where you write your Semgrep rule.

• This rule will then be ran against the "Test Code" section at the bottom.
• Any matches will be shown in the right hand side column.

The // ruleid:juice-shop-eval comments you see in the Test Code are a special syntax - they're telling Semgrep, "Hey, I expect Semgrep to find a match here."

• This makes it easy to write unit tests for your Semgrep rules, to ensure they're working as you expect. See the docs for more details.

If you click on the "Advanced" tab (next to "Simple" under the "Semgrep Rule" header on the left hand side), you'll see the raw YAML for the Semgrep rule you're writing. The "Simple" view is just a simplified interface so you don't have to write raw YAML and mess with indentation, etc.

Rule Writing Basics

At a high level, Semgrep rules are just the code you're targeting + a few abstractions.

The Ellipsis Operator

Sometimes you want to abstract away some details from the code you're matching, to make it more generic.

The ellipsis operator (...) lets you match zero or more arguments, statements, and more.

Here are a few examples:

// insecure_function(...) would match
insecure_function("MALICIOUS_STRING", arg1, arg2)

// var x = ...; would match each of these
var x = "semgrep";
var x = foo && bar || baz;
var x = foo(something);

You can think of the ellipsis operator like .* in regular expressions.

Metavariables

Sometimes you want to match something, but you don't know what it is ahead of time.

For example, the name of a function, the value of an argument, and so forth.

Metavariables let you do that by using an identifier that starts with a $ and is only uppercase letters, _, or digits. $X or $FOO for example.

Here are a few examples:

// foo($X)
foo(1);      // matches, $X = 1
foo(a);      // matches, $X = "a"

// foo($X) doesn't match, foo() called with >1 arg
foo(a, b, c);

// Ellipsis operator and metavariables can be combined!

foo(a, b, c);   // foo($X, ...) matches, $X = a
foo(a, b, c);   // foo(..., $Y) matches, $Y = c
foo(a);         // foo(..., $Y) matches, $Y = a

Note that within one pattern, metavariables are enforced to be the same.

So:

// bar($X, $X)
bar(a, a)   // matches
bar(10, 10) // matches
bar(a, b)   // does not match, a != b

You can think of metavariables kind of like capture groups in regular expressions.

Combining Patterns

Sometimes you want to combine Semgrep patterns, like:

• Find calls to a() or b()
• Find calls to foo() but not if the first parameter is a string literal
• Find calls to bar() but only if it occurs inside the MyClass class.

You can add additional pattern clauses in the simple editor by clicking the + button on the right hand side of the pattern.

Currently on a few Semgrep operators are available in the simple editor. See the rule syntax docs for all of the tools in your Semgrep rule writing toolbelt.

We'll cover a number of Semgrep's capabilities in this lab, but there are many we won't!

⌨️ Activity: Write Your First Custom Rule

Navigate to https://semgrep.dev/s/clintgibler:juice-shop-eval-try.

1. Update the pattern (currently TODO) to match all calls to my_eval(), regardless of the passed in arguments.
2. Update the pattern to only match calls to my_eval() with only 1 argument.
3. Update the pattern to only match calls to my_eval() when the first argument is not a string literal.
4. Save the rule, click the "Add to Policy" button in the top right, and select "Starter Policy."
   1. Navigate to the checks page for this PR and click the "Re-run jobs" button, to scan the changes with the new rule you wrote.

Hints

1: Match all calls to my_eval()

Try using the ellipsis operator, ....

2: Match all calls to my_eval() with 1 argument

Try using a metavariable, like $ARG.

3: Match all calls to my_eval() where the first argument is not a string literal

In Semgrep, "..." will match any string, regardless of its value (docs).
And pattern-not filters out matches.
Try clicking the + button to add a new pattern and select "and is not", which if you switch to the Advanced view, you can see is represented by pattern-not under the hood.

---

Comment on this Pull Request once you've re-scanned this PR with your new my_eval() rule.