Welcome to Semgrep!

[github-learning-lab[bot]]

Welcome!

I'm excited you're here! 👋

Together we're going to see how we can quickly and easily set up continuous code scanning using Semgrep, an open source, lightweight static analysis tool.

We'll see how Semgrep's out-of-the-box rules can find and block a broad variety of vulnerabilities and enforce secure guardrails (also called "paved road" or "secure defaults").

We'll use the awesome OWASP Juice Shop project as the repo we'll scan, and we'll use GitHub Actions to scan every Pull Request (PR).

How This Lab Works

Basically, at each stage you'll be provided with some information, either as a GitHub issue, PR, or a comment on one of those.

Then, there'll be an ⌨️ Activity section at the bottom, that has you complete some concrete steps, either in this repo (like editing files, opening or closing PRs or Issues) or on Semgrep-related sites (e.g writing new rules, setting up and configuring your dashboard, etc.).

After you complete the steps in the Activity section, the bot will either autodetect what you've done and move you to the next step, or perhaps respond to a comment we ask you to write.

💡 Important Notes

If at any point throughout this lab you're not seeing a bot response or scan update that you'd expect to, try refreshing the page, sometimes things get in a wonky state.

⌨️ Activity: See Docs Links

1. We created a new Issue with useful documentation for you to review if you get stuck. Give it a quick skim.
2. Comment on this issue and the bot will respond with next steps 🚀

---

I'll respond in this pull request when I detect a comment posted to it.

[tveiga1689]

ok

[github-learning-lab[bot]]

Getting Started

Alright, first we'll do a few quick things to get you up and running.

At a high level, here's what we're going to do:

Join the r2c Community Slack - There's a channel for this workshop you can ask questions in, and we'll use it to set up notifications when Semgrep finds issues.

Create a free Semgrep App account - This lets us easily manage Semgrep in CI, set up notifications, configure scanning policy, view results over time, and more.

⌨️ Activity: Create a Dashboard Account, Set up Slack Notifications

1. Join a Slack channel that allows you to add webhook notifications, or create a new Slack instance if you don't have one available.
2. Log in to the Semgrep Dashboard.
3. Set up Slack Notifications.
   1. Visit the Slack App Directory (https://your_slacks_name.slack.com/apps), search "Incoming WebHooks", and in "Post to Channel" choose your name. This way, all notifications are going to be sent to you via direct message.
   2. Copy the "Webhook URL" generated on the next page (it should look like: https://hooks.slack.com/services/...) and go to the Semgrep Integrations page (you may need to click on "Integrations" in the left hand side navbar), create a new integration, select "Slack", provide a name, paste in the webhook url, then save it.
   3. Click the "Test" button, and you should see a message from Semgrep in Slack.
   4. See the Slack integration docs for additional details.
4. Now, on the Semgrep Policies page, click on each policy, go to Settings -> Integrations -> Add, select the Slack notification you set up, and click "Save".

Feel free to join the r2c community Slack and ask questions in #general or #workshop-2021-owasp-devslop if anything is unclear.

---

Comment on this pull request when you're ready and I'll respond with the next step.

[tveiga1689]

ok

[github-learning-lab[bot]]

Great! Now we're going to set up Semgrep scanning every PR via GitHub actions by creating a semgrep.yml.

Though we're going to be using GitHub Actions in this workshop, because Semgrep is nice and portable, easily runnable as a standalone binary or Docker, it's pretty easy to set up Semgrep in pretty much any CI platform under the sun.

See these docs for info about setting up Semgrep in GitLab, Buildkit, CircleCI, or other providers, and see here for more info about Semgrep in CI.

⌨️ Activity: Set up Semgrep in CI

1. On the Projects page, select the "Add CI job to GitHub project" option, and click the "Get started" button.
   1. Semgrep's GitHub App is going to ask for a few, minimal permissions so it can auto-set things up for you (create a PR adding semgrep.yml to repos you want to onboard, etc.).
   2. If you want, you can only add the Semgrep GitHub App to this intro-to-semgrep repo. If you want to add more repos, you select "All repositories" or hand select a few more. You can always update this later via your GitHub profile Installed Applications settings.
2. After you've authorized the Semgrep GitHub App, navigate back to the Projects page, click the "Refresh projects from GitHub button", and then click the "Add CI job" button next to the intro-to-semgrep repo row.
3. On the next page, click the "Commit file" button, then follow the instructions for setting up a GitHub Secret on the intro-to-semgrep repo.
   1. Note that we're adding the Secret to just this repo. If you want to run Semgrep across many of your or your org's repos, you probably want to add this Secret at your profile or org level, so you don't have to add it one repo at a time.
4. A Semgrep scan will automatically start. We'll examine the results later, for now, comment on this PR and let's get writing some rules!

---

Comment on this Pull Request once you've finished onboarding Semgrep (semgrep.yml GitHub Action) to this repo.

[tveiga1689]

ok 3

[github-learning-lab[bot]]

Houston, We Have Scan Results 🚀

Great! I've merged in the Semgrep config you set up into this branch so we can iterate on rules and see the results right in this PR.

After the check suite finishes, you should see a PR comment warning about the use of eval() in the code this PR is adding in eval_test.ts.

And check your notifications in the r2c community Slack, you should see a message from the webhook flagging this issue as well.

[github-learning-lab[bot]]

The Power of Democratizing Static Analysis

One of the key differentiators about Semgrep is how easy it is to write custom rules.

This fundamentally changes how you can leverage static analysis to scale your AppSec program.

Rather than being a black box, one-size-fits-all, "I sure hope the vendor built all the use cases I could ever need," single purpose tool, Semgrep is a Swiss army knife and your imagination is the limit.

Yes, there are over 1,000 out-of-the-box security checks you get for free.

But you can also use Semgrep for:

• Enforcing secure guardrails: "You should be using our internal_auth library for all auth purposes."
• Just-in-time developer education: "Hey there, I see you're doing something dangerous like crypto or parsing XML, here's how we do it in our company: link to internal docs."
• Enforcing org or code base specific business logic: "foo() should always be called before bar(), else it's a bug."
• And much more

As Semgrep rules look just like the code you're targeting (with some helpful abstractions), many developers and engineering orgs can write custom rules as well (or better!) than security teams.

Why have separate tools when developers and the security team can solve their respective problems with the same tool! 🤝

Alright, let's get into it.

---

Writing Our First Rule

Semgrep Playground Overview

For these exercises we're going to be using the Semgrep playground: https://semgrep.dev/editor, as it's a convenient way to iterate on rules right from your browser, without installing anything.

If you'd prefer, you can also write Semgrep rules offline in your IDE of choice. After all, they're just YAML!

This is the rule we're going to start on, open it in a separate browser tab: https://semgrep.dev/s/clintgibler:juice-shop-eval-try.

UI Overview

In the top left, you can select a "Language" for the rule you're currently writing. In this case, we're using "TypeScript," because Juice Shop is mostly in TypeScript.

The "code is" section is where you write your Semgrep rule.

• This rule will then be ran against the "Test Code" section at the bottom.
• Any matches will be shown in the right hand side column.

The // ruleid:juice-shop-eval comments you see in the Test Code are a special syntax - they're telling Semgrep, "Hey, I expect Semgrep to find a match here."

• This makes it easy to write unit tests for your Semgrep rules, to ensure they're working as you expect. See the docs for more details.

If you click on the "Advanced" tab (next to "Simple" under the "Semgrep Rule" header on the left hand side), you'll see the raw YAML for the Semgrep rule you're writing. The "Simple" view is just a simplified interface so you don't have to write raw YAML and mess with indentation, etc.

Rule Writing Basics

At a high level, Semgrep rules are just the code you're targeting + a few abstractions.

The Ellipsis Operator

Sometimes you want to abstract away some details from the code you're matching, to make it more generic.

The ellipsis operator (...) lets you match zero or more arguments, statements, and more.

Here are a few examples:

// insecure_function(...) would match
insecure_function("MALICIOUS_STRING", arg1, arg2)

// var x = ...; would match each of these
var x = "semgrep";
var x = foo && bar || baz;
var x = foo(something);

You can think of the ellipsis operator like .* in regular expressions.

Metavariables

Sometimes you want to match something, but you don't know what it is ahead of time.

For example, the name of a function, the value of an argument, and so forth.

Metavariables let you do that by using an identifier that starts with a $ and is only uppercase letters, _, or digits. $X or $FOO for example.

Here are a few examples:

// foo($X)
foo(1);      // matches, $X = 1
foo(a);      // matches, $X = "a"

// foo($X) doesn't match, foo() called with >1 arg
foo(a, b, c);

// Ellipsis operator and metavariables can be combined!

foo(a, b, c);   // foo($X, ...) matches, $X = a
foo(a, b, c);   // foo(..., $Y) matches, $Y = c
foo(a);         // foo(..., $Y) matches, $Y = a

Note that within one pattern, metavariables are enforced to be the same.

So:

// bar($X, $X)
bar(a, a)   // matches
bar(10, 10) // matches
bar(a, b)   // does not match, a != b

You can think of metavariables kind of like capture groups in regular expressions.

Combining Patterns

Sometimes you want to combine Semgrep patterns, like:

• Find calls to a() or b()
• Find calls to foo() but not if the first parameter is a string literal
• Find calls to bar() but only if it occurs inside the MyClass class.

You can add additional pattern clauses in the simple editor by clicking the + button on the right hand side of the pattern.

Currently on a few Semgrep operators are available in the simple editor. See the rule syntax docs for all of the tools in your Semgrep rule writing toolbelt.

We'll cover a number of Semgrep's capabilities in this lab, but there are many we won't!

⌨️ Activity: Write Your First Custom Rule

Navigate to https://semgrep.dev/s/clintgibler:juice-shop-eval-try.

1. Update the pattern (currently TODO) to match all calls to my_eval(), regardless of the passed in arguments.
2. Update the pattern to only match calls to my_eval() with only 1 argument.
3. Update the pattern to only match calls to my_eval() when the first argument is not a string literal.
4. Save the rule, click the "Add to Policy" button in the top right, and select "Starter Policy."
   1. Navigate to the checks page for this PR and click the "Re-run jobs" button, to scan the changes with the new rule you wrote.

Hints

1: Match all calls to my_eval()

Try using the ellipsis operator, ....

2: Match all calls to my_eval() with 1 argument

Try using a metavariable, like $ARG.

3: Match all calls to my_eval() where the first argument is not a string literal

In Semgrep, "..." will match any string, regardless of its value (docs).
And pattern-not filters out matches.
Try clicking the + button to add a new pattern and select "and is not", which if you switch to the Advanced view, you can see is represented by pattern-not under the hood.

---

Comment on this Pull Request once you've re-scanned this PR with your new my_eval() rule.

[tveiga1689]

ok

[github-learning-lab[bot]]

Congrats on writing your first rule! 🎉

Holy moly, you:

• Rolled out Semgrep in CI
• Learned how to write custom rules
• Wrote a custom rule
• and rolled it out to scan every new PR

...all in a few minutes 🤯

Automate the Boring Parts of Code Review

Are there comments you (or developers at your company) often write on PRs?

Wouldn't it be nice if you could automate that work and spend your time on higher leveraged things? I think you know where I'm going with this 😉

Scaling to Thousands of Repos

Oh another thing - did you notice how easy it was to add new rules you write to your scanning policy, with one click from the Playground?

Well imagine you're scanning 100s or 1,000s of repos with Semgrep, and there's something new you'd like to enforce, whether it's a secure guardrail, a new anti-pattern you'd like to block, based on a recent penetration test report or bug bounty submission, etc.

So you quickly write the rule in the Playground, add it to one of your scanning policies, and then boom, that rule is immediately going to run on every new PR for repos using that policy.

No need to file PRs on hundreds of repos, no need to wait on developers or DevOps teams acting on your request, just quick security coverage, everywhere.

(Note: of course you want to roll out new rules carefully, to ensure they're high signal, don't bother our developer friends, etc.)

⌨️ Activity: Audit Routes

Time for the next rule writing challenge!

I've opened up a new PR with more code to match: click here to continue.

---

Visit the next PR to continue.