Implement Go code coverage

[koltiradw]

Hi! I'm currently trying to start using Wuppie for fuzzing go apps. But unfortunately, after a while, crashes appear and Wuppie execution terminates.

How can I debug it?

[ThomasTNO]

Hi @koltiradw,

Finding crashes is the goal of WuppieFuzz. Depending on the configuration crashes are either 500 internal server errors, or deviations from the specification.

Crash files are written to the crashes directory. You can reproduce the find by the following command:

cargo run -- reproduce [OPTIONS] <CRASH_FILE>

use the --help flag if you need more info.

It seems like the fuzzer gracefully exited. If you specify a timeout this is expected behaviour. Just to be sure, what flags / configuration did you use?

[koltiradw]

None of the crashes are reproducible.

## CONFIG FOR USING LCOV TO CREATE COVERAGE REPORTS FOR FUZZING PYTHON CODE

# format used for coverage
coverage_format: lcov

# Optional total fuzzing time-out in seconds.
timeout: 60
# If present, ask the coverage monitor to generate a report after the time-out passes.
#report: false

# Must be one of {'json', 'human-readable'}
output_format: human-readable

## CHANGE TO YOUR OWN PATHS:

# The path to the open api specification of the target.
openapi_spec: openapi.yaml

# Where is the coverage host running, can be either a hostname or an IP address. Must include a port.
coverage_host: 127.0.0.1:3001

# When generating a coverage report, look for source files in this directory
source_dir: "/home/koltiradw/swag/example/celler"
## ADDITIONAL OPTIONAL FIELDS:

## Must be one of {'off', 'error', 'warn', 'info', 'debug', 'trace'}
# log_level: info

## The path to an initial corpus given in yaml.
initial_corpus: corpus

## Per-request time-out in milliseconds. Defaults to 30 seconds.
# request_timeout: 30000

## If present, reproduces the crash given an input file, then quits.
# reproduce: test_request.yaml

## How to log in to the API server, if applicable. See login.md.
# authentication: api_authentication.yaml

[ThomasTNO]

Could you share the steps you took? i.e. the target and how you instrumented it.

I will check if I can reproduce it locally to see what goes wrong here.

[koltiradw]

I increased the timeout. It helped a little. So it turns out that the timeout is how much time has passed since the last increase in coverage?

[ThomasTNO]

The timeout is the the total time the fuzzer should run (minimally) in seconds.

As soon as it finishes the fuzz_one call on https://github.com/TNO-S3/WuppieFuzz/blob/c7b9352e7df99d1dfb22d65ca3fc72b1947a81dd/src/fuzzer.rs#L295

For that reason the campaign can take longer than the set timeout.

If you do not specify a timeout it should run indefinitely

[koltiradw]

Oh. i see. Thx.

[koltiradw]

Configuration:

• git clone https://github.com/swaggo/swag.git

• cd swag/example/celler/

• Add _ "github.com/koltiradw/gcs" to imports block

• go get github.com/koltiradw/gcs

• go build -covermode=atomic -cover

• go install github.com/jandelgado/gcov2lcov@latest

Run:

• mkdir /tmp/coverage/
• GOCOVERDIR="/tmp/coverage/" ./cellar

[grebnetiew]

None of the crashes are reproducible.

Unfortunate :( To quickly see some examples of 'crashing' requests and responses, you can open the endpoint coverage report in the fuzzer's working directory located at ./reports/<datetime of fuzzing session>/endpointcoverage/index.html.

If you need more context for debugging a specific crash, I recommend using the grafana dashboard to inspect the database generated during fuzzing. See https://github.com/TNO-S3/WuppieFuzz-dashboard for instructions on how to use it. This allows you to see all requests and responses in a fuzzing session, so you can also see what happened that led to the crash.

[ThomasTNO]

For completeness I attached the converted and used openapi.json

A crash file as an example: cc86faf759acfe23.txt (added the txt extension to be able to upload it here..)

When I reproduce

cargo run -- reproduce --openapi-spec <path to penapi.json> cc86faf759acfe23.txt

I get

Input file "crashes/cc86faf759acfe23" contains 4 inputs
[2024-09-09T12:18:10Z INFO  wuppiefuzz::reproducer]
    -----
    Sending request:
    PATCH /accounts/{id}
      id in Path: "\rF\u0002\u0000Xlv;"Contents in body: {"id": Bytes([73, 36, 238, 73, 36, 0, 9, 68]), "name": Bytes([111, 119, 0, 17, 119, 119, 3, 232]), "uuid": Bytes([35, 81, 40, 20, 10, 5, 2, 65])}
[2024-09-09T12:18:10Z INFO  wuppiefuzz::reproducer] Converted to CURL command:
    echo eyJpZCI6Ikkk77+9SSRcdTAwMDBcdEQiLCJuYW1lIjoib3dcdTAwMDBcdTAwMTF3d1x1MDAwM++/vSIsInV1aWQiOiIjUShcdTAwMTRcblx1MDAwNVx1MDAwMkEifQ== | \
    base64 --decode | \
    curl http://localhost:8080/api/v1/accounts/%0DF%02%00Xlv%3B? \
        --request PATCH \
        --header 'accept: application/json' \
        --header 'content-type: application/json' \
        --data @-
[2024-09-09T12:18:10Z ERROR wuppiefuzz::reproducer] Error sending the request: error sending request for url (http://localhost:8080/api/v1/accounts/%0DF%02%00Xlv%3B?)
 thomas@PC-40078   ~/GIT/wuppiefuzz_github     setup_cargo_dist   9  cargo run -- reproduce --openapi-spec ../go_wup/swag/openapi.json crashes/cc86faf759acfe23
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.14s
     Running `target/debug/wuppiefuzz reproduce --openapi-spec ../go_wup/swag/openapi.json crashes/cc86faf759acfe23`
Input file "crashes/cc86faf759acfe23" contains 4 inputs
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer]
    -----
    Sending request:
    PATCH /accounts/{id}
      id in Path: "\rF\u0002\u0000Xlv;"Contents in body: {"id": Bytes([73, 36, 238, 73, 36, 0, 9, 68]), "name": Bytes([111, 119, 0, 17, 119, 119, 3, 232]), "uuid": Bytes([35, 81, 40, 20, 10, 5, 2, 65])}
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Converted to CURL command:
    echo eyJpZCI6Ikkk77+9SSRcdTAwMDBcdEQiLCJuYW1lIjoib3dcdTAwMDBcdTAwMTF3d1x1MDAwM++/vSIsInV1aWQiOiIjUShcdTAwMTRcblx1MDAwNVx1MDAwMkEifQ== | \
    base64 --decode | \
    curl http://localhost:8080/api/v1/accounts/%0DF%02%00Xlv%3B? \
        --request PATCH \
        --header 'accept: application/json' \
        --header 'content-type: application/json' \
        --data @-
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Request successful (400 Bad Request)
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Response matches specification
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Response contents printed below:
    {"code":400,"message":"strconv.Atoi: parsing \"\\rF\\x02\\x00Xlv;\": invalid syntax"}
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer]
    -----
    Sending request:
    POST /examples/postContents in body: {"id": Bytes([100, 45, 45]), "name": Bytes([34, 32, 79, 82, 32, 49, 61, 49]), "uuid": Bytes([35, 81, 40, 20, 10, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 2, 65])}
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Converted to CURL command:
    echo eyJpZCI6ImQtLSIsIm5hbWUiOiJcIiBPUiAxPTEiLCJ1dWlkIjoiI1EoXHUwMDE0XG5cdTAwMDVcdTAwMDVcdTAwMDVcdTAwMDVcdTAwMDVcdTAwMDVcdTAwMDVcdTAwMDVcdTAwMDVcdTAwMDVcdTAwMDJBIn0= | \
    base64 --decode | \
    curl http://localhost:8080/api/v1/examples/post? \
        --request POST \
        --header 'accept: application/json' \
        --header 'content-type: application/json' \
        --data @-
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Request successful (404 Not Found)
[2024-09-09T12:18:20Z WARN  wuppiefuzz::reproducer] Validation error: Returned HTTP status 404 Not Found not allowed for this path
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Response contents printed below:
    404 page not found
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer]
    -----
    Sending request:
    DELETE /accounts/{id}
      id in Path: MAAAf/9VKlU=
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Converted to CURL command:
    curl http://localhost:8080/api/v1/accounts/0%00%00%7F%FFU%2AU? \
        --request DELETE \
        --header 'accept: application/json'
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Request successful (400 Bad Request)
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Response matches specification
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Response contents printed below:
    {"code":400,"message":"strconv.Atoi: parsing \"0\\x00\\x00\\x7f\\xffU*U\": invalid syntax"}
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer]
    -----
    Sending request:
    GET /examples/calc
      val1 in Query: b3d7PQZPZ3M=
      val2 in Query: CEQiUWh0Oh0=
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Converted to CURL command:
    curl http://localhost:8080/api/v1/examples/calc?val1=ow%257B%253D%2506Ogs&val2=%2508D%2522Qht%253A%251D \
        --request GET \
        --header 'accept: application/json'
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Request successful (400 Bad Request)
[2024-09-09T12:18:20Z WARN  wuppiefuzz::reproducer] Validation error: Response object incorrect: Expected type String(StringType { format: Empty, pattern: None, enumeration: [], min_length: None, max_length: None }) and actual response type Object {"code": Number(400), "message": String("strconv.Atoi: parsing \"ow%7B%3D%06Ogs\": invalid syntax")} do not match.
[2024-09-09T12:18:20Z INFO  wuppiefuzz::reproducer] Response contents printed below:
    {"code":400,"message":"strconv.Atoi: parsing \"ow%7B%3D%06Ogs\": invalid syntax"}

Note the validation error

[2024-09-09T12:18:20Z WARN  wuppiefuzz::reproducer] Validation error: Returned HTTP status 404 Not Found not allowed for this path

If you are only interested in labeling 5xx status codes as errors you can use the --crash-criterion only5xx flag.

By default any deviation from the spec is considered a 'crash'.

[ThomasTNO]

Also, awesome work on getting Go coverage to work. Let's see if we can boost its performance and then add it to WuppieFuzz!

[ThomasTNO]

Also, awesome work on getting Go coverage to work. Let's see if we can boost its performance and then add it to WuppieFuzz!

This can be achieved by replacing the exec.Commands with native Go code.

[koltiradw]

@grebnetiew, @ThomasTNO thanks for the explanations and such a cool project. At the moment I am working on transferring the receipt of lcov to native Go code.

[ThomasTNO]

@koltiradw I see you already made progress, awesome! I believe you released a speed-up of at least a factor 10 already!

Two thinks I noticed:

1. If the fuzzer stops, the application under test closes as well. It would be nice to have the coverage server accepting connections (one at a time) indefinitely. Probably wrapping some code with an infinite loop would work.
2. Performance difference between instrumented and non-instrumented is about a factor 20 still (10 seq/s vs 200 seq/s). Do you see any opportunity to get rid of the other exec.Command as well?

[koltiradw]

@ThomasTNO At the moment I'm trying to use internal parts of the standard library to get rid of the go tool utility.

[koltiradw]

@ThomasTNO I updated master. I boosted performance to 2k seq/s for test PUT.

[ThomasTNO]

@koltiradw That sounds awesome. What is the blackbox mode performance on your machine for comparison?

I created a repo to wrap up the work and eventually make it available to the entire community.

https://github.com/TNO-S3/WuppieFuzz-Golang

Do you want to make a Pull Request there in which we can tweak final stuff / review and write a short README?

[koltiradw]

@ThomasTNO I will get it soon.

[ThomasTNO]

I just tested your current implementation on the improved version of the fuzzer in #5.

It seems like that sometimes no coverage information is returned (0 coverage). You 'll see an error of the form

Error: Error in the fuzzing loop

Caused by:
    Invalid corpus: This testcase doesnot trigger trigger any edges. Check your instrumentation!

Most likely cause is that the coverage agent sends an empty coverage over the TCP stream to the fuzzer (as if nothing was covered).

[ThomasTNO]

I just tested your current implementation on the improved version of the fuzzer in #5.

It seems like that sometimes no coverage information is returned (0 coverage). You 'll see an error of the form

Error: Error in the fuzzing loop

Caused by:
    Invalid corpus: This testcase doesnot trigger trigger any edges. Check your instrumentation!

Most likely cause is that the coverage agent sends an empty coverage over the TCP stream to the fuzzer (as if nothing was covered).

This seems to be an issue on our end. Also fails for python-lcov.

[ThomasTNO]

This seems to be an issue on our end. Also fails for python-lcov.

This was a lie, I accidentally used a pre-release, outdated version of WuppieFuzz-Python. With the latest version it does work. Will post a reproducible setup soon.

In that case there might be something off in the go-coverage gathering.

[koltiradw]

This may happen. Now I am reworking the network part and the part related to coverage.

[koltiradw]

But a reproducible example would be very helpful.

[ThomasTNO]

But a reproducible example would be very helpful.

If you use the just released v1.1.0 you'll see the behaviour directly using the cellar example you provided earlier.

This behaviour is only there when empty coverage is received from the coverage agent. i.e. no lines covered.

[ThomasTNO]

@koltiradw, did you already manage to reproduce the behaviour? If you do not observe any breaks in the fuzzer (using WuppieFuzz v1.1.0) I need to check what goes wrong at my end.

[koltiradw]

Maybe I found the reason. Try the latest implementation.

[ThomasTNO]

No success here. Still seems to crash the fuzzer

[koltiradw]

I found the reason for this behavior. Zero coverage comes after POST /api/v1/examples/post, because this function is not implemented in the code:

[ThomasTNO]

I found the reason for this behavior. Zero coverage comes after POST /api/v1/examples/post, because this function is not implemented in the code:

Nice find, and interesting.. You'd expect at least some coverage in Go regardless, right? This of course depends on how much of the code is actually instrumented (including its dependencies). Are dependencies also instrumented / measured in your current setup?

[koltiradw]

Instrumenting dependencies solves this problem. I think it is necessary to instrument the most important modules from dependencies.