near / near-one-project-tracking

A repository for tracking work items that NEAR One is working on.
0 stars 0 forks source link

[ProjectTracking] Stabilize the fuzzing infrastructure #56

Open Ekleog-NEAR opened 8 months ago

Ekleog-NEAR commented 8 months ago

Goals

Background

While developing the p2p handshake fuzzer, I noticed by pure chance that clusterfuzz had stopped running the new fuzzers, and seems to be currently running only the release- and finite-wasm- jobs.

Why should NEAR One work on this

We spend significant effort developing fuzzers, in order to guarantee security in our protocol. However, when the fuzzer infrastructure stops running the fuzzers, we lose all the guarantees for all the new code we write, which is the most interesting one to actually fuzz.

What needs to be accomplished

We need to setup proper alerting on common fuzzer failure modes, so that we are at least informed of the issue timely, and don’t have to rely on my randomly wandering through the clusterfuzz pages in search of something else to discover the issue by pure chance.

We also need to figure out why the fuzzers from master currently do not run on clusterfuzz.

Main use case

Security is the main use case.

Links to external documentations and discussions

None

Estimated effort

The SRE team is expected to work on this effort, with support from the security team. Estimated effort is 1 month.

Assumptions

None

Pre-requisites

None

Out of scope

None

Task list

### Tasks
- [ ] Setup alerting whenever the number of fuzzers goes down
- [ ] Setup alerting whenever the number of runs/hr of one specific fuzzer goes significantly down
- [ ] Setup alerting whenever the total number of fuzzer runs/hr goes significantly down
- [ ] Setup alerting when fuzzer builds fail (both ondemand and master builds)
- [ ] Figure out why the fuzzers from master and rc currently seem to not run on clusterfuzz
- [ ] Ideally, figure out why login is so wobbly, and often requires doing black magic tricks to manage to login, if this can fit in the allocated time frame