search

filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.84k stars 1.26k forks source link

Makefile target to run itests #11780

Open rvagg opened 6 months ago

rvagg commented 6 months ago

Checklist

Lotus component

What is the motivation behind this feature request? Is your feature request related to a problem? Please describe.

Be able to execute the same paths CI runs but locally if you have the resources to do so.

Describe the solution you'd like

We now have make unittests to run everything but itests, we should have a make itests to do the itests. I'm not sure if a make test is a good idea though, we don't really want to give the impression this is a good idea for any contributor to run (maybe it could print documentation though).

Currently itests are each run separately, and they are configured independently in the CI config. It would not be ideal to have yet another place to have to remember to add your new itest to make it run, so some deduplication is needed. Either the Makefile target reads CI config (new GitHub Actions one) to figure out what and how to run, or we make CI config and Makefile depend on some new configuration, perhaps a text file in the itests directory, or perhaps we decorate itests in source and extract metadata from there.

Describe alternatives you've considered

No response

Additional context

No response

BigLep commented 3 months ago

As an example of the impact of this problem, ecosystem partners are forced to to open a fake PR to get itests to invoke. https://github.com/filecoin-project/lotus/pull/11987 is an example. https://filecoinproject.slack.com/archives/CP50PPW2X/p1714698675147429?thread_ts=1711282222.982289&cid=CP50PPW2X is a #fil-lotus-dev thread on this.

ribasushi commented 1 month ago

An update to this issue. With https://github.com/filecoin-project/lotus/pull/12407/files in place a plain test now succeeds! Specifically this just works™️ 

~lotus-checkout$ time -v go test -count=1 -timeout=1h ./...
... < snip a bunch of lines > ...
ok      github.com/filecoin-project/lotus/cmd/lotus-miner       57.810s
ok      github.com/filecoin-project/lotus/cmd/lotus-sim/simulation/stages       0.021s
ok      github.com/filecoin-project/lotus/cmd/tvx       0.016s        
ok      github.com/filecoin-project/lotus/conformance   86.246s       
ok      github.com/filecoin-project/lotus/conformance/chaos     0.005s
ok      github.com/filecoin-project/lotus/gateway       0.156s
ok      github.com/filecoin-project/lotus/itests        3016.220s
ok      github.com/filecoin-project/lotus/journal       0.003s        
ok      github.com/filecoin-project/lotus/journal/alerting      0.003s  
ok      github.com/filecoin-project/lotus/lib/backupds  0.120s                
ok      github.com/filecoin-project/lotus/lib/filler    0.003s                
ok      github.com/filecoin-project/lotus/lib/rpcenc    0.979s        
ok      github.com/filecoin-project/lotus/lib/shardedmutex      0.018s
ok      github.com/filecoin-project/lotus/lib/sigs/bls  0.014s                
ok      github.com/filecoin-project/lotus/lib/sqlite    0.057s                
ok      github.com/filecoin-project/lotus/lib/stati     0.003s        
ok      github.com/filecoin-project/lotus/lib/strle     0.009s                
ok      github.com/filecoin-project/lotus/lib/tablewriter       0.009s
ok      github.com/filecoin-project/lotus/lib/ulimit    0.009s
ok      github.com/filecoin-project/lotus/node  0.031s
ok      github.com/filecoin-project/lotus/node/config   0.175s
ok      github.com/filecoin-project/lotus/node/impl/full        1.232s
ok      github.com/filecoin-project/lotus/node/modules/tracer   0.013s
ok      github.com/filecoin-project/lotus/node/repo     0.022s
ok      github.com/filecoin-project/lotus/paychmgr      0.053s
ok      github.com/filecoin-project/lotus/storage/paths 0.232s
ok      github.com/filecoin-project/lotus/storage/pipeline      5.020s
ok      github.com/filecoin-project/lotus/storage/sealer        117.473s
ok      github.com/filecoin-project/lotus/storage/sealer/commitment     0.009s
ok      github.com/filecoin-project/lotus/storage/sealer/ffiwrapper     6.465s
ok      github.com/filecoin-project/lotus/storage/sealer/fr32   1.525s
ok      github.com/filecoin-project/lotus/storage/sealer/mock   0.523s
ok      github.com/filecoin-project/lotus/storage/sealer/proofpaths     0.014s
ok      github.com/filecoin-project/lotus/storage/sealer/storiface      0.032s
ok      github.com/filecoin-project/lotus/storage/wdpost        0.054s
ok      github.com/filecoin-project/lotus/tools/stats/headbuffer        0.020s
        Command being timed: "go test -count=1 -timeout=1h ./..."
        User time (seconds): 7262.52
        System time (seconds): 305.78
        Percent of CPU this job got: 243%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 51:44.72
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 15242136
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 418946
        Minor (reclaiming a frame) page faults: 64746459
        Voluntary context switches: 31578368
        Involuntary context switches: 4344579
        Swaps: 0
        File system inputs: 17924080
        File system outputs: 52764440
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

The need for the timeout comes from itest being unable to run in parallel: go runs each namespace sequentially. During a run the CPU utilization often looks like below:

image

I suspect that simply separating the individual test files into a ./itest/subtestdir scheme would be sufficient to make things work much faster.