search

seladb / PcapPlusPlus

PcapPlusPlus is a multiplatform C++ library for capturing, parsing and crafting of network packets. It is designed to be efficient, powerful and easy to use. It provides C++ wrappers for the most popular packet processing engines such as libpcap, Npcap, WinPcap, DPDK, AF_XDP and PF_RING.
https://pcapplusplus.github.io/
The Unlicense
2.63k stars 639 forks source link

Improve the CI time #1463

Open tigercosmos opened 1 week ago

tigercosmos commented 1 week ago

Currently, we have too many GitHub action items, which results in a long building and testing time. We need to refactor the current CI files and make the CI faster.

seladb commented 1 week ago

how can we improve it? Maybe we can list the action items so anyone can work on it

tigercosmos commented 1 week ago

@seladb let's brainstorm some ideas to improve this. cc @Dimi1010 @egecetin @clementperon

clementperon commented 1 week ago

I didn't look at the pipeline but using a cache + ccache could avoid to rebuild some files. Else limiting the number of OS and run a nightly pipeline with all the OS / configurations

egecetin commented 1 week ago

My suggestion is that if the cache is not enough, we can distribute the workflow to several providers to prevent reaching compute limits. For example,

It should make it harder to maintain a bit, but we can handle it if we have to

tigercosmos commented 6 days ago

@egecetin I think it could be the last step. The issue is the current CI script is not efficient enough. Just like that we need to improve the algorithm before trying to use multithread(?)

egecetin commented 6 days ago

@tigercosmos Yes of course. If possible, I agree we should improve the pipeline first. To be honest I did not check all workflows but most of the runs between 5-8 mins (approx 2-3 mins from build so it is like 30%-50% of run is build), right? So, I'm not sure we can dramatically improve CI runtime (even with caching) unless we limit the runner OS/version/config or improve parallelism. I think for "multiplatform" definition the code should be tested as much as possible. So, I don't know what we can do.

egecetin commented 6 days ago

Something that comes to my mind, testing consumes approx 1 minute. If I'm not mistaken current test suite does not support multithreading (@seladb correct me if I'm wrong pls). So maybe using a test library supporting multithreading might lead better testing times? (eg. integrating GoogleTest + ctest to run tests parallel to efficiently saturate CPU)

or maybe we can just remove "Check Installation" step which can halve the workflow time but not sure about this.

tigercosmos commented 6 days ago

@egecetin true, I think the parallelism of test cases may be the key. Probably we should focus on it in the next step.

tigercosmos commented 5 days ago

@seladb what do you think?