search

relipmoc / skewer

MIT License
95 stars 17 forks source link

gzip compressed output reduces speed by two-thirds #31

Open jblachly opened 8 years ago

jblachly commented 8 years ago

please disregard the following and see the followup comment - speed issues tracked down to output compression option

When using -m any I can max out 8-9 cores (Xeon X5670 @ 2.93 GHz).

However, using the exact same adapter file (2 adapters), in either -m head or -m tail mode my CPU usage pegs at < 300%. This means up to 8 times longer runtime.

In addition, I am going to file a probably related bug in that 5 prime adapters are not found and removed when run in -m any mode, while they are in the slower but apparently more effective -m head mode.

jblachly commented 8 years ago

I need to issue a correction to the above: it turns out the key difference was the -z parameter.

When doing gzip compression on the output, skewer's CPU usage maxes at around a third (3 full cores vs 9) of when not using -z