High cpu & threads

[vikasprogrammer]

For a large filesystem the first update takes a lot of time and puts the whole server to 100% cpu for hours. Is there a way to optimize this, for e.g:

• Slow scan, takes longer but doesn't choke the CPU
• Limit threads
• I am not sure whether if I kill the process after , say 15 mins, it will resume the scan from the same point. If the update is resumable that also can help a lot!

[zeux]

Not sure how much data it is so that the process takes hours? But yeah it's possible to use fewer threads but you'd need to patch the source for that. I'd suggest trying something else first, in constants.hpp:

// File data compression level, 0-9
const int kFileDataCompressionLevel = 5;

Try reducing this to 0 and see if that helps. If it does I could add this as an official option.

It's possible to use fewer threads but that would mean the process takes much longer to complete so that's probably not desirable. The update process is not resumable/restartable.

[zeux]

P.S. If the above doesn't help, it would be helpful to get the profile of qgrep as it runs (eg using perf), so that I have a more precise idea of the source of performance problem.

[vikasprogrammer]

I don't have the tools to re-compile after the changes. I am going to use nice and cpulimit to limit the cpu.

Also, I am getting this

qgrep: src/build.cpp:583: void buildAppendFilePart(BuildContext*, const char*, unsigned int, const char*, size_t, uint64_t, uint64_t): Assertion `file.startLine < startLine' failed.
Aborted (core dumped)5243 Mb in, 1742 Mb out

[zeux]

The assertion has been fixed a few months ago, the next version (which should release this week) will fix that (see https://github.com/zeux/qgrep/issues/17). I'm going to double check if there's something meaningful that can be done for update performance, a few optimizations have been committed already but they don't address the bulk of the cost. Regardless using nice is a reasonable workaround.

[vikasprogrammer]

Thank you @zeux . Appreciate the response, will wait for your update.

[vikasprogrammer]

Please do a release for buildAppendFilePart we are facing a lot of issues.

[zeux]

A new release has been published. It accumulates several performance improvements to project update as well as the assertion fix. I don't think there's much else that can be done here, as there's not a lot of performance left on the table after the recent fixes, so I'm going to close this.

[vikasprogrammer]

Thanks. How can I sponsor this project?