pilerimport -z introduce Load Average based delay

[dilyanpalauzov]

On the system where pilerimport was running I observed steadily increasing load average. So I thought pilerimport was feeding too fast data to searchd (manticore). I developed this change to pause pilerimport, when the load average increases, pilerimport -z 2 -d <directory> imported as long files, as the load average was below 2, otherwise it waited. At the end I fixed the cause of the high load average in a completely different way. But I think this change is still useful, better than the delay introduces by -Z <ms>.

• This changeset always holds pilerimport, whenever the load average on the system is above a certain value. Once the load average goes below the threshold, pilerimport continues.

[jsuto]

I like this patch, however, I'm not sure if manticore couldn't handle a high rate of inserts.

At the end I fixed the cause of the high load average in a completely different way

Can you tell me more about it?

[dilyanpalauzov]

At the end I fixed the cause of the high load average in a completely different way

As mentioned at https://github.com/jsuto/piler/issues/123, the high load average was caused by inability for one of the .service files to create a pid file, in turn systemd terminated the process, but not its children (or alike) and then there were many searchd processes running, resulting very high load average. Another reason I mentioned there was when piler.conf did not contain the right way to connect to mysql (missing mysqlhost=) and this again resulted very high load average.

I'm not sure if manticore couldn't handle a high rate of inserts.

It possibly could. The -z option makes the system usable to do other things, like serving the web interface, unpacking files, etc. Only when nothing else requires resources, then pilerimport -z 2 make progress.

[jsuto]

OK, thank you.