search

voxpupuli / puppet-augeasproviders_sysctl

Augeas-based sysctl type and provider for Puppet
Apache License 2.0
10 stars 36 forks source link

x86_64 server with 1792 cpu cores - sysctl -a parsing takes >30min #51

Closed marcelfischer closed 3 years ago

marcelfischer commented 3 years ago

Hi, we have a huge x86_64 machine which has 1792 logical cpus. This means we have 122903 sysctl entries and 120853 of them are cpu related. It looks like the parsing of "sysctl -a" takes more than 30 minutes to complete. Currently I have no idea on how to fix that, expect for tuning the parsing somehow. I will clarify if there is any option to hide specific parameters, but so far I havn't found something.

Best Marcel

trevor-vaughan commented 3 years ago

My guess is that this is due to the prefetch portion creating an object for every single entry.

I think that it makes sense to either not prefetch or to selectively create only the object that is under scrutiny.

Alternatively, perhaps an exclude/include list mechanism could be passed to filter out some items and may even default to excluding CPU items since they aren't usually modified.

marcelfischer commented 3 years ago

Hi Trevor, yes probably prefetch is the problem. Sorry Im not so familiar with providers, is there a way to disable prefetching for this provider on node level? An exclude mechanism would be also awesome, somehow changeable on node level via parameters

trevor-vaughan commented 3 years ago

There's not, unfortunately. That particular subsystem is pretty darn complex so it will take some fiddling to figure out.

marcelfischer commented 3 years ago

And just having a exclude via "sysctl -a | grep -v something..."?

raphink commented 3 years ago

Prefetching is global to the provider so it usually ignores resources parameters. There's therefore no (standard) way to pass parameters to it.

genebean commented 3 years ago

I know it doesn't solve the problem you are having with this module, but how about simple approaching the problem from a different angle and using something like thias/sysctl instead?

trevor-vaughan commented 3 years ago

@marcelfischer Can you give the linked PR a shot (#52)?

ghost commented 3 years ago

@marcelfischer Can you give the linked PR a shot (#52)?

Hi, I am a workmate of @marcelfischer . This PR (4681998) works! Thank you and best regards

trevor-vaughan commented 3 years ago

@rutilus-rutilus Awesome, thanks for verifying!

@genebean This is probably a pattern that should be suggested across all prefetch calls.