CPU surge and Thread explosion on Xcode Build

[Noirdemort]

I noticed the amount of threads and CPU usage goes dramatically up while doing xcode builds. It slows the whole system down with it resulting in regressions. Should be able to replicate by running BlockBlock and then running xcode builds or run make for a decently big project in terminal - Terminal commands run super slow especially when doing make builds

[art-divin]

Hey @Noirdemort ,

I just came across the same issue, got the sample here:

Sample of BlockBlock.txt

I am trying to build & run BlockBlock app to pause and see where it hangs during Xcode build invocation. ⏳

[art-divin]

Update

I have investigated, and seems like the best candidate for the big consumption of the CPU is the line 85 in file named PluginBase.m.

[self.regexes enumerateObjectsUsingBlock:^(NSRegularExpression* regex, NSUInteger index, BOOL * _Nonnull stop) {

The following happens:

1. when Daemon is running, in its main function Monitor instance is created
2. then -[Monitor start] is called from the same main function of the Daemon
3. in -[Monitor start] there is the following code:

   FileCallbackBlock block = ^(File* file)
   {
   ...
           //process file event
           [self processEvent:file];
       }
   };
   ...
   //start monitoring
   started = [self.fileMon start:events count:sizeof(events)/sizeof(events[0]) csOption:csNone callback:block];

4. for every file registered by the Monitor (are all files in the system registered, or only some? I do not know exactly), callback is executed. It means that during compilation in Xcode, every file that is being created, written etc. produces these events. Xcode compilation happens in parallel on different cores, thus produces hundreds if not thousands of events.
5. Events arrive via the callback block to -[Monitor processEvent:]
6. Upon processing of the event, a "plugin" is being looked up in all registered "plugins":

   // ...that cares about the path/file that was just created
   plugin = [self findPlugin:file];

7. found "plugin" is an instance of PluginBase, the same I have mentioned in the beginning
8. message -[PluginBase isMatch:] is sent then, which uses regex to match against the given "event", where "event' is a file system event

As can be seen in the sample I have provided in the previous message, ICU library is responsible for such enormous CPU consumption.

Conclusion

Due to the nature of file processing, it is not that easy to optimize the process when there is such big bulk operation with file reads/writes occurs in the system.

That being said, there are a few techniques which might improve the situation not only for Xcode, but for all other applications/file operations as well:

1. use parallel enumeration when evaluating "plugin" against the given event
2. use parallel enumeration when searching for a "plugin" in all registered plugins
3. exit early in case the event should be "ignored"

cc Patrick @objective-see

[art-divin]

I am creating a PR which tries to fix this, but only Patrick will be able to really test this, I cannot invest much time into the XPC debugging setup at the moment, unfortunately 😢

[art-divin]

I have created #69 , which addresses the slowness. It introduces concurrent perform into the essential evaluations of rule lookup and event processing.

It might really speed up the event processing, but also might not even work.

cc @objective-see

[art-divin]

👋🏻 Hello,

A small update: the last version, 2.2.0 which was just released, on average consumes 50% less of CPU during compilation of Swift compiler. 🎉

Before	After

Test is not deterministic, but overall performance is much better

[objective-see]

Hooray! Moving to BTM events (on macOS 14+) allowed file monitoring/processing to drop drastically.

• coupled with improvements from @art-divin 🙏🏽

...other improvements coming soon(ish).