Improve performance for loading many classes

[chisandrei]

The following code snippet loads 100000 classes and takes in Pharo 10 a long time (around 1 or 2 hours).

1 to: 100000 do: [:i | 
Object subclass: (#Foo , i printString) asSymbol
    instanceVariableNames: ''
    classVariableNames: ''
    package: 'Data' ].

We can get it to work in basically a few minutes if we do a bulk loading and update the structure of subclasses and using SystemOrganizer only at the end of the load (https://github.com/feenkcom/gtoolkit-utility/blob/main/src/GToolkit-Utility-ClassLoader/GtBulkShiftClassInstaller.class.st).

GtBulkShiftClassInstaller startBulkInstaller.
1 to: 100000 do: [:i | 
Object subclass: (#Foo , i printString) asSymbol
    instanceVariableNames: ''
    classVariableNames: ''
    package: 'Data' ].
GtBulkShiftClassInstaller finishBulkInstaller

[Ducasse]

Ok this is fun. We will look at it.

[MarcusDenker]

In Pharo12, SystemOrganizer is now using a dictionary, that should already speed things up a bit

[jecisc]

I tried to run:

[
1 to: 100000 do: [ :i |
    Object classInstaller make: [ :builder |
        builder
            superclass: Object;
            name: (#Foo , i printString) asSymbol;
            package: 'Data' ] ] ] timeToRun

in the image produced by this PR: https://github.com/pharo-project/pharo/pull/15239

And the result is less than 11min.

Also 90% of the time is spent in GC because we need to make the memory grow in Behavior>>#handleFailingBasicNew: Maybe more memory could be allocated in that case?

[jecisc]

I got it down to 3min with this script:

Smalltalk growMemoryByAtLeast: 300 * 1024 * 1024.

[
1 to: 100000 do: [ :i |
    Object classInstaller make: [ :builder |
        builder
            superclass: Object;
            name: (#Foo , i printString) asSymbol;
            package: 'Data' ] ] ] timeToRun

[jecisc]

Would that be enough for you @chisandrei if the PR gets integrated?

[chisandrei]

Definitely a lot better than before!

Curios what the results would be with these VM parameters:

Smalltalk vm parameterAt: 45 put: 159744960. "desired eden size"
Smalltalk vm parameterAt: 25 put: 67108864. "growth headroom"
Smalltalk vm parameterAt: 24 put: 134217728. "shrink threshold"
Smalltalk vm parameterAt: 55 put: 1.0d0. "full GC ratio:"

Maybe it would be still interesting to consider adding some support for bulk loading of classes/methods (But no idea if it makes sense in the current implementation, or if if brings a significant speed-up)

[jecisc]

I think that bulk loading would not improve anything currently except if we do more than just bulk loading.

We would need to create specific announcements for bulk operations and then we could gain in performance if we update the listeners so that they deal with those bulked operations in a more optimized way.

But I think that performance wise we have better to do. For example, we often creates classes when we load code, and one of the source of slowness in loading code comes from Metacello that is recreating all the time objects instead of keeping the ones already created. Or for example, I've shown right before that the GC management can impact a lot the perfs.

I've tried with you GC parameters and the result is 3min and 11seconds! Which is way better than the 1 or 2h :)