search

astrorama / SourceXtractorPlusPlus

SourceXtractor++, the next generation SExtractor
https://astrorama.github.io/SourceXtractorPlusPlus/
GNU Lesser General Public License v3.0
72 stars 9 forks source link

Lazy stamps #390

Closed ayllon closed 2 years ago

ayllon commented 3 years ago

Helps #361.

They seem to reduce the memory footprint quite considerably for some use cases, without impacting the performance. This is for problem #380.

lazystamps380

For other cases, it does reduce a little, and (surprisingly) it is faster. Likely I just got lucky with the node lesta assigned me (though they are all supposed to be the same and I do not think shared, since I use every physical cores: 16). This is the benchmark I use for multi-threading (from the challenge data):

lazystamps_challenge

At the very least, it doesn't seem to slow down things.

Anyhow, maybe wait for merging. I will try with now limiting the queue size and bumping the priority of measurements. It may help more, it may even be better if combined with this (?)

ayllon commented 3 years ago

I have re-run develop over the sample from the challenge data, and as I expected, it is not slower.

lazystamps_challenge

But this branch isn't substantially slower either.

Still, to be left on hold. The preliminary from the thread pool with priorities and with a limited queue look very promising.

ayllon commented 2 years ago

Obsoleted