cytomining / CytoTable

Transform CellProfiler and DeepProfiler data for processing image-based profiling readouts with Pycytominer and other Cytomining tools.
https://cytomining.github.io/CytoTable/
BSD 3-Clause "New" or "Revised" License
5 stars 5 forks source link

Remove duplicative all-at-once concatenate with join operations #168

Closed d33bs closed 4 months ago

d33bs commented 4 months ago

Description

This PR removes a duplicative and all-at-once concatenate during CytoTable join operations. It was discovered while trying to replicate / confirm findings for #38 with SQ00014613.sqlite (which appears to no longer face direct challenges with data value translation but faces large memory resource consumption issues as a result of the removed block in this PR). After this is applied we should see performance benefits (as join concatenation will only take place once) and better opportunity to avoid memory resource constraint issues.

References #38 Likely effects discussion on recommendations in #163

What is the nature of your change?

Checklist

Please ensure that all boxes are checked before indicating that a pull request is ready for review.

d33bs commented 4 months ago

Thanks @kenibrewer ! Merging this in.