cytomining / CytoTable

Transform CellProfiler and DeepProfiler data for processing image-based profiling readouts with Pycytominer and other Cytomining tools.
https://cytomining.github.io/CytoTable/
BSD 3-Clause "New" or "Revised" License
7 stars 5 forks source link

Use real AWS S3 data tests and apply related fixes #212

Closed d33bs closed 3 months ago

d33bs commented 3 months ago

Description

This changes in this PR address #198 by removing moto and related tests to avoid existing and future challenges with mocked s3 resources (in addition to the failing tests, a short justification can be found here). Moving forward, CytoTable will now be tested on a CSV and SQLite resource from the cellpainting-gallery (as outlined in the fixtures). As a result, I believe we now are addressing #193 within reason because of the SQLite addition (please don't hesitate to let me know if you feel otherwise and we should keep that issue open).

In the process of developing towards this fix I added a new preset which enables compatibility with JUMP data (cpg0016-jump) from the cellpainting-gallery in order to effectively perform a test from an S3 SQlite object (no other presets appeared to exactly match this need). CC @jenna-tomkinson

Some notes:

Closes #198 Closes #193

What is the nature of your change?

Checklist

Please ensure that all boxes are checked before indicating that a pull request is ready for review.

d33bs commented 3 months ago

Thank you @gwaybio ! After making some changes I feel good about how this now looks. I plan to create a new issue to explore memory resource reductions for sorted joins.