cytomining / CytoTable

Transform CellProfiler and DeepProfiler data for processing image-based profiling readouts with Pycytominer and other Cytomining tools.
https://cytomining.github.io/CytoTable/
BSD 3-Clause "New" or "Revised" License
7 stars 5 forks source link

Document system and workflow architecture options #17

Open d33bs opened 1 year ago

d33bs commented 1 year ago

Document system and workflow architecture options in order to improve developer understanding of remote and/or scalable options. For example, see the following reference sketch for a potential future state (from discussions in #10):

flowchart LR
subgraph actor["Actor / Initiator"]
pycytominer-transform
end

subgraph Storage
direction TB
subgraph sourcestorage["Source"]
s3source[S3 or Cloud Source]
end
subgraph deststorage["Destination"]
s3dest[S3 or Cloud Destination]
end
end
subgraph worker["Worker"]
dask[Dask or Ray Cluster]
end
pycytominer-transform --> |initiates work| dask
dask --> |creates data| s3dest
s3source --> |gathers data| dask
gwaybio commented 1 year ago

I think this would be very helpful in an advanced document describing the work. I don't think this needs to happen prior to the first pip release. Something like this probably also belongs in a figure in a paper.

It might also be important to note that, IIRC, the current diagram above is no longer accurate