alan-turing-institute / pixelflow

BSD 3-Clause "New" or "Revised" License
5 stars 4 forks source link

Allow pixelflow to map pytorch (or other ML framework) models to image data #33

Open quantumjot opened 12 months ago

quantumjot commented 12 months ago

An example use case for pixelflow is to apply a DNN such as a convolutional classifier to a set of objects in the image.

It would be useful if we could extend pixelflow to do this. That is, given a mask/centroids/bounding boxes and an image, use pixelflow to map the CNN to each object in the image, returning the output of the model for each object.

One interesting caveat is that it may be good to collect the image patches so that they can be passed as batches to the DNN (for example, to utilise a GPU properly), before returning the PixelflowResult.

marjanfamili commented 12 months ago

Very interesting thought. I think this can be good for Affinity-VAE. Especially in projects similar to the Astrophysics one where you are searching for unknown object in image data that has very many objects in. One reason this collaboration has not taken off yet, is the complexity of data prep. I think having this automated feature would certainly help in some cases.

@evangeline-corcoran if there are multiple seed pods in one scan, do you think this would be helpful?

quantumjot commented 11 months ago

One interesting caveat is that it may be good to collect the image patches so that they can be passed as batches to the DNN (for example, to utilise a GPU properly), before returning the PixelflowResult.

is somewhat related to #14