change pipeline content to be a dictionary

[jg10545]

this way it's easy to capture information (like model output details) you may need later

example workflows to eventually enable

• simultaneously training multiple patches
• incorporating patch position as part of loss function

[jg10545]

How should this actually work?

For now I think I'd like to maintain the current functionality of pipeline stages and Pipeline objects, where you can pass a tensor to forward() and get a result, e.g.

result = pipeline(mytensor)

But if you pass a dictionary...

result_dict = pipeline({"patch":mytensor})

...result_dict could contain other information. So each stage would need to:

• know which dictionary key(s) to operate on
• add additional information to the output dictionary in forward() if the input was a dict
• still be reproducible (so there's another step before I could use this new API for multi-patch training)

[jg10545]

functionality for specific pipeline stages

• I think most create stages won't really change. They'll look for an input key named "patch" or something and output a result with the same key.
• I imagine implant stages adding dictionary keys giving the coordinates the patch was implanted in. This could be useful, for example, for lining up object detections with the patch.
• Augmentation stages could record information on how each image in the batch was modified. One thing I haven't figured out is how these stages would need to play nicely with the implant stages; any geometric transformations on the patch coordinates would need to be propagated through here.
  • It would be inelegant but I could break out a separate kornia wrapper from KorniaAugmentationPipeline that manages affine transforms only, and applies the sampled transformation to any coordinates passed in the input dictionary.

[jg10545]

Alternate idea- have every pipeline stage forward() method accept a kwargs dictionary, and change

return x

to

return x, kwargs

So every stage could read from, add to, or manipulate whatever's in that dictionary.

[jg10545]

Next question is whether x has to be a tensor or if it can be a dictionary or something. And how much I need to micromanage the different pipeline stages to handle this.

• create stages: most of these would realistically apply the same way to every patch in the dict. can add an option to only apply to specific keys.
• implant stages: this is where it gets complicated; need to define the label data it saves to implant multiple patches in an image
• anything after implant: works as normal

[jg10545]

Stages that should be able to accept a dictionary of tensors for x and output a corresponding dict

• [x] PatchSaver
• [x] PatchResizer
• [x] PatchStacker
• [x] PatchTiler
• [x] PatchScroller
• [x] HighPassFilter
• [x] SoftProofer
• [x] SpectrumSimulationAttack

Stages that should be able to input a dict for x and output a tensor

• [ ] RectanglePatchImplanter
• [ ] FixedRatioRectanglePatchImplanter
• [ ] ScaleToBoxRectanglePatchImplanter
• [ ] WarpPatchImplanter

Other functions and objects that need to handle multiple patches

• [x] PatchWrapper
• [ ] Pipeline.initialize_patch_params()

[jg10545]

OK, the implant part is annoying. Thoughts on how to implement an API for implanting multiple patches

1. update every implanter step to behave differently if input is a dict of patches

• detect type of inputs to forward()
• recursively call forward() adding one patch at a time
• need to be able to pass batch of target images to forward() since it will sometimes be implanting in a target that already has a patch in it
• need to be able to separately capture offsets for different patches in lastsample dict
• need to be able to specify different box dictionaries for each input patch in constructors

2. define a MultiPatchImplanter class that wraps a dictionary of implanter objects, one per patch

during forward():

• iterate over patches
• send each partially-implanted image batch to the forward call of the next implanter

thoughts on this:

• we've got to update all the other implanter classes to be able to accept a batch of custom target images
• need to also input a key or index for each target in the batch so it knows which box coords to use
• does this mean we need to keep a copy of each image in memory separately for each separate implanter??

possible implementation:

• modify implanters so you could initialize without target images, but pass to forward instead

3. just implement a multipatch capability for WarpPatchImplanter

4. make mutlipatch the default for all implanters

and add a couple lines of code to wrap bare input tensors in a dict