jg10545 / electricmayhem

MIT License
0 stars 0 forks source link

change pipeline content to be a dictionary #111

Open jg10545 opened 10 months ago

jg10545 commented 10 months ago

this way it's easy to capture information (like model output details) you may need later

example workflows to eventually enable

jg10545 commented 3 months ago

How should this actually work?

For now I think I'd like to maintain the current functionality of pipeline stages and Pipeline objects, where you can pass a tensor to forward() and get a result, e.g.

result = pipeline(mytensor)

But if you pass a dictionary...

result_dict = pipeline({"patch":mytensor})

...result_dict could contain other information. So each stage would need to:

jg10545 commented 3 months ago

functionality for specific pipeline stages

jg10545 commented 3 months ago

Alternate idea- have every pipeline stage forward() method accept a kwargs dictionary, and change

return x

to

return x, kwargs

So every stage could read from, add to, or manipulate whatever's in that dictionary.

jg10545 commented 2 months ago

Next question is whether x has to be a tensor or if it can be a dictionary or something. And how much I need to micromanage the different pipeline stages to handle this.

jg10545 commented 2 months ago

Stages that should be able to accept a dictionary of tensors for x and output a corresponding dict

Stages that should be able to input a dict for x and output a tensor

Other functions and objects that need to handle multiple patches

jg10545 commented 2 months ago

OK, the implant part is annoying. Thoughts on how to implement an API for implanting multiple patches

1. update every implanter step to behave differently if input is a dict of patches

2. define a MultiPatchImplanter class that wraps a dictionary of implanter objects, one per patch

during forward():

thoughts on this:

possible implementation:

3. just implement a multipatch capability for WarpPatchImplanter

4. make mutlipatch the default for all implanters

and add a couple lines of code to wrap bare input tensors in a dict