Move gear category into the manifest

[kofalt]

The gear category was historically remote from the manifest, because I didn't agree with the distinction: gears are f(x) = y filesystem transforms, and where outputs are stored is none of the gear's business. There's no reason a utility gear couldn't be ran as an "analysis", or vice-versa; you can today via the API.

As the gear concept continues to mutate, this separation has become less true over time. The two(-ish) categories have gelled pretty hard, gear authors have opinions about where the gears are displayed in various UIs, and the upcoming hierarchy inputs (#13) will well & truly kick the category-separation bucket.

I propose the following:

1. Drop converter and qa categories. I don't have a reason for these to exist.
2. Move the utility and analysis category options into the manifest.
3. Enforce the enum with the jSchema.

Result:

1. Gear categories are no longer an ops concern
2. Gear authors can record their choice and reference it later

[gsfr]

@kofalt 👍

@lmperry ?

[ryansanford]

@kofalt Would the processing of inputs/outputs be any different between utility and analysis? Are you suggesting that #13 be restricted to analysis gears? Has the idea of using an analysis bool instead been considered?

@lmperry How do QA gears fit into this proposal. They are the most tangible example to me of a gear that could be considered utility or analysis.

[kofalt]

Hierarchy inputs could obsolete the distinction, give or take. One of the things being discussed is that hierarchy.... "inputs" could arbitrarily modify the tree, which throws the concept of a destination out the window. To repeat, I'm really not a fan of the category at all because it raises needless questions like this, but the product seems to want it. If nothing else, it does separate out what could be a long list of gears into some semantically-useful lists.

To answer directly:

• A gear category does not change processing, I'm not sure what differences could exist there.
• No, I don't think hierarchy inputs should be restricted to a category. I would agree that a hierarchy input on a utility gear seems a little strange on the face of it.
• I hadn't considered a boolean, but I'd rather keep it a string. Forward-compatible. It would also be an instance of that one design pattern where you have func paint(bool) that I can't remember the name of offhand, but it's irritating :slightly_smiling_face:

[lmperry]

I agree with @kofalt's proposal.

QA, depending on the exact algorithm, could either be utility or analysis -- I have examples of both and the line is only slightly blurry.

At the end of the day, we have one question for the user: "Do you want the outputs of gear X to be an analysis object, or saved to the acquisition from whence the inputs came?" Right now analysis gears get analysis objects and utility gears' output go back on the acquisition container. Maybe we should be asking them and do away with the utility/analysis categories.

Issues with this distinction/not asking the user:

1. If I choose inputs from 3 different acquisitions as inputs to a utility gear (e.g. Flywheel Example Gear on Dev), where do the outputs go? - A: Back to the acquisition of the first file in the input array.
2. For a utility gear, if I choose an input from an existing analysis, where do the outputs go? - A: Right smack on the analysis object as an output file!!! I would argue this is a serious problem we should address very soon - I didn't know this until I tested it just now.

Either we should: 1. ask the user where the outputs should go (at least if it's ambiguous), or 2. Not allow analysis outputs to be inputs to utility gears - utility gears can only operate on acquisition data and write results out to those acquisitions, or new acquisitions? Need to think through that...

[gsfr]

Glad we're having this discussion. Looping in @travisr.

@lmperry A gear that has inputs from 3 different acquisitions should be considered a session-level gear, and so it's outputs should land on that session. Your second example is less clear to me.

[kofalt]

@lmperry

Right now analysis gears get analysis objects and utility gears' output go back on the acquisition container. Maybe we should be asking them and do away with the utility/analysis categories.

This was precisely my original vision for gears. Maybe we're headed back in that direction :slightly_smiling_face:

Both of your questions are reasonable, but neither have a ton to do with the gear spec... I get that we need to talk about category consequences in the product but let's be careful here. To answer:

1. Any container that held an input file. The API chooses one if you don't specify. I don't think the web UI offers the choise right now, but it could.
2. Analysis containers should definitely be immutable. Your change makes sense. I would vote that a new analysis is spawned and that's the destination of a utility gear that takes analysis files as input.

If we're okay with removing category as a concept, that works well. We can even leave the web UI alone, remove the key, and have it list both types of gears under each button. The "category" determines the destination.

On the downside, if a gear creates 30 files because of how complex it is, running that with a same-container destination might be frustrating.

[kofalt]

Offhand, this is also a lot of text to discuss moving a key from doc A to doc B. I'd be content to mark the key move as settled, and discuss the merits of reworking the category concept another time.

Objections?

[travisr]

I agree that the site (and ultimately the project) should be able to determine whether to treat a gear as a Utility Gear or an Analysis Gear. Subsequently, it would be preferred not to have the manifest have to know about the category concept. However, I can imagine the possibility of gears with outputs that don't naturally map back to the source acquisition(s). In those cases, it might be useful for the gear author to indicate compatibility as a Utility Gear, Analysis Gear, or both.

[kofalt]

Closing this for now. We can revisit gear categories after hierarchy inputs or other significant changes to how file destinations are chosen.