Feature Request - clean up part preservation?

[dustymc]

Refs https://github.com/ArctosDB/arctos/issues/3654, https://github.com/ArctosDB/arctos/issues/4366

Is your feature request related to a problem? Please describe.

There are three potential problems with the path we've gone down.

1. It's not "clean"; we have "formalin" and various forms of "formalin and some other stuff." I think users probably expect that here and it's not (various combinations of preservation) x (various combinations of parts) denormalized - anything we do here is still a huge improvement over the old model. I'm completely willing to accept the idea that "entry usability" is more important than "user usability," but that should be documented.
2. There are commas in the data, and commas are a natural delimiter for lots of languages, See eg.,https://github.com/ArctosDB/arctos/issues/3645
3. The commas serve (at least) two purposes; our format isn't consistent. ethanol, 70% is a modifier, formalin-fixed, stored in 70% ethanol is a separator.

(And we have two ways of saying "70% ethanol" in the two examples above.)

Describe what you're trying to accomplish

Not paint ourselves into any (familiar!) corners.

Describe the solution you'd like

Develop a plan and documentation.

Minimally, any concerns a CT operator should be looking for should be spelled out in the documentation. ("Just accept whatever, see linked issue." would even be useful.)

A more cautious plan might involve:

• format (we should only have one way to saying "70% ethanol," even when it's embedded in a combo-preservation).
• dedicated separator character (a user encountering a semicolon/pipe/whatever should know they're moving on to a second thing)
• dedicated "modifier follows" character (a user encountering a semicolon/pipe/whatever should know the previous bit is being clarified)

(and would require some cleanup).

Describe alternatives you've considered

Don't.

[ewommack]

Good point. Getting the huge strings of values in the larger aggregators for specimens can be really hard to handle. It would be really nice if we could figure out something that works across the board. When we tried separating preservation we seemed to run into the problem of creating a difficult number of steps for objects that have multiple types of preservation. So I wonder if the problem isn't the desire for clarity, but how we make that clarity easy to implement?

[dustymc]

aggregators

One option is https://github.com/ArctosDB/arctos/issues/2141 - just don't try to squish complex data down into simple strings. (Or, that looks like a "them" problem, not an "us" problem...)

clarity....easy to implement

Yes, agreed there's a balance. Being consistent in formatting and avoiding things that various languages expect as "action characters" in the data isn't an exhaustive solution, but it doesn't seem like any kind of a burden for us (using "A; B" is the same effort as using "A, B"), and it should avoid the most obvious of problems (eg, value "A, B" being hard to distinguish from list "A ,B").

It also seems useful to have one substring with which "everything involving 70% ethanol" can be discovered, and the current data do not support that.