Larger write-up is being put aside from now, but some notes from a large discussion today around transformation semantics:
do transforms always run?
how do transforms and body's provided at the same time interact?
our current model treats the input dataset to a transform as the user provided body, is this best?
when is it ok to drop data, such as bodies provided to transformation scripts that get dropped without being recorded everywhere. for things like reduce we probably want this
General outcome:
yes, transforms always run
we should consider things like the "--body" flag to be a "user-transformation", and running save with both "--transform" and "--body" is trying to run two transformations at once
instead, the body should always override the transformations output, with a warning or error about changing both
transform might need another concept, like "user-input.json", that take the role that body passed into transform currently fills
Larger write-up is being put aside from now, but some notes from a large discussion today around transformation semantics:
reducewe probably want thisGeneral outcome: