LosD
commented
9 years ago This seems to be a bit of an incomplete idea, and it seems the hard part is actually what goes where. I'm not sure what you want in the component part? The more I think about it, the more questions I see:
- How do we avoid endless chains? We can do it programmatically, but it would be really nice if the schema explicitly disallowed it (unfortunately, I'm not sure that is possible without a hugely complex schema... Or at all).
- If the definition is in the
part, and the transformation is just a reference (as <transfomer ref="foo" />seems to suggest), how would you define what columns of which stream should be consumed by the transformer? If you do it on the transformer, the definition of the streams inputs and outputs is not contained within the stream, which would be rather confusing. - What should the stream source contain? How does it reference an output?
My suggestion would be a simpler approach: Make a new fusers element that contains components which are allowed to reference output data streams, but otherwise behaves like a normal component. It will require source ids to be unique across the job, but I think that problem is already there, at least I remember an issue that was solved by adding some form of counting IDs to columns.
In theory, we could allow all components to reference all outputstreams (schema-wise), but I like that it's limited to the special fuse cases, as it makes them more visible.
An example (extension of example-job-output-dataset.analysis.xml). Sorry about the size, but it needs to be reasonably complete to make sense. The new element is in the end of the file:
<?xml version="1.0" encoding="UTF-8"?>
<job xmlns="http://eobjects.org/analyzerbeans/job/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<source>
<data-context ref="my database" />
<columns>
<column id="col_fn" path="PUBLIC.EMPLOYEES.FIRSTNAME" />
<column id="col_ln" path="PUBLIC.EMPLOYEES.LASTNAME" />
<column id="col_supervisor0" path="PUBLIC.EMPLOYEES.REPORTSTO" />
</columns>
</source>
<transformation>
<transformer>
<descriptor ref="Concatenator" />
<properties>
<property name="Separator" value=" " />
</properties>
<input ref="col_fn" />
<input ref="col_ln" />
<output id="col_fullname0" />
</transformer>
</transformation>
<analysis>
<analyzer requires="_any_">
<descriptor ref="Completeness analyzer" />
<properties>
<property name="Conditions" value="[NOT_BLANK_OR_NULL,NOT_BLANK_OR_NULL,NOT_BLANK_OR_NULL]" />
<property name="Evaluation mode" value="ALL_FIELDS"/>
</properties>
<input ref="col_fullname0" />
<input ref="col_supervisor0" />
<output-data-stream name="Complete rows">
<job>
<source>
<columns>
<column id="col_supervisor1" path="REPORTSTO"/>
<column id="col_fullname1" path="Concat of FIRSTNAME,LASTNAME"/>
</columns>
</source>
<analysis>
<analyzer>
<descriptor ref="String analyzer"/>
<input ref="col_fullname1" />
</analyzer>
<analyzer>
<descriptor ref="Number analyzer"/>
<input ref="col_supervisor1" />
</analyzer>
</analysis>
</job>
</output-data-stream>
<output-data-stream name="Incomplete rows">
<job>
<source>
<columns>
<column id="col_supervisor2" path="REPORTSTO"/>
<column id="col_fullname2" path="Concat of FIRSTNAME,LASTNAME"/>
</columns>
</source>
<analysis>
<analyzer>
<descriptor ref="String analyzer"/>
<input ref="col_fullname2" />
</analyzer>
<analyzer>
<descriptor ref="Number analyzer"/>
<input ref="col_supervisor2" />
</analyzer>
</analysis>
</job>
</output-data-stream>
</analyzer>
</analysis>
<fusers>
<transformer>
<descriptor ref="Coalesce multiple fields"/>
<properties>
<property value="true" name="Consider empty string as null"/>
<property value="[[REPORTSTO,REPORTSTO],[Concat of FIRSTNAME%2CLASTNAME,Concat of FIRSTNAME%2CLASTNAME]]" name="Units"/>
</properties>
<input ref="col_supervisor1"/>
<input ref="col_supervisor2"/>
<input ref="col_fullname1"/>
<input ref="col_fullname2"/>
<output id="col_reportsto" name="__reportsto_or_reportsto"/>
<output id="col_concat" name="__concat_of_firstname_lastname_or_concat_of_firstname_lastname"/>
<output-data-stream name="Fused rows">
<job>
<source>
<columns>
<column id="col_reportsto" path="__reportsto_or_reportsto"/>
<column id="col_concat" path="__concat_of_firstname_lastname_or_concat_of_firstname_lastname"/>
</columns>
</source>
<analysis>
<analyzer>
<descriptor ref="String analyzer"/>
<input ref="col_concat" />
</analyzer>
<analyzer>
<descriptor ref="Number analyzer"/>
<input ref="col_reportsto" />
</analyzer>
</analysis>
</job>
</output-data-stream>
</transformer>
</fusers>
</job>Another alternative could be a more graphML-like approach with edges (connections between input and output columns) and nodes/ports (components/columns), but that would be much bigger change from now... Although I think that I like this approach best, for it's clarity and extensibility. However, it might be even harder to control.
kaspersorensen
With the prospect of adding a "multi stream" component in #620 we encounter the problem that we get "diamond shaped" component graphs and our current .analysis.xml format only supports "hierarchy shaped" graphs.
Today every output-data-stream is represented as a job which is a child to a component. And a component is a child to a job. Rather we would need to allow some components to exist in multiple streams. And for the future I can very well we should also need to represent multiple streams at the outer-most level (for instance two datastores - or one rainy day to properly model two tables from the same datastore as separate streams in the same job).
My suggestion (off the top of my head) is to preserve compatibility of the existing format but keep it only as a shorthand version of a format like this:
Quite a big change ... Maybe start with first having the lower part "components" tag to allow shared components. Maybe the streams wrapping we can make a separate story.