Support Transforming data en-flight

[docktermj]

Is your feature request related to a problem? Please describe. Support "transforming" in stream-producer.py This would make a true "ETL".

• E: - JSON, CSV, Avro, Parquet
• T: - Call a senzing.Transformer class
• L: - RabbitMQ, Kafka, (SQS, Kinesis, etc.)

This would be similar to techniques used at:

• https://github.com/Senzing/stream-loader/blob/dc5528ddd7b192b565b9aa25aecd479198d3c8b4/stream-loader.py#L1261-L1285
• https://github.com/Senzing/stream-loader/blob/dc5528ddd7b192b565b9aa25aecd479198d3c8b4/stream-loader.py#L3510-L3524

Where "plugins" would be used to perform the Transforming. The plugin would receive a "record" as a python dictionary and return a list of 0 or more python dictionaries

class Transformer:

    def __init__(self, g2_engine=None, *args, **kwargs):
        self.g2_engine = g2_engine

    def transform(self, message=None, hint=None, *args, **kwargs):
        assert type(message) == dict
        messages = [dict1, dict2]        
        return messages

[jbutcher21]

Sounds good. I'm ready to code these classes. I assume I will create ... DNBTransformer.py IJICTransformer.py DowJonesTransformer.py etc?

[roncewind]

Ref.: https://senzing.zendesk.com/hc/en-us/articles/360054097153-Advanced-mapping-concepts