RADAR-base / radar-output-restructure

Reads avro files in HDFS and outputs json or csv per topic per user in local file system
Apache License 2.0
1 stars 0 forks source link

Add per-topic formatting with record data lookup #528

Closed blootsvoets closed 1 year ago

blootsvoets commented 1 year ago

Allow changing the output path per topic based on simple data lookups. This is particularly useful if we add a questionnaire_response topic, which would get multiple questionnaire types. Then the output folder can still be changed based on the name of the questionnaire even though all those questionnaires were originally put in the same topic.

FormattedPathFactory now has an plugin structure, where multiple variable parsers can be inserted via configuration. This can be used in the future for example for adding ManagementPortal properties to the file structure (e.g. externalId, project group or set of related projects).