AbsaOSS / cobrix

A COBOL parser and Mainframe/EBCDIC data source for Apache Spark
Apache License 2.0
138 stars 78 forks source link

Add support for `_` for key generation #689

Closed yruslan closed 4 months ago

yruslan commented 4 months ago

Background

When data has multiple levels of hierarchy and Cobrix is used to generate corresponding keys, sometimes the list of segments can be long for the leaf level of the hierarchy:

.option("segment_id_level0", "SEG001")
.option("segment_id_level1", "SEG002,SEG003")
.option("segment_id_level2", "SEG004,SEG005,SEG006,SEG007,SEG008,SEG009")

Feature

Add support for * and _ for key generation.

Proposed Solution

It could be nice to add _ meaning "the rest of segments.

Example

.option("segment_id_level0", "SEG001")
.option("segment_id_level1", "SEG002,SEG003")
.option("segment_id_level2", "_")