Drop/Ignore a column from slice via metadata.

Rucal-Data-Solutions / datalakefoundation

Datalakehouse Foundation

GNU General Public License v3.0

3 stars 0 forks source link

Drop/Ignore a column from slice via metadata. #18

Closed Gateway1024 closed 4 months ago

Gateway1024 commented 8 months ago

"this is a reenactment (shortened) of a real discussion I had. I would like to gather more reasons why this would be a good idea, please leave a comment."

I would like to be able to drop a column during processing.

I have a column in a slice file that I would like to be able to drop, this column contains blob data and should not be ingested in silver.

Gateway1024 commented 8 months ago

It seems to me that the source query needs to exclude that column. this way it doesn't even get delivered and doesn't need to be dropped.