Open adrianbr opened 1 year ago
As a novice dlt user, this is a feature I was expecting and how I thought it might be implemented.
I would strongly consider adding this for table names as well. Many services have unique identifiers for constructs which may have their actual name as renamable objects. Airtable, Google Sheets, etc.
I think there are other ideas for implementation. Any would be good.
@dapomeranz @adrianbr
we'll go for Python for now. renaming directly in yaml requires a full lineage data to be kept - in essence dlt
would need to generate unique id for every named entity in schema (tables and columns) based on the names in source (ie. API endpoints and filed names in json). if we are able to map each single data item into an entity in schema via separate automatic id, the names are free to be changed. until now they function as ids.
we have plenty of ticket requesting the behavior above. this will be quite a big implementation step
what I plan for now
do you think it makes sense to do (2) it in Python? any ideas for better interface?
if table name exists in dictionary, use the value from that dictionary instead of the table name
It is a simple implementation but should be effective for the problem at hand.
A user already expressed interest in this feature for renaming columns coming from apis as hashes.
This would be an easy way for a user to reconfigure field names already listed to them, instead of them having to figure out the names upfront in python.
This is an overlap with the previously discussed name mapping via the schema for long column names for databases with short names support such as Postgres.