Open chantzlarge opened 5 months ago
Thanks for the issue, @chantzlarge ! This is something we've had on our radar, but hadn't gotten an issue filed for yet, so thanks for that.
While we work to get this prioritized, have you tried working around this by using Ingest Pipelines with a Rename Processor to adjust the field names before they are indexed?
I have not @seanstory ! Thanks for the recommendation 👍
Problem Description
The current implementation of the elastic/connectors service generates lengthy and complex field names by automatically prefixing column names with database schema and table names. This can lead to issues with query readability and field name management in Elasticsearch indices.
Proposed Solution
Introduce flexibility in the
map_column_names
function to allow users to customize or omit these prefixes. This could involve optional parameters for setting a custom prefix or entirely disabling auto-prefixing, enhancing usability and field name clarity.Alternatives
One alternative is to provide a global configuration option for prefix management, simplifying the approach for users with uniform prefixing preferences. Another is allowing post-processing of column names via a user-defined callback for maximum customization.
Additional Context
Shorter, more intuitive field names can improve usability, especially for those unfamiliar with the database schema, and facilitate easier query construction and data analysis in Elasticsearch.