search

elastic / connectors

Source code for all Elastic connectors, developed by the Search team at Elastic, and home of our Python connector development framework
https://www.elastic.co/guide/en/enterprise-search/master/index.html
Other
68 stars 122 forks source link

Add support for omitting or redefining field names dependent on `map_column_names` #2310

Open chantzlarge opened 5 months ago

chantzlarge commented 5 months ago

Problem Description

The current implementation of the elastic/connectors service generates lengthy and complex field names by automatically prefixing column names with database schema and table names. This can lead to issues with query readability and field name management in Elasticsearch indices.

Proposed Solution

Introduce flexibility in the map_column_names function to allow users to customize or omit these prefixes. This could involve optional parameters for setting a custom prefix or entirely disabling auto-prefixing, enhancing usability and field name clarity.

Alternatives

One alternative is to provide a global configuration option for prefix management, simplifying the approach for users with uniform prefixing preferences. Another is allowing post-processing of column names via a user-defined callback for maximum customization.

Additional Context

Shorter, more intuitive field names can improve usability, especially for those unfamiliar with the database schema, and facilitate easier query construction and data analysis in Elasticsearch.

seanstory commented 5 months ago

Thanks for the issue, @chantzlarge ! This is something we've had on our radar, but hadn't gotten an issue filed for yet, so thanks for that.

While we work to get this prioritized, have you tried working around this by using Ingest Pipelines with a Rename Processor to adjust the field names before they are indexed?

chantzlarge commented 5 months ago

I have not @seanstory ! Thanks for the recommendation 👍