[sink elasticsearch] mapper_parsing_exception, as object, but found a concrete value, internal_log_rate_limit=true

[StefanSa]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I am testing vector after a major update (0.25.2 to latest) and see these error messages:

023-09-12T10:53:25.601898Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}:request{request_id=3}: vector::sinks::util::retries: Not retriable; dropping the request. reason="error type: mapper_parsing_exception, reason: object mapping for [destination.geo] tried to parse field [geo] as object, but found a concrete value" internal_log_rate_limit=true

2023-09-12T11:53:13.690820Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector::sinks::util::retries: Internal log [Not retriable; dropping the request.] is being suppressed to avoid flooding.
2023-09-12T11:53:13.690855Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.
2023-09-12T11:53:13.690986Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
2023-09-12T11:53:22.894776Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector::sinks::util::retries: Internal log [Not retriable; dropping the request.] has been suppressed 9 times.
2023-09-12T11:53:22.894800Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector::sinks::util::retries: Not retriable; dropping the request. reason="error type: mapper_parsing_exception, reason: object mapping for [source.as] tried to parse field [as] as object, but found a concrete value" internal_log_rate_limit=true
2023-09-12T11:53:22.894828Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] has been suppressed 9 times.
2023-09-12T11:53:22.894838Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=23 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2023-09-12T11:53:22.894862Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 9 times.
2023-09-12T11:53:22.894872Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=406 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2023-09-12T11:53:23.894770Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector::sinks::util::retries: Internal log [Not retriable; dropping the request.] is being suppressed to avoid flooding.
2023-09-12T11:53:23.894808Z ERROR sink{component_kind="sink" component_id=out_sophos-xg component_type=elasticsearch component_name=out_sophos-xg}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.

as a sink is used opensearch v2.9 i did not see these problems with the previous vector version 0.25.2 The biggest changes in the migration were related to "geoip".

Any idea ? Thanks for any help

Configuration

[enrichment_tables.geoip_city]
type = "geoip"
path =  "/etc/vector/GeoIP/GeoLite2-City.mmdb"

[enrichment_tables.geoip_asn]
type = "geoip"
path =  "/etc/vector/GeoIP/GeoLite2-ASN.mmdb"

[transforms.geo_mapping_src_fw]
type = "remap"
inputs = ["route_ip_fw.gipsrc" ]
source = '''
.source.geo = get_enrichment_table_record("geoip_city", { "ip" : .source.ip }) ?? .source.ip
'''

[transforms.geo_mapping_dst_fw]
type = "remap"
inputs = ["route_ip_fw.gipdst" ]
source = '''
.destination.geo = get_enrichment_table_record("geoip_city", { "ip" : .destination.ip }) ?? .destination.ip
'''

### Version

vector 0.32.1 (x86_64-unknown-linux-gnu 9965884 2023-08-21 14:52:38.330227446)

### Debug Output

_No response_

### Example Data

_No response_

### Additional Context

_No response_

### References

_No response_

[jszwedko]

I think the discussion you'd opened is a better place for this so I'll close this issue and we can continue over there.