Open hhromic opened 3 years ago
This is a good suggestion, thanks @hhromic . To date, we've been a bit hit and miss with how much detail when errors occur.
👍 also dovetails with our roadmapped dead letter queue work, today we could do this though:
sources:
generator:
type: generator
format: syslog
interval: 1
transforms:
example:
type: remap
inputs:
- generator
drop_on_error: true
source: |-
data, err = parse_json(.message)
if err != nil {
log(.message, level: "error")
abort
}
sinks:
console:
type: console
inputs:
- example
encoding:
codec: text
@spencergilbert we are aware of catching errors like you propose, however you probably can imagine how much of boiler plate that pattern starts to become when the VRL program is larger and most function can fail.
I do admit (shamefully) that we could have done what you suggest, i.e. capturing the error and then logging the event, once we knew which function was failing :)
In any case, I hope you consider adding any of the solutions we proposed here. We operate Vector inside a Docker Swarm cluster and quickly enabling more verbose logging is far more efficient/simpler than temporarily modifying the Vector configuration files with such debugging code because we store them in Swarm Config resources that needs to be rotated for changes.
@hhromic oh definitely - just wanted to make sure you had a possible workaround, I agree with adding it to the logging itself :)
Hi @spencergilbert, @jszwedko , we just hit another example where it would be very useful for Vector to log the offending message alongside the error. In this case the error occurs in a source stage so no error handling in VRL can help.
Consider this simple config
sources:
syslog:
type: syslog
address: 0.0.0.0:8888
mode: udp
sinks:
console:
type: console
inputs:
- syslog
encoding: json
And the following syslog message:
bad="<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - the secret is $(head -n1 /dev/urandom)"
Ignore any warning that the shell might produce when assigning the above variable. The test case is having binary data in the middle of the syslog packet.
Then, when you send the above message to Vector listening for syslog data:
echo "$bad" | socat - udp4-send:localhost:8888
Vector will log the following error:
Jul 05 10:53:13.226 ERROR source{component_kind="source" component_name=syslog component_type=syslog}: vector::internal_events::syslog: Error converting bytes to UTF8 string in UDP mode. error=Utf8Error { valid_up_to: 79, error_len: Some(1) } internal_log_rate_secs=10
While the error message very accurately describes what is the problem, there is no indication of the offending message itself, making debugging of the above very difficult. Specially on scenarios like ours where thousands of messages are sent per second and these bad messages are sparse.
Having sort of an input
field in the error log containing the input data that caused the error would be really handy. In the above case I imagine it could be something like input=b'...'
so it can also properly display binary data.
Another example:
https://discord.com/channels/742820443487993987/753400475545501836/861990635958173767
Another warning we've seen with Vector on the end to end development looks like this: Jul 06 15:05:03.516 WARN transform{component_kind="transform" component_name=remap_to_json component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"object\" at (7:37): function call error for \"parse_json\" at (15:36): unable to parse json: expected value at line 1 column 1" internal_log_rate_secs=30 11 Jul 06 15:05:03.516 WARN transform{component_kind="transform" component_name=remap_to_json component_type=remap}: vector::internal_events::remap: Internal log [Mapping failed with event.] is being rate limited.
We don't have too much insight on what's happening here.
While debugging #8459, I recalled this issue. It would have been very easy to debug if Vector could show the exact payload that is causing the error. Note that in that case, again is about a source component and not a remapping stage.
I am facing similar issue with Elasticsearch sink. If there's some mapping issue, resulting log will look as follows:
2021-12-17T15:36:26.721315Z ERROR sink{component_kind="sink" component_id=out_kafka_audit_es component_type=elasticsearch component_name=out_kafka_audit_es}:request{request_id=14282}: vector::sinks::elasticsearch::service: Response contained errors. response=Response { status: 200, version: HTTP/1.1, headers: {"x-elastic-product": "Elasticsearch", "content-type": "application/json; charset=UTF-8", "content-length": "39436"}, body: b"{\"took\":34,\"errors\":true,\"items\":[{\"create\":{\"_index\":\".ds-fluentd.kube.kube-apiserver.audit-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"1voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":53170,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.kube.kube-apiserver.audit-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"1_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":53237,\"_primary_term\":8,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"2PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597953,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"2foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599139,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"2voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597954,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"2_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597955,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"3PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597038,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"3foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597956,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"3voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599140,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"3_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597957,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"4PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597039,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"4foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597958,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"4voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599141,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"4_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599142,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"5PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599143,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"5foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599144,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"5voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597959,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"5_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597960,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"6PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597961,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"6foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597040,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"6voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599145,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"6_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597962,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"7PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599146,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"7foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597963,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"7voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597041,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"7_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597964,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"8PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599147,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"8foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597042,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"8voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597965,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"8_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597043,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"9PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597966,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"9foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597967,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"9voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597968,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"9_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597044,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"-PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599148,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"-foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597045,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"-voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599149,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"-_oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597969,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"_PoJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597970,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"_foJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599150,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"_voJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597046,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"__oJyX0BS88umjbJj_I9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597047,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"APoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599151,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"AfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597971,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"AvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597972,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"A_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597973,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"BPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597974,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"BfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597048,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"BvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599152,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"B_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597975,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"CPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599153,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"CfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597049,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"CvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597976,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"C_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599154,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"DPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597977,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"DfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597978,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"DvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597979,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"D_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597050,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"EPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599155,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"EfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597980,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"EvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599156,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"E_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597051,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"FPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599157,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"FfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597981,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"FvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597052,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"F_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599158,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"GPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597982,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"GfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597983,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"GvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597053,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"G_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599159,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"HPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599160,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"HfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597054,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"HvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597984,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"H_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599161,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"IPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597985,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"IfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597055,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"IvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599162,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"I_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597056,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"JPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597057,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"JfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597058,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"JvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597059,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"J_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597986,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"KPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597060,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"KfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597987,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"KvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597988,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"K_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597061,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"LPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597989,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"LfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597990,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"LvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597062,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"L_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597991,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"MPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597992,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"MfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597063,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"MvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597993,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"M_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599163,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"NPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597064,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"NfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597994,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"NvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597995,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"N_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597996,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"OPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599164,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"OfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599165,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"OvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599166,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"O_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597065,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"PPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597997,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"PfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597066,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"PvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597067,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"P_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597998,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"QPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597999,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"QfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598000,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"QvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599167,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"Q_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599168,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"RPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597068,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"RfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598001,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"RvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598002,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"R_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598003,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"SPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599169,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"SfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598004,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"SvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597069,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"S_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599170,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"TPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599171,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"TfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599172,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"TvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599173,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"T_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599174,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"UPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599175,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"UfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597070,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"UvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598005,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"U_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597071,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"VPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597072,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"VfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598006,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"VvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599176,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"V_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597073,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"WPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597074,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"WfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599177,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"WvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598007,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"W_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597075,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"XPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597076,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"XfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599178,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"XvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599179,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"X_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":597077,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"YPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598008,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"YfoJyX0BS88umjbJj_M9\",\"status\":400,\"error\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [1000] has been exceeded while adding new fields [1]\"}}}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"YvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599180,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"Y_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598009,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"ZPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598010,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"ZfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599181,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"ZvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598011,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"Z_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598012,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"aPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598013,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"afoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599182,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"avoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599183,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"a_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599184,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"bPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":599185,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"bfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598014,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"bvoJyX0BS88umjbJj_M9\",\"status\":400,\"error\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [1000] has been exceeded while adding new fields [1]\"}}}},{\"create\":{\"_index\":\".ds-cloudtrail-2021.12.10-000001\",\"_type\":\"_doc\",\"_id\":\"b_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":598015,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"cPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":94042,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"cfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":94043,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"cvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":93662,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"c_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":93663,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"dPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":93664,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"dfoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":93735,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"dvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":94044,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"d_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":93665,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"ePoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":94045,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.useractivity-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"efoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":94046,\"_primary_term\":9,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.kube.event-exporter-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"evoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":49617,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.kube.event-exporter-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"e_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":49270,\"_primary_term\":8,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.kube.event-exporter-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"fPoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":49618,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.kube.event-exporter-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"ffoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":49619,\"_primary_term\":7,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.kube.event-exporter-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"fvoJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":49271,\"_primary_term\":8,\"status\":201}},{\"create\":{\"_index\":\".ds-fluentd.svcfw.publicaudit-2021.12.13-000026\",\"_type\":\"_doc\",\"_id\":\"f_oJyX0BS88umjbJj_M9\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":9366,\"_primary_term\":8,\"status\":201}}]}" }
2021-12-17T10:52:16.836002Z ERROR sink{component_kind="sink" component_id=out_kafka_audit_es component_type=elasticsearch component_name=out_kafka_audit_es}:request{request_id=12877}: vector::sinks::util::retries: Not retriable; dropping the request. reason="error type: mapper_parsing_exception, reason: failed to parse"
One needs to look for error field to find the reason, eg.:
{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [1000] has been exceeded while adding new fields [1]\"}}}}
Current Vector Version
Use-cases
We use Vector to process large volumes of observability data such as syslog and audit logging. Most of the times we do not control the formatting of the source data, therefore we normally monitor our Vector pipelines constantly in case the source data format breaks/changes.
Vector already provides very nice metrics and logging facilities, however we recently found it very hard to debug sporadic broken messages we were receiving in one fast-pacing feed. We normally configure our
remap
stages withdrop_on_error = true
to ensure malformed events get logged and do not advance further in the pipeline.However, when messages get dropped, only the VRL code that failed to event processing is logged and not the event itself that caused the error. This makes it very difficult to understand why the event is being dropped, i.e. if parsing or pattern matching did not work, what was the string trying to be parsed/matched.
Attempted Solutions
In a recent wave of dropped event warnings, we re-started Vector using
-vv
to enable more verbose logging (TRACE level). While VRL functions themselves still do not emit the argument values that make them fail, at least for the case of the syslog source we are able to see the event being sent to theremap
transform and then manually trace from there.Of course, the above does not scale well, and we have a very hard time trying to find the exact events that trigger errors in a stream of fast-moving messages and a sea of other TRACE-level logging events.
Also, it does not seem to be guaranteed that every source emits the event being sent. For example, the
generator
source does not TRACE the events being generated.Proposal
We think that there are two possible approaches to improve on this:
(1) Make functions in VRL to log their parameters in DEBUG-level when failing -> more effort (2) Make vector log the event being processed in DEBUG-level when any transform fails -> easier
Maybe solution (2) is more general and consistent even.
Example Configurations
Using the
generator
source is impossible to see the event that generates an error, no matter the logging level:Output:
Here is an example using the
syslog
source, where the only way to see the event is by using-vv
:Send some UDP syslog data to Vector:
Output: