Kinesis put record error

[rgsriram]

I am keep getting this error. streams/client.go:97 retrying 5 events on error: failed to put records: InvalidParameter: 1 validation error(s) found. minimum field size of 1, PutRecordsInput.Records

Below is my filebeat.yml file. I couldn't figure out what is the error.

filebeat.prospectors:
   input_type: log
    paths:
    /tmp/test_beat/*.log
       json.keys_under_root: true
           json.add_error_key: true
processors:
- add_cloud_metadata:

output:
  streams:
    enabled: true
    region: XX
    stream_name: XX
    shard_count: XX
    partition_key: XX
    access_key_id: XXX
    secret_access_key: XXX

queue.mem:
  events: 100
  flush.min_events: 5
  flush.timeout: 3s

logging.level: info
logging.to_syslog: false
logging.to_files: true
logging.files:
  path: /tmp/
  name: filebeat.log

`

[mumoshu]

@rgsriram Thanks for reporting!

I think this happens whenever the underlying beat is unable to produce valid events for your logs. So, my best bet is you need to fix your log format somehow.

To see what's causing the issue, you can enable debug logs so awsbeat dumps relevant events and records that helps further investigating the cause.

https://github.com/s12v/awsbeats/blob/5b58657b3e1b9b7e4d19fd863c7ed28def83ae67/streams/client.go#L90

[mumoshu]

You can add an command-line flag -d '*' to the beat to enable debug logging.

[rgsriram]

Thanks for the reply.

On Sun, Jan 6, 2019 at 11:46 AM KUOKA Yusuke notifications@github.com wrote:

You can add an command-line flag -d '*' to the beat to enable debug logging.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/s12v/awsbeats/issues/50#issuecomment-451718554, or mute the thread https://github.com/notifications/unsubscribe-auth/AGvZR7r_tTrSMe4hY8CunVDRrXUA6P0mks5vAZTIgaJpZM4YSK8n .

-- Sriram G Tech

[superwhykz]

Shouldn't it be agnostic on message format? Filebeat does not care what it is shipping if it is ascii files.

[superwhykz]

my debug log looks like this

2019-03-05T22:47:45.241Z    DEBUG   [kinesis]   streams/client.go:87    received events: [{{2019-03-05 22:47:40.928463896 +0000 UTC m=+0.034389025 null {"beat":{"hostname":"ip-172-31-37-3","name":"ip-172-31-37-3","version":"6.6.1"},"host":{"name":"ip-172-31-37-3"},"log":{"file":{"path":"/home/ubuntu/beats.log"}},"message":"{'name':'helllo world'}","meta":{"cloud":{"availability_zone":"us-east-2c","instance_id":"i-06b8474a7736caa81","machine_type":"t2.micro","provider":"ec2","region":"us-east-2"}},"offset":0,"source":"/home/ubuntu/beats.log"} { false 0xc00018cdd0 /home/ubuntu/beats.log 24 2019-03-05 22:47:40.927222619 +0000 UTC m=+0.033147766 -1ns log map[] 256333-51713}} 1}]
2019-03-05T22:47:45.241Z    DEBUG   [kinesis]   streams/client.go:114   failed to map event({{2019-03-05 22:47:40.928463896 +0000 UTC m=+0.034389025 null {"beat":{"hostname":"ip-172-31-37-3","name":"ip-172-31-37-3","version":"6.6.1"},"host":{"name":"ip-172-31-37-3"},"log":{"file":{"path":"/home/ubuntu/beats.log"}},"message":"{'name':'helllo world'}","meta":{"cloud":{"availability_zone":"us-east-2c","instance_id":"i-06b8474a7736caa81","machine_type":"t2.micro","provider":"ec2","region":"us-east-2"}},"offset":0,"source":"/home/ubuntu/beats.log"} { false 0xc00018cdd0 /home/ubuntu/beats.log 24 2019-03-05 22:47:40.927222619 +0000 UTC m=+0.033147766 -1ns log map[] 256333-51713}} 1}): failed to get parititon key: failed to get parition key: key not found
2019-03-05T22:47:45.241Z    DEBUG   [kinesis]   streams/client.go:90    sent 0 records: []
2019-03-05T22:47:45.241Z    DEBUG   [kinesis]   streams/client.go:94    mapped to records: []
2019-03-05T22:47:45.241Z    INFO    streams/client.go:101   retrying 1 events on error: failed to put records: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, PutRecordsInput.Records.

[abdalians]

I seem to be getting the same error.. Aug 11 16:48:10 master metricbeat-v7.5.0-go1.11-linux-amd64: 2020-08-11T16:48:10.635Z#011INFO#011streams/client.go:104#011retrying 16 events on error: failed to put records: InvalidParameter: 1 validation error(s) found. Aug 11 16:48:10 master metricbeat-v7.5.0-go1.11-linux-amd64: - minimum field size of 1, PutRecordsInput.Records. Aug 11 16:48:10 master metricbeat-v7.5.0-go1.11-linux-amd64: 2020-08-11T16:48:10.635Z#011INFO#011[publisher]#011pipeline/retry.go:162#011Drop batch

[abdalians]

I solved this with a clue from the metricbeat.yml file in the project repo..

go get github.com/rs/xid then run make and add the partition_key_provider: xid to metricbeat.yml in output.streams