search

harlow / kinesis-consumer

Golang library for consuming Kinesis stream data
MIT License
264 stars 90 forks source link

PutRecords #112

Closed jney closed 4 years ago

jney commented 4 years ago

As I decided to update my producer from PutRecord to PutRecords it doesn't work anymore. I just saw this ticket #22 many years ago. Is it supported now? Thank you

harlow commented 4 years ago

Hi @jney sorry to hear you are having issues with the consumer. Are you using the standard PutRecords functionality from the AWS SDK? if so everything should work as expected.

The sample producer in this repo uses PutRecords and the consumer has historically worked correctly with the produced data:

https://github.com/harlow/kinesis-consumer/blob/master/cmd/producer/main.go#L88-L91

harlow commented 4 years ago

Please let me know if you are still having issues and I can try to help debug

jney commented 4 years ago

doing this:

    consumer.Scan(ctx, func(r *consumer.Record) error {
        fmt.Println(string(r.Data))
        return nil
    })

I have something like this:

"\xf3\x89\x9a\xc2\n.new-items:2c8bcce7-bab3-4b26-bd9a-6b9b1cbcc01a\x1a\x80\x06\b\x00\x1a\xfb\x05{\"created_at\":\"2020-05-13T18:09:40Z\",\"version\":\"2\"}\xd1\x13\xc0ѭ\xd5\xcc\xef\x9b\"\x8d\x9dُQ^"

Instead of having a string representation of my json as I used to have using PutRecord

harlow commented 4 years ago

@jney do you have a snippet of the producer loop you are using? I was able to replicate some issues when using scanner.Bytes() from STDIN and using PutRecords wondering if these could be related

harlow commented 4 years ago

I'm going to close this for now, but please let me know if this continues to be an issue. Feels like an issue w/ the Producer and not the consumer