search

prashanthpai / sqscat

A netcat/kafkacat like utility for AWS SQS.
MIT License
5 stars 0 forks source link

Not continuously reading from stdin #1

Open jackmead515 opened 2 years ago

jackmead515 commented 2 years ago

Maybe I'm using it wrong. And forgive, I don't know much go. Trying to pair this up with https://github.com/cloudamqp/amqpcat and consume from a rabbitmq queue and pipe to an sqs queue like so:

amqpcat --consumer --uri=amqp://@127.0.0.1:5672 --queue=my-queue | sqscat test-queue

But it keeps failing for some reason?

2021/12/07 11:14:59 run() failed: bufio.Scanner: Read returned impossible count
Error writing file: Broken pipe

Can this tool not constantly read from stdin? Or does it only spin up a certain number of threads and then die? Any help would be appreciated thanks.

ppai-plivo commented 2 years ago

Can this tool not constantly read from stdin?

As you can see in examples, sqscat certainly reads constantly from stdin.

Or does it only spin up a certain number of threads and then die?

By default will continue to read indefinitely until it encounters an error or until it reaches EOF. Specifying -n will read exact specified number of entries before exiting.

Can you pipe the results of your first command to a file and confirm that there's newline as delimiter between items/entries? And also confirm the maximum size of each line.

jackmead515 commented 2 years ago

Absolutely confirmed that there is a new line terminator. amqpcat automatically does this by default.

I have also tried gstbuf -oL amqpcat ... to force a new line buffering activity yet without luck.

But you say that it will read until EOF and then die? The -n will just die after a certain number of lines? Or will it just push the lines to sqs once that count has been reached and then wait for the next batch from stdin?