-
I'm working with broadway to process SQS messages containing references to files on s3 and for each SQS message I start up a flow processing pipeline (they're big files, over 1 gb each).
After a fe…
-
Hi Everyone,
It will be very useful to have **from** ref in handle_demand
to indicate who is the consumer that is requesting for demand.
Imagine this scenario:
-step 1) consumer(s) asking f…
-
I have a need to dynamically increase or decrease the processor and/or batcher stage count depending on throughput.
Or is there a suggested best practice? A monitor process that restarts the pip…
-
The following worker (c&p from the docs) fails to connect to the queue.
```elixir
defmodule MyBroadway.Worker do
use Broadway
def start_link(_) do
Broadway.start_link(__MODULE__,
…
-
Here's a test that reproduces it:
```elixir
defmodule EmptyProducer do
use GenStage
@behaviour Broadway.Producer
@impl true
def init(_args) do
{:producer, []}
end
@impl …
-
We could keep start_time and store the number of events received, the number of event message, the amount of demand received and the amount of demand requests. Those should be relatively cheap to stor…
-
I have a custom producer that consumes from an api so my producer achieve a point where there is no events to produce anymore and when all events are successfully acknowledged my broadway tree can go …
-
Currently the Broadway producer has a buffer limit of 10k messages per producer which comes from the defaults set in GenStage.
Lets update this to default to `:infinity` so faktory worker's default…
-
Library version 0.3.0
erlang 21.3.6
elixir 1.8.1
```
** (CaseClauseError) no case clause matching: {:error, :unknown_host}
(broadway_rabbitmq) lib/broadway_rabbitmq/producer.ex:278: Broadwa…
-
Hello @josevalim,
This is my current GenStage based data ingestion flow:
`[Uploader] -> [MetadataExtractor] -> [Persister]`
An `Uploader` is a process hooked up to a Phoenix route, which tak…