Open mohitmarwal opened 7 months ago
I'm facing a similar problem when using the elasticsearch
output, which I cannot handle the connection error and the pipeline gets stuck in a loop. The code below never reaches the stdout
output:
input:
generate:
mapping: 'root = {"hello": "world"}'
interval: 1s
count: 1
pipeline:
processors:
- log:
level: INFO
message: "processing event: ${!content()}"
output:
broker:
pattern: fan_out_sequential_fail_fast
outputs:
- elasticsearch:
urls:
- https://localhost:1234
index: "my-index"
id: ${!timestamp_unix()}
max_retries: 1
tls:
enabled: true
skip_cert_verify: true
basic_auth:
enabled: true
username: elastic
password: admin
healthcheck: false
sniff: false
processors:
- mapping: '{"message": "elasticsearch preprocessor", "timestamp": timestamp_unix()}'
- stdout:
codec: lines
processors:
- mapping: '{"message": "stdout preprocessor", "timestamp": timestamp_unix()}'
As the problem seems related to the BatchOutput
interface, other components, such as aws_s3
, behave in the same way.
@mohitmarwal when you say "previously", do you mean version 4.26.0? And is there a an example you can provide where this behaviour shows up? For example, the following config behaves as I'd expect:
input:
http_server: {}
output:
http_client:
url: example.com/not/going/to/work
Running curl http://localhost:4195/post -d "hello world"
returns an error as expected.
@FerroEduardo this is a separate issue, unfortunately for some components when they are failing during a connection loop they will block traffic even when a DLQ is configured. There's an existing issue for this: https://github.com/benthosdev/benthos/issues/1210
@Jeffail Issue occured when i updated from version V 4.24.0 and 4.25.1.
@Jeffail l here is the suedo code where the issue occurs type myBatchOutput struct { count int // Just one simple field }
func (m *myBatchOutput) WriteBatch(ctx context.Context, batch service.MessageBatch) error { // Implement your WriteBatch method here for _, msg := range batch { // Process each message fmt.Println("Processing message:", msg) } return nil }
func (m *myBatchOutput) Close(ctx context.Context) error { fmt.Println("disconnected:") return nil }
func (m *myBatchOutput) Connect(ctx context.Context) error {
// Implement your Connect method here
return nil
}
func main() { // Initialize your BatchOutput implementation and use it as needed err := service.RegisterBatchOutput("my_batch_output", service.NewConfigSpec(), newMyBatchOutput) if err != nil { panic(err) } service.RunCLI(context.Background()) }
func newMyBatchOutput(conf service.ParsedConfig, mgr service.Resources) ( output service.BatchOutput, batchPolicy service.BatchPolicy, maxInFlight int, err error, ) {
output = &myBatchOutput{
count: 10, // Set a default value or configure from conf
}
// Assign a default max in flight
maxInFlight = 10
fmt.Println("newbatchoutputfunc:")
return
}
input: type: http_server address: ":80" path: "/test" http_server: path: "/test" allowed_verbs:
pipeline: processors: []
output: type: plugin plugin: name: your_custom_plugin_name
output: type: stdout
@Jeffail in my setup even this doesnt return the hello world but gives me request time out error
input: http_server: { path: /test } output: http_client: url: https://httpbin.org/hidden-basic-auth/:user/:passwd
MINGW64 ~/Downloads/benthos_4.27.0_windows_amd64.tar/benthos_4.27.0_windows_amd64 $ curl -X POST http://localhost:4195/test -d "hello world" Request timed out
@Jeffail any updates on this please let me know if you want more
@Jeffail any updates on this , or are you planning to fix this in near future ?
@mohitmarwal I'm really, really struggling to reproduce the issue you're seeing here. Can you please share a self-contained step-by-step example (with proper code blocks please) of the issue and which version works and which doesn't? Please make sure you share the full code of the custom plugin so I can just copy paste it, run go build and then run it myself locally. Don't use services like https://httpbin.org (or if you do, please make sure we have access to them).
hi @mihaitodor we can use services which are opensource https://httpbin.org/ as we cannot expose organization api in public. this perfectly works and accessible everywhere
follow these steps to reproduce 1) create a config.yaml file add these content
input: http_server: { path: /test } output: http_client: url: https://httpbin.org/hidden-basic-auth/:user/:passwd
2) now run it in benthos using ./benthos -c config.yaml
3) open any power-shell and run the command curl -X POST http://localhost:4195/test -d "hello world"
4) you will get output Request timed out instead of expected https://httpbin.org/hidden-basic-auth/:user/:passwd: HTTP request returned unexpected response code (405): 405 Method Not Allowed, Error: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
The method is not allowed for the requested URL.
Please try this and let me know
Hey @mohitmarwal, sorry for the long delay. I just tried running the following on my Macbook:
config_http_response.yaml
input:
http_server:
path: /test
output:
http_client:
url: https://httpbin.org/hidden-basic-auth/:user/:passwd
$ ./redpanda-connect --version
Version: 4.37.0
Date: 2024-09-26T20:10:07Z
$ ./redpanda-connect run ./config_http_response.yaml
INFO Running main config from specified file @service=redpanda-connect benthos_version=4.37.0 path=./tmp/config_http_response.yaml
INFO Listening for HTTP requests at: http://0.0.0.0:4195 @service=redpanda-connect
INFO Launching a Redpanda Connect instance, use CTRL+C to close @service=redpanda-connect
INFO Output type http_client is now active @service=redpanda-connect label="" path=root.output
ERRO Failed to send message to http_client: https://httpbin.org/hidden-basic-auth/:user/:passwd: HTTP request returned unexpected response code (405): 405 Method Not Allowed, Error: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"><title>405 Method Not Allowed</title><h1>Method Not Allowed</h1><p>The method is not allowed for the requested URL.</p> @service=redpanda-connect label="" path=root.output
$ curl -X POST http://localhost:4195/test -d "hello world"
https://httpbin.org/hidden-basic-auth/:user/:passwd: HTTP request returned unexpected response code (405): 405 Method Not Allowed, Error: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"><title>405 Method Not Allowed</title><h1>Method Not Allowed</h1><p>The method is not allowed for the requested URL.</p>
I also tried the above with v4.23.0
and I see exactly the same output.
I guess you're seeing something different, but I'm still not sure what is incorrect here and in which version it used to work differently.
Discussed in https://github.com/benthosdev/benthos/discussions/2548