Open trahim opened 3 months ago
Hey @trahim, is this with the latest version of the redpanda docker container? The issue has been fixed since the first release of rpk connect
, but it's possible there are still some rough edges in build artifacts.
Hey @Jeffail apologies thought I but the versions in the above but yes with the latest version: v24.1.8
Hello, I'm having the same issue. It is working with the last version of ghcr.io/benthosdev/benthos but not the last version of ocker.redpanda.com/redpandadata/redpanda:latest
Just a quick update, this has been fixed in a newer version of rpk, we also have stand alone docker images just containing connect without rpk as an alternative. I've also updated the cli for the next release of connect so that all flags can be made after the subcommand. This will make it possible to run connect streams -r ./foo.yaml ./streams/*.yaml
and so on, which is much more intuitive and will be what we recommend in the future.
@Jeffail I tested latest redpanda-connect binary and the issue still exists:
$ redpanda-connect --version
Version: 4.31.0
Date: 2024-07-19T14:44:05Z
$ redpanda-connect streams -c config.yaml -r resources/*.yaml streams/*.yaml
Incorrect Usage: flag provided but not defined: -c
...
I also get the same on the latest
redpanda@1bbe6351af6d:/$ rpk connect --version
Version: 4.32.1
Date: 2024-07-26T22:53:24Z
redpanda@1bbe6351af6d:/$ rpk connect -c /benthos.yaml --resources '/resources/*.yaml' streams '/streams/*.yaml'
Incorrect Usage: flag provided but not defined: -c
There are some documentation updates in progress, but you should be able to use rpk connect streams -o /benthos.yaml --resources '/resources/*.yaml' '/streams/*.yaml'
. Please try that and report back if it doesn't work.
@mihaitodor unfortunately this does still not seem to work with 4.32.1:
Running as:
redpanda-connect streams -o ./config.yaml --resources './resources/one/cache.yaml' streams './streams/one/stream.yaml'
Exits with:
ERRO stream configuration file read error: failed to load config 'streams/one/stream.yaml': stream id (one_stream) collision from file: streams/one/stream.yaml @service=benthos
In the resources directory there is only one file with a memory cache resource referenced in the stream definition.
HTH
@brknstrngz Oops, my bad! 🙈 I forgot to remove the second streams
. Would you mind trying this?
> redpanda-connect streams -o ./config.yaml --resources './resources/one/cache.yaml' './streams/one/stream.yaml'
Thank you @mihaitodor, it works!
We have found that global options that were in the Benthos binary no longer work when calling a command.
For example this command we used to run.
This starts fine in streams mode. When running now with the new new redpanda image
We get
Same issues when running the streams command in help for streams above. We have found that using the global
-X
flag seems to work.Any idea if the plan is to use
-X
going forward or will the old Benthos global flags be usable again in the rpk binary?