search

apache / camel-kafka-connector

Camel Kafka Connector allows you to use all Camel components as Kafka Connect connectors
https://camel.apache.org
Apache License 2.0
151 stars 99 forks source link

Couchbase fails due to incorrect URI format #629

Open orpiske opened 3 years ago

orpiske commented 3 years ago

When trying Couchbase with the given configuration, the connector fails complaining that the URL format is invalid:

2020-10-22 14:08:06,300 [SinkConnector-0] ERROR org.apache.camel.processor.errorhandler.DefaultErrorHandler - Failed delivery for (MessageId: E1DBFDB5E174861-0000000000000000 on ExchangeId: E1DBFDB5E174861-0000000000000000). Exhausted after delivery attempt: 1 caught: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: couchbase://http:my-redacted-host:8091?bucket=testBucket85&password=xxxxxx due to: Invalid URI. Format must be of the form couchbase:http[s]://hostname[:port]/bucket?[options...]

The configuration used for this test was:

connector.class=org.apache.camel.kafkaconnector.couchbase.CamelCouchbaseSinkConnector
camel.sink.path.port=32816
topics=org.apache.camel.kafkaconnector.couchbase.sink.CamelSinkCouchbaseITCase
tasks.max=1
camel.sink.endpoint.bucket=testBucket
camel.sink.path.protocol=http
camel.sink.endpoint.password=password
camel.sink.path.hostname=localhost
camel.sink.endpoint.username=Administrator
name=CamelCouchbaseSinkConnector
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
oscerd commented 3 years ago

I worked on this here https://github.com/apache/camel/commit/46e23d1b7fa324f3a5ad2edf8d53e061262012ff#diff-118ec4ad74df487260d52308cd40005080748272e546997dda6d23d91890ecbe

orpiske commented 3 years ago

@oscerd that was quick! Thanks Andrea.

oscerd commented 3 years ago

I'll move camel-master to 3.7 soon

orpiske commented 3 years ago

@oscerd I gave it a try with Couchbase and 3.7. The URL formatting via camel.sink.url is OK now.

However, when using only properties, our code is still formatting the URL incorrectly.

2020-11-16 10:07:10,748 [SinkConnector-0] ERROR org.apache.camel.processor.errorhandler.DefaultErrorHandler - Failed delivery for (MessageId: 2073D9C11D9CFB0-0000000000000000 on ExchangeId: 2073D9C11D9CFB0-0000000000000000). Exhausted after delivery attempt: 1 caught: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: couchbase://http:localhost:32858?bucket=testBucket20&password=xxxxxx due to: Invalid URI. Format must be of the form couchbase:http[s]://hostname[:port]?[options...]

orpiske commented 3 years ago

It does seem that the CKC gets the URL correctly, though: 

2020-11-16 10:07:10,621 [SinkConnector-0] INFO org.apache.camel.kafkaconnector.utils.CamelKafkaConnectMain - Creating Camel route from(direct:start)
2020-11-16 10:07:10,621 [SinkConnector-0] INFO org.apache.camel.kafkaconnector.utils.CamelKafkaConnectMain - .to(couchbase:http:localhost:32858?bucket=testBucket20&password=RAW(password)&username=RAW(Administrator))

So, it's seems like it's somewhere else (maybe in the component code itself).

oscerd commented 3 years ago

Probably is in the component. I'll have a look

oscerd commented 3 years ago

The only point possible is at CouchBaseEndpoint level, there a constructor raising that exception message.

orpiske commented 3 years ago

This issue still persists and the connector cannot, yet, work with Couchbase when using the properties.

I took some time to debug the problem and this is the stack trace and where it is messing up with the URL:

normalizeUri:785, AbstractCamelContext (org.apache.camel.impl.engine) -----> this is adding a // on the couchbase URI and rendering it invalid
prepareRecipient:269, SendDynamicProcessor (org.apache.camel.processor) 
process:144, SendDynamicProcessor (org.apache.camel.processor)   
run:395, RedeliveryErrorHandler$SimpleTask (org.apache.camel.processor.errorhandler)
schedule:148, DefaultReactiveExecutor$Worker (org.apache.camel.impl.engine)
scheduleMain:60, DefaultReactiveExecutor (org.apache.camel.impl.engine)
process:147, Pipeline (org.apache.camel.processor)
process:312, CamelInternalProcessor (org.apache.camel.impl.engine)
process:84, DirectProducer (org.apache.camel.component.direct)
process:218, SharedCamelInternalProcessor (org.apache.camel.impl.engine)
process:112, SharedCamelInternalProcessor$1 (org.apache.camel.impl.engine)
process:83, DefaultAsyncProcessorAwaitManager (org.apache.camel.impl.engine)
process:109, SharedCamelInternalProcessor (org.apache.camel.impl.engine)
send:189, DefaultProducerCache (org.apache.camel.support.cache)
send:176, DefaultProducerTemplate (org.apache.camel.impl.engine)
send:148, DefaultProducerTemplate (org.apache.camel.impl.engine)
put:175, CamelSinkTask (org.apache.camel.kafkaconnector)
oscerd commented 3 years ago

Is this done with 3.8.0 @orpiske ?

orpiske commented 3 years ago

Is this done with 3.8.0 @orpiske ?

No, this is still ongoing. I'll continue investigating this one next week.

orpiske commented 3 years ago

This should do the trick: https://github.com/apache/camel/pull/5156/files

Thanks to @tadayosi, who found a similar problem on Netty and led me to the root cause of the problem on Couchbase.

valdar commented 3 years ago

This should be fixed both in 0.7.3 and 0.9.0 (out today). @orpiske is there anything left on our part left to do? like eneablening/adding itests?

orpiske commented 3 years ago

This should be fixed both in 0.7.3 and 0.9.0 (out today). @orpiske is there anything left on our part left to do? like eneablening/adding itests?

@valdar I am double checking if we need to re-enable any additional test besides the MongoDB source one. I'll share an update in a couple of minutes.

orpiske commented 3 years ago

@valdar we're good to go. I re-enabled 2 other tests that started working well w/ 3.9. This one, though, is still flaky (failing 1 out of 10 times on Linux).

valdar commented 3 years ago

Ok so I marked as flaky test, leave it open and assigned to yourself @orpiske feel free to close it when/if you think it is done.