search

redis-field-engineering / redis-connect-dist

Real-Time Event Streaming & Change Data Capture
https://redis-field-engineering.github.io/redis-connect-dist/
Other
44 stars 11 forks source link

Windows version can't create a job due to credentialsFilePath #9

Closed ronan-vargas closed 2 years ago

ronan-vargas commented 2 years ago

Hello, I successfully started Redis Connect in my Windows Machine, but when I try to create a job for Oracle integration through API, it throws an error:

"ERROR: Failed to save configuration for JobName: cdc_job Validation Failure: Invalid CredentialsFilePath: C:\credentialscdc".

print_api_error_create_job

curl -X POST "http://localhost:8282/connect/api/v1/job/config/cdc_job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@cdc_job.json;type=application/json"

print_c_credentials_cdc
viragtripathi commented 2 years ago

Hi @ronan-vargas, I don't see your Redis (target) credentials file (or maybe it's not visible in the screenshot you have provided). Can you please try this,

Move all credentials file from the location you have chosen for credentialsFilePath except these,

The file format for Redis Connect credentials are as follows, redisconnectcredentials<source|target>_.properties, the source name is same as database product name on the jdbc url and target is always redis. I noticed you have one file with UPPERCASE ORACLE and that shouldn't be there either.

redisconnect_credentials_jobmanager.properties, there will always be one jobmanager file per Redis Connect instance/node.

The fact that you're able to start Redis Connect instance means you had no issue connecting to your metadata Redis DB and the credentialsPath for this must be correct in the jobmanager.properties file which is read at the Instance startup to connect to Redis so all of Redis Connect metadata can be persisted in Redis. Also, your Redis Enterprise DB endpoint doesn't seem correct so please fix the Redis URI and provide a valid Redis Enterprise DB endpoint.

ronan-vargas commented 2 years ago

Hello @viragtripathi, thank you very much for answering me.

After reviewing some steps (before your answer) I've successfully created a job through Redis Connect. For to do that, I also needed to put "ojdbc*.jar" in the "lib" directory and also used "cdc-job" as jobName, instead of "cdc_job". However, now Redis Connect throws an exception when I try to start the job for the first time, using 'load' as jobType.

2022-07-26 10:42:32,773 ERROR redis-connect-manager [JobManagerThreadpool-2] Instance: 22996@IBM-PF3R4N79 failed to initialize JobId: {connect}:task:partition:2:cdc-job:C##RCUSER.EMP EXCEPTION MESSAGE: RedisConnectValidationException: STACKTRACE: [Validation Failure: InstanceId: 22996@IBM-PF3R4N79 will not start JobId: {connect}:task:partition:2:cdc-job:C##RCUSER.EMP because there are no records in Table: C##RCUSER.EMP, at com.redis.connect.pipeline.producer.impl.load.RDBInitialLoadProducer.startSourceConnection(RDBInitialLoadProducer.java:78), at com.redis.connect.cluster.service.JobManager.startJobProducer(JobManager.java:713), at com.redis.connect.cluster.service.JobClaimer.initializeJob(JobClaimer.java:214), at com.redis.connect.cluster.service.JobClaimer.claimJob(JobClaimer.java:171), at com.redis.connect.cluster.service.JobClaimer.run(JobClaimer.java:81), at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539), at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305), at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305), at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136), at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635), at java.base/java.lang.Thread.run(Thread.java:833)]

The problem is that my C##RCUSER.EMP table has records and these records was inserted before starting the job.

select count(*) from C##RCUSER.emp;

count(*)

11

My configuration files and Connect logs are in attachment:

cdc-job.json.txt redisconnect_credentials_jobmanager.properties.txt redisconnect_credentials_oracle_cdc-job.properties.txt redisconnect_credentials_redis_cdc-job.properties.txt redis-connect-22996.log.txt redis-connect-manager-22996.log.txt

viragtripathi commented 2 years ago

Thanks @ronan-vargas you already made good progress. You're right for Oracle, you do need the client driver jar as we can't include it due to licensing. I see you have some issue with the driver you're using. From the connector log, No suitable driver STACKTRACE: [java.sql.SQLException: No suitable driver

Can you try this Oracle JDBC Driver and restart your instance then re-run the oracle-job? Also, please change the partition to 1 if you only have 11 records to load otherwise you would fail in another validation as <500 records can only have 1 partition OR if you want to add more records then see here, https://github.com/redis-field-engineering/redis-connect-dist/tree/main/examples/oracle/demo#initial-loader-steps

ronan-vargas commented 2 years ago

Hello, Considering initial loader steps link, I've used a distinct Oracle docker image, and this one doesn't have "sqlldr" tool. Consequently, I've inserted records on "emp" table directly by Oracle SQL Developer. Moreover, I've noticed that "load_sql.sh" script references a table called "employees". However, I think that previous script only creates "emp" table.

About your last answer, even changing "partitions" property in .json configuration file, the error "there are no records in Table: C##RCUSER.EMP" is still occurring.

Couldn't be the "#" in user name (C##RCUSER) a problem? My point is that "#" character is commonly used for comments in Properties files.

viragtripathi commented 2 years ago

Please see, https://github.com/redis-field-engineering/redis-connect-dist/blob/main/examples/oracle/demo/load_sql.sh#L26 for the load if you still need it.

The error was due to the driver you're using earlier. Have you replaced the driver with the link I sent?

ronan-vargas commented 2 years ago

The driver that I was using is also called "ojdbc8.jar", but the file in my machine was a bit larger (4963 KiB vs. 4369 KiB). I've replaced the file with the one you suggested, restarted Redis Connect, updated the job again and tried to start. Unfortunately, the exception "there are no records in Table..." is still occurring. Now I'll try this new configuration (I'm using Oracle 12c) for *JAVA_OPTIONS and I'll send a message in case of any problem.

Thanks in advance.

viragtripathi commented 2 years ago

If you're using 12c then you shouldn't see Got minus one from a read call error I mentioned earlier. Are you still seeing any driver related error in the connector log i.e. redis-connect-<PID>.log?

viragtripathi commented 2 years ago

Hi @ronan-vargas as discussed on the call, I am going to close this issue as all of your setup issue has been resolved and you have both load and stream job running with Oracle as a source and Redis Enterprise DB as target.

ronan-vargas commented 2 years ago

Hi @viragtripathi , OK, but the only thing that isn`t so clear to me is the fact that the setup works in "Initial Load" scenario (even with the Oracle's service name with ".localdomain" suffix) but it doesn't work in "CDC" scenario. I'll send by email the contact of everyone that was in the meeting.

viragtripathi commented 2 years ago

Hi @ronan-vargas, it probably works with initial load as it's a different connection using the databaseURL in the payload but stream job i.e. cdc uses a different connection that is started and managed by Debezium based on the database. properties and when you pass database.dbname with .localdomain suffix, you see the oracle driver error with TNS listener issue. This is why your Oracle developer connection also worked with .localdomain as it's also using the same JDBC URL like the initial loader.