getindata / flink-http-connector

Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Apache License 2.0
136 stars 39 forks source link

HTTP-63 caching for sync lookups #94

Open davidradl opened 2 months ago

davidradl commented 2 months ago

Description

Introduce caching similar to the Flink JDBC connector, so long running synchronous Flink jobs can cache content resulting in less API calls.

Resolves https://github.com/getindata/flink-http-connector/issues/63

PR Checklist
davidradl commented 2 months ago

@kristoffSC @grzegorz8 the 1.18.1 job is failing with

14:50:32,522 [INFO] ------------------------------------------------------------------------
14:50:32,522 [INFO] BUILD SUCCESS
14:50:32,522 [INFO] ------------------------------------------------------------------------
14:50:32,523 [INFO] Total time:  02:01 min
14:50:32,523 [INFO] Finished at: 2024-05-09T14:50:32Z
14:50:32,523 [INFO] ------------------------------------------------------------------------
cat: target/site/jacoco/index.html: No such file or directory

Error: Process completed with exit code 1.

This looks like an infrastructure issue - as the build is successful then there is a cat error.

davidradl commented 2 months ago

@kristoffSC @grzegorz8 it looks like the build is successful but fails - is the machine running out of disk space / lost access to something maybe?

kristoffSC commented 1 month ago

I will take a look why build fails.

davidradl commented 1 month ago

@AdrianVasiliu thank you very much for your detailed review, I really appreciate it. Great feedback :-)

davidradl commented 1 month ago

@kristoffSC a gentle nudge, we are looking to adopt this change, any news on the build issue?

MarekMaj commented 1 month ago

Hi @davidradl, we merged changes related to the build issue. Could you please rebase your commit?

davidradl commented 4 weeks ago

@MarekMaj thanks for making this change - it works for 2 of the releases bu not for 1.18.1, where I now see the CI fail with error

Run madrapps/jacoco-report@v1.2
Event is pull_request
base sha: 9bfbb4c8a1b722eac8dd1f1b1b71b1fcbf1f02ce
head sha: 18f2d82ac13333344bacd4a02e7e2f486a649ae6
node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: ENOENT: no such file or directory, open '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/runner/work/flink-http-connector/flink-http-connector/target/site/jacoco/jacoco.xml'

}

I suspect this is related to your recent jakoco commit

MarekMaj commented 4 weeks ago

@davidradl indeed, for PR created from forked repositories it caused problems due to insufficient permissions. I tested build again using forked repo, please rebase your code again:)

davidradl commented 2 weeks ago

@kristoffSC @MarekMaj hope to get to this this week. Thanks for your feedback.