Closed planetf1 closed 1 year ago
At some point we need to update the gradle build files/cicd to publish a maven artefact - update here if/when you need help with that/are ready to do it.
The hardest part is probably deciding the name(s) of the artifacts(assuming we keep them all distinct)
This will mean the connector gets published -- even as a snapshot -- so it will be downloadable. Later we can consider released versions.
At that time we should probably add an 'enforcedPlatform' reference in the top level gradle file -- this will cause us to use the same versions as egeria (this is how we did it in the hive megastore connector)
Agreed in team meeting 30 Aug that I will make the change to get the new connectors published as maven artifacts
After the recent changes, we generate an action attachment containing the three connectors:
These are all being published - but with names based on the directory name. Either we need to move the directory to use as the default component name (our usual pattern in egeria) or explicitly specify the project name (probably the simplest for now)
^^ These can be accessed NOW (since I merged the PR that I wrote before my vacation, but was awaiting review so only merged this morning)
I'll update the artifact names later in the week
^^ @dwolfson @lpalashevski @mandy-chessell
@planetf1 Thanks - I looked in both Maven Central and Sonatype for a jdbc.snapshot.jar but didn't see one? Am I looking for the wrong thing or looking in the wrong place?
The artifacts are only available on the SNAPSHOT repository, since we have not yet done a release for the database connectors (at least not the new ones). I did ask at the f2f but the consensus was not yet.
In the snapshot URL, as an example, the resource connector can currently be found under:
However, as these are snapshots, all files include the timestamp ,making it awkward to fetch
So you can issue a query like
wget http://oss.sonatype.org/service/local/artifact/maven/redirect\?r\=snapshots\&g\=org.odpi.egeria\&a\=jdbc-resource-connector\&v\=3.13-SNAPSHOT\&c\=jar
What this shows, however, is that the actual connector is not published ie:
We have javadoc, sources, but no actual code ;-)
Thanks for spotting.
Connectors now being published with new names:
And...
As an example we can see this also now includes the jar correctly:
At this point the http query isn't working - this may depend on an index update (or another bug?)
The http query will work -- for the jar, do not specify a classifier ie:
Note that both jars are currently built without dependencies. The integration connector does require the resource connector. Both require a jdbc connector for the database being used. I've not verified the runtime behaviour to see if anything else extra is required, though there should be no change here from previously
^ @davidradl @dwolfson @lcpopa
(extra points if anyone spots my error in the current code. cosmetic only!)
Inspecting .module file -- this is a gradle addition that provides additional metadata for gradle consumers
The top level metadata looks good:
"component": {
"group": "org.odpi.egeria",
"module": "egeria-connector-integration-jdbc",
"version": "3.13-SNAPSHOT",
"attributes": {
"org.gradle.status": "integration"
}
The java jar is properly represented with:
Checking the pom
And the files
And the maven-metadata.xml
timestamps, versions & list of files (pom, sources jar, module,java jar, javadoc jar) all look correct.
By manual inspection I do not see issues..... at least when publishing to local maven repo
Checking retrieving the jar via curl:
➜ newc curl -L -X GET "http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-resource-jdbc&v=3.13-SNAPSHOT" --output egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 134 100 134 0 0 740 0 --:--:-- --:--:-- --:--:-- 757
100 242 100 242 0 0 96 0 0:00:02 0:00:02 --:--:-- 250
100 4550 100 4550 0 0 1741 0 0:00:02 0:00:02 --:--:-- 0
➜ newc ls
egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
➜ newc file egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar: Zip archive data, at least v1.0 to extract, compression method=deflate
➜ newc jar -tf egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
META-INF/
META-INF/MANIFEST.MF
org/
org/odpi/
org/odpi/openmetadata/
org/odpi/openmetadata/adapters/
org/odpi/openmetadata/adapters/connectors/
org/odpi/openmetadata/adapters/connectors/resource/
org/odpi/openmetadata/adapters/connectors/resource/jdbc/
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnectorProvider.class
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnector.class
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnector$JdbcConnectorAsDataSource.class
➜ newc
This looks ok (repeat for any other connectors)
Next we check with the proper maven tools - and can see that again, the maven artifact is correctly downloaded:
➜ newc ls
➜ newc mvn dependency:get -DrepoUrl=https://oss.sonatype.org/content/repositories/snapshots -Dartifact=org.odpi.egeria:egeria-connector-resource-jdbc:3.13-SNAPSHOT -Ddest=egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:get (default-cli) @ standalone-pom ---
[WARNING] repositoryUrl parameter is deprecated. Use remoteRepositories instead
[INFO] Resolving org.odpi.egeria:egeria-connector-resource-jdbc:jar:3.13-SNAPSHOT with transitive dependencies
Downloading from temp: https://oss.sonatype.org/content/repositories/snapshots/org/odpi/egeria/egeria-connector-resource-jdbc/3.13-SNAPSHOT/maven-metadata.xml
Downloaded from temp: https://oss.sonatype.org/content/repositories/snapshots/org/odpi/egeria/egeria-connector-resource-jdbc/3.13-SNAPSHOT/maven-metadata.xml (1.4 kB at 2.4 kB/s)
[WARNING] destination/dest parameter is deprecated: it will disappear in future version.
[INFO] Copying /Users/jonesn/.m2/repository/org/odpi/egeria/egeria-connector-resource-jdbc/3.13-SNAPSHOT/egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar to /private/tmp/newc/egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
[WARNING] Notice transitive dependencies won't be copied.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.976 s
[INFO] Finished at: 2022-11-07T14:21:05Z
[INFO] ------------------------------------------------------------------------
➜ newc ls
egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
➜ newc file *jar
egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar: Zip archive data, at least v1.0 to extract, compression method=deflate
➜ newc jar -tf egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
META-INF/
META-INF/MANIFEST.MF
org/
org/odpi/
org/odpi/openmetadata/
org/odpi/openmetadata/adapters/
org/odpi/openmetadata/adapters/connectors/
org/odpi/openmetadata/adapters/connectors/resource/
org/odpi/openmetadata/adapters/connectors/resource/jdbc/
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnector$JdbcConnectorAsDataSource.class
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnectorProvider.class
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnector.class
In summary - for this repo's publish flow:
the maven client works to download the snapshot
More broadly we need to think about usability
I suggest these general issues are followed up in
Does that help @dwolfson @mandy-chessell ?
Tested with postgres connectors:
Using a file ~/etc/jdbc.yaml:
extralibs:
- url: http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-resource-jdbc&v=3.13-SNAPSHOT"
filename: egeria-connector-resource-jdbc.jar
- url: http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-integration-jdbc&v=3.13-SNAPSHOT"
filename: egeria-connector-integration-jdbc.jar
- url: https://jdbc.postgresql.org/download/postgresql-42.5.0.jar
filename: postgresql.jar
We can install an instance of the helm chart ie:
helm install lab egeria/odpi-egeria-lab --set jupyter.tokenPlain='s3cr3t!' --set strimzi.enabled=false --devel -f ~/etc/jdbc.yaml
This
Specifies the additional override above (all the settings could be moved into that file)
After the container is started, we can see the connector is present:
➜ etc kubectl logs lab-odpi-egeria-lab-core-0 -c init-connector
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 134 100 134 0 0 685 0 --:--:-- --:--:-- --:--:-- 687
100 787 0 787 0 0 1488 0 --:--:-- --:--:-- --:--:-- 3306
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 134 100 134 0 0 853 0 --:--:-- --:--:-- --:--:-- 858
100 793 0 793 0 0 1625 0 --:--:-- --:--:-- --:--:-- 1625
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1021k 100 1021k 0 0 714k 0 0:00:01 0:00:01 --:--:-- 714k
Downloads complete.
➜ etc kubectl exec -it lab-odpi-egeria-lab-core-0 -- /bin/ls -la /deployments/server/extralib Defaulted container "egeria" out of: egeria, init-connector (init) total 1044 drwxrwsrwx. 2 root 1000840000 4096 Nov 7 15:16 . drwxr-xr-x. 1 root root 4096 Nov 7 15:16 .. -rw-r--r--. 1 1000840000 1000840000 793 Nov 7 15:16 egeria-connector-integration-jdbc.jar -rw-r--r--. 1 1000840000 1000840000 787 Nov 7 15:16 egeria-connector-resource-jdbc.jar -rw-r--r--. 1 1000840000 1000840000 1046274 Nov 7 15:16 postgresql.jar ➜ etc
Update on above - the connectors are the wrong content .The reason is that the specific URL specified utilizes a query which results in a http redirect as a response. In the charts we do not follow redirects, so the http redirect is downloaded, not the target of the redirect..
This error was a user problem - missing quotes in the URL which then failed (incorrect end result, not a failure....) on the curl request Will look at whether any additional error checking can be done in the helm charts
The correct jdbc.yaml is:
extralibs:
- url: "http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-resource-jdbc&v=3.13-SNAPSHOT"
filename: egeria-connector-resource-jdbc.jar
- url: "http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-integration-jdbc&v=3.13-SNAPSHOT"
filename: egeria-connector-integration-jdbc.jar
- url: "https://jdbc.postgresql.org/download/postgresql-42.5.0.jar"
filename: postgresql.jar
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.
this was completed as part of the original work