Closed mandy-chessell closed 1 year ago
If the contents are different, we need to correct :
previously I did check the content, but this would have been prior to the addition of the content pack
there is a final validation check step indicated in the gradle issue at #3370 to check we're producting the same artifacts as before (or explain where we don't).
We can update the gradle archive by updating the build.gradle. I'd be inclined to unpack maven + gradle, then do a diff to confirm all the different files - you can make the changes, or if you prefer I'm happy to do so
Unpacking
I have no issues unpacking the archive ie:
/tmp gunzip -c /Users/jonesn/src/egeria/v4/open-metadata-distribution/open-metadata-assemblies/build/distributions/egeria-4.0-SNAPSHOT-distribution.tar.gz | tar -xvf -
But note this archive is a .gz, not a tar. 'modern' versions of tar can handle gz files natively, but the invocation above works regardless (old hat unix...). That being said, I checked and even /usr/bin/tar on MacOS 13 works just fine ie:
/usr/bin/tar -xvf /Users/jonesn/src/egeria/v4/open-metadata-distribution/open-metadata-assemblies/build/distributions/egeria-4.0-SNAPSHOT-distribution.tar.gz
So it's likely a permissions issue locally. I would check where you are unpacking to.
Great to see more use of the gradle build so we can flush out these issues (hopefully will be minor, as I use it all the time) before we flip over to use as default
One thing to consider - should the sample data & content packs be in this archive?
We have discussed creating container images with more of our contents and sample content, perhaps less so what we'd do for anyone just consuming directly (perhaps the answer is to only document) -- since these content packs and samples probably shouldn't be in the core.
That being said, in the content of a build migration, consistency as a first step does make sense, so unless we wish to prereq that change, then we should add them
The sample-data and content-packs need to be in the install image because they are used by the labs. Not everyone will run with K8 so we need to insure the install image is complete.
The sample-data and content-packs need to be in the install image because they are used by the labs. Not everyone will run with K8 so we need to insure the install image is complete.
In that case I think the right longer term solution would be to create an assembly outside the core egeria repository, which can aggregate across repos. Just as we will for an aggregate container image. (maybe it's the same repo doing this aggregation). Then we can have samples, content in the best place, including additional connectors etc, whilst also ensuring easy to use across k8s and non-k8s environments.
But, as above, still makes sense to keep these 2 aspects separate, and add in the content here for now to maintain consistency with the maven build.
I have done some more investigation:
If I use gunzip
the file name changes from XXX.tar.gz
to XXX.tar
. I can then call the tar
command and it unpacks ok. However the resulting directory name is XXX.tar.gz
. This is the cause of the error. The resulting directory name should be XXX
or egeria-3.14
in this example.
The failure to directly unpack the .gz file is because it is readonly (correctly) and so the tar command is not able to over-write the original tar file.
Here is the command sequence I used ...
cd ~/egeria-install
mandy-chessell@Amandas-MacBook-Pro egeria-install % gunzip *
mandy-chessell@Amandas-MacBook-Pro egeria-install % ls
egeria-3.14-distribution.tar
mandy-chessell@Amandas-MacBook-Pro egeria-install % tar -xvf egeria-3.14-distribution.tar
x egeria-3.14-distribution.tar.gz/
x egeria-3.14-distribution.tar.gz/server/
x egeria-3.14-distribution.tar.gz/server/server-chassis-spring-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/
x egeria-3.14-distribution.tar.gz/server/lib/audit-log-console-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/audit-log-event-topic-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/audit-log-file-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/audit-log-slf4j-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/cohort-registry-file-store-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/open-metadata-archive-file-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/open-metadata-archive-directory-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/inmemory-repository-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/graph-repository-connector-jar-with-dependencies-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/omrs-rest-repository-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/avro-file-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/basic-file-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/csv-file-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/data-folder-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/configuration-encrypted-file-store-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/configuration-file-store-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/discovery-service-connectors-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/dynamic-archiver-connectors-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/governance-action-connectors-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/files-integration-connectors-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/kafka-integration-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/openapi-integration-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/openlineage-integration-connectors-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/elasticsearch-integration-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/inmemory-open-metadata-topic-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/kafka-open-metadata-topic-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/open-lineage-janus-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/open-metadata-security-samples-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/spring-rest-client-connector-3.14.jar
x egeria-3.14-distribution.tar.gz/server/lib/governance-services-sample-3.14.jar
x egeria-3.14-distribution.tar.gz/utilities/
x egeria-3.14-distribution.tar.gz/utilities/glossary-canonical-model-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/utilities/open-metadata-types-utility-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/samples/
x egeria-3.14-distribution.tar.gz/samples/asset-create-avro-sample-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/samples/asset-create-csv-sample-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/samples/asset-reader-avro-sample-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/samples/asset-reader-csv-sample-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/samples/governance-leadership-sample-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/samples/governance-subject-area-sample-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/samples/governance-zone-create-sample-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/samples/subject-area-definition-sample-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/samples/admin-services-config-metadata-server-sample-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/user-interface/
x egeria-3.14-distribution.tar.gz/user-interface/ui-chassis-spring-3.14-plain.jar
x egeria-3.14-distribution.tar.gz/conformance-suite/
x egeria-3.14-distribution.tar.gz/conformance-suite/open-metadata-conformance-suite-client-3.14-jar-with-dependencies.jar
x egeria-3.14-distribution.tar.gz/LICENSE
x egeria-3.14-distribution.tar.gz/truststore.p12
x egeria-3.14-distribution.tar.gz/NOTICE
x egeria-3.14-distribution.tar.gz/keystore.p12
x egeria-3.14-distribution.tar.gz/content-packs/
x egeria-3.14-distribution.tar.gz/sample-data/
mandy-chessell@Amandas-MacBook-Pro egeria-install % ls
egeria-3.14-distribution.tar egeria-3.14-distribution.tar.gz
mandy-chessell@Amandas-MacBook-Pro egeria-install % cd *gz
mandy-chessell@Amandas-MacBook-Pro egeria-3.14-distribution.tar.gz % ls
LICENSE keystore.p12 truststore.p12
NOTICE sample-data user-interface
conformance-suite samples utilities
content-packs server
mandy-chessell@Amandas-MacBook-Pro egeria-3.14-distribution.tar.gz %
The directory structure is created by the distribution plugin. There's no obvious, simple way to customize.
When originally coded I felt it appropriate to stick with default conventions setup by our build tools -- even though in this case gradle differs from maven.
However a similar issue is reported in stack overflow which adds some customization - but it may have other implications.
Additionally the steps that unpack & run FVTs will need minor changes if the distribution archive is changed.
If no immediate change is obvious, I suggest we split these observations into individually manageble chunks so
ok - I will do the doc update - was making changes to original instructions when discovered this.
I have:
These changes means that the resources needed by the developer dojo can be sourced from the gradle build (and I can continue converting it from using maven to gradle).
Sounds good.
Existing/related issue?
No response
Current Behavior
The gradle build creates a distribution file that is supposed to be the same as the maven distribution file. Trying the 3.14 release there are two problems:
This is a list of the contents:
These are the errors when it is unpacked:
Expected Behavior
The following command should unpack the distribution file and create a directory called
egeria-3.14
with the full distribution contents.Steps To Reproduce
Build release 3.14 using gradle then:
List the files:
The distribution tar file is
{{release}}-distribution.tar.gz
oregeria-3.14-distribution.tar.gz
in this example.Create a directory for the install and copy the tar file into it. The two commands shown below create an install directory at the same level in the file system as the egeria build library and then copies the egeria distribution file into it.
These next commands change to the new directory and lists its contents.
It is now possible to unpack the tar file.
Environment
Any Further Information?
No response