opengeospatial / teamengine-docker

7 stars 22 forks source link

Solve problem with duplicated dependencies #29

Closed dstenger closed 2 years ago

dstenger commented 5 years ago

E.g. create script to remove duplicated dependencies automatically.

Description of the problem: Currently, in many cases, there are the same dependencies in different versions in the classpath (lib folder of TEAM Engine webapp). This problem is caused by different test suites having the same dependency but not in the same version. Due to that fact, a random version of those multiple dependencies is chosen leading to system related test failures in different test suites (e.g. ets-wfs20; java.lang.NoSuchMethodError: org.opengis.cite.validation.SchematronValidator.validate(Ljavax/xml/transform/Source;Z)Ljavax/xml/transform/Result;). Because of that, we are deleting older versions of multiple dependencies manually so that just the newest version stays in the classpath (of course, this is not a good workflow and more a workaround; a general solution is needed here). When a Docker Container is started this task of deleting multiple dependencies should be executed automatically (e.g. via script).

kstegemoller commented 3 years ago

To solidify requirements for this removal script, I am using the find-repeated-jars.sh documented in https://github.com/opengeospatial/cite-private/issues/212 suggested by @dstenger. I got these results with my test... I would expect us to keep the newest version by number. Other expectations? See my questions to the right.

=== Current jars             ===|===   jars to be Removed; jars to be kept marked with '<' ===

ets-gml311-3.1.jar         <
ets-gml32-0.1.jar           ets-gml32-0.1.jar   ### Do we need to keep different versions of ets?
ets-gml32-1.21.jar          ets-gml32-1.21.jar
ets-gml32-1.25.jar          ets-gml32-1.25.jar
ets-gml32-3.2.jar          <
fii-boo-zoom-fast-4.7.0.jar     fii-boo-zoom-fast-4.7.0.jar
fii-boo-zoom-fast-4.7.2.jar     fii-boo-zoom-fast-4.7.2.jar
fii-boo-zoom-fast-4.7.jar      <                        ### the 4.7.2 version should be kept. Do we need to worry about this use case?
foo-bar-baz-2.6.2.jar           foo-bar-baz-2.6.2.jar
foo-bar-baz-2.7.2.jar           foo-bar-baz-2.7.2.jar
foo-bar-baz-2.7.9.jar          <
geotk-utility-3.21.jar         <
geotk-utility-pending-3.21.jar      geotk-utility-pending-3.21.jar ### Should both pending and "non pending" versions be kept?
json-20080701.jar           json-20080701.jar
json-20090211.jar          <
liba-2.1.jar                liba-2.1.jar
liba-2.2.jar               <
lib-client-1.17.1.jar           lib-client-1.17.1.jar
lib-client-1.18.1.jar          <
lib-server-3.16.jar         lib-server-3.16.jar
lib-server-3.19.jar        <
teamengine-spi-5.3.jar          teamengine-spi-5.3.jar
teamengine-spi-ctl-5.3.jar     <                              ### BUG This appears to be a different jar should and should keep both.
wii-gee-2019-11-11.jar         <                              ### BUG Does not work -- will not fix?
wii-gee-2020-09-00.jar         <                              ### BUG Does not work -- will not fix?
zii-bee-20201110.jar            zii-bee-20201110.jar
zii-bee-20210990.jar           <
kstegemoller commented 3 years ago

A real example

=== Current jars             ===|===   Jars to be Removed; Jars to be kept marked with '<' ===
activation-1.1.jar         <
cite1-utils-1.1.1.jar          <
collection-0.6.jar         <
commons-cli-1.3.jar        <
commons-codec-1.10.jar         <
commons-codec-1.9.jar           commons-codec-1.9.jar
commons-fileupload-1.3.1.jar       <
commons-io-2.4.jar          commons-io-2.4.jar
commons-io-2.5.jar         <
commons-logging-1.2.jar        <
derby-10.10.2.0.jar        <
do                 <
ets-cat30-1.0.jar          <
ets-gml32-1.21.jar          ets-gml32-1.21.jar   ### REMOVE or KEEP?
ets-gml32-1.25.jar         <
ets-gpkg10-1.0.jar         <
ets-gpkg12-1.0.jar         <
ets-kml22-1.12.jar         <
ets-sta10-1.0.jar          <
ets-wfs11-1.29.jar         <
ets-wfs20-1.26.jar         <
ets-wms-client13-1.2.jar       <
geoapi-pending-3.1-M04.jar     <
geomatics-geotk-1.10.jar        geomatics-geotk-1.10.jar
geomatics-geotk-1.14.jar       <
geomatics-geotk-1.9.jar         geomatics-geotk-1.9.jar
geotk-epsg-3.21.jar        <
geotk-geometry-3.21.jar        <
geotk-jtswrapper-3.21.jar      <
geotk-metadata-3.21.jar        <
geotk-metadata-sql-3.21.jar    <
geotk-referencing-3.21.jar     <
geotk-temporal-3.21.jar        <
geotk-utility-3.21.jar         <                                     ### REMOVE OR KEEP non pending
geotk-utility-pending-3.21.jar      geotk-utility-pending-3.21.jar   ### REMOVE OR KEEP pending
geotk-xml-base-3.21.jar        <
geotk-xml-gml-3.21.jar         <
gson-2.2.2.jar             <
hamcrest-core-1.3.jar          <
httpclient-4.5.jar         <
httpcore-4.4.1.jar         <
isorelax-20030108.jar          <
jai-imageio-core-1.3.1.jar     <
javax.annotation-api-1.2-b03.jar   <
javax.servlet.jsp.jstl-1.2.3.jar   <
jaxen-1.0-FCS.jar          <
jcip-annotations-1.0.jar       <
jcommander-1.48.jar        <
jena-base-3.1.0.jar        <
jena-core-3.1.0.jar        <
jena-iri-1.1.2.jar          jena-iri-1.1.2.jar
jena-iri-3.1.0.jar         <
jena-shaded-guava-3.1.0.jar    <
jersey-client-1.19.jar         <
jersey-multipart-1.19.jar      <
jing-20091111.jar          <
joda-time-2.8.jar           joda-time-2.8.jar   ### Should have same number of decimals
joda-time-2.9.4.jar        <
json-20080701.jar           json-20080701.jar   ### BUG need to fix
json-20090211.jar           json-20090211.jar   ### BUG need to fix
json-simple-1.1.1.jar          <
jsr-275-0.9.3.jar          <
jstl-api-1.2.jar           <
jts-1.13.jar               <
junit-4.12.jar             <
mail-1.4.7.jar             <
mimepull-1.9.3.jar         <
saxon9-9.0.0.8.jar         <
saxpath-1.0-FCS.jar        <
schema-utils-1.7.jar            schema-utils-1.7.jar
schema-utils-1.8.jar           <
slf4j-api-1.7.20.jar           <
slf4j-jdk14-1.7.21.jar         <
sqlite-jdbc-3.8.11.2.jar       <
teamengine-core-5.3.jar        <
teamengine-spi-5.3.jar          teamengine-spi-5.3.jar   ### BUG should be kept
teamengine-spi-ctl-5.3.jar     <
testng-6.9.10.jar          <
vecmath-1.5.2.jar          <
webservices-api-2.3.jar        <
webservices-rt-2.3.jar         <
xercesImpl-2.11.0.jar          <
xml-apis-1.4.01.jar        <
xml-resolver-1.2.jar           <
kstegemoller commented 3 years ago

from teamengine docker /usr/local/tomcat/webapps/teamengine/WEB-INF/lib

kstegemoller commented 3 years ago

@dstenger @ghobona

Do these rules work for removing jars?

As seen above...

Assumptions:

ghobona commented 3 years ago

@kstegemoller I think we only need to keep the latest version of each ets jar.

@dstenger Please confirm.

dstenger commented 3 years ago

ets-gml32-0.1.jar ets-gml32-0.1.jar ### Do we need to keep different versions of ets?

No, just one version. However, be careful that some names are longer: e.g. ets-wms13-dgiwg-1.0.jar. The current script is not able to handle those.

fii-boo-zoom-fast-4.7.jar < ### the 4.7.2 version should be kept. Do we need to worry about this use case?

As this is not a clean workflow anyway, it probably depends on the dependency. So, I would not worry about this use case for now.

geotk-utility-pending-3.21.jar geotk-utility-pending-3.21.jar ### Should both pending and "non pending" versions be kept?

Same as above. Please check how I did the manual deleting.

teamengine-spi-ctl-5.3.jar < ### BUG This appears to be a different jar should and should keep both.

Correct.

ets-gml32-1.21.jar ets-gml32-1.21.jar ### REMOVE or KEEP?

Old versions can be removed.

dstenger commented 3 years ago

As far as I remember, I kept both versions if one was marked with "pending".

As this is just a workaround (which should be fixed by adjusting the versions in the test suites itself), in some cases there is no right or wrong (there might even be the case that one test suite needs one version and the other test suite another version; so both test suites are not compatible with each other).

So, I propose to keep it rather simple for now and do extensive testing of the new Beta environment instead.

dstenger commented 2 years ago

Was solved by script placed next to Dockerfiles: