search

Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
26 stars 5 forks source link

jar management #156

Open soxofaan opened 2 years ago

soxofaan commented 2 years ago

we have several places in openeo-geopyspark-driver where we declare/get geotrellis related jars.

e.g. geotrellis-backend-assembly:

scripts/submit-batch-test.sh
18:backend_assembly=https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.7-openeo.jar

tests/pre_test.sh
8:curl -L -O -C - https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.6-openeo_2.12.jar

scripts/submit.sh
17:backend_assembly=https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.7-openeo.jar

scripts/get-jars.py
49:        "https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.6-openeo_2.12.jar"

benchmarks/benchmark.sh
16:backend_assembly=https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.7-openeo.jar

scripts/submit-local.sh
21:backend_assembly=https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.7-openeo.jar

openeogeotrellis/deploy/submit_batch_job_spark24.sh
64:   backend_assembly=https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.6-openeo_2.12.jar

openeogeotrellis/deploy/submit_batch_job_spark3.sh
64:   backend_assembly=https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.6-openeo_2.12.jar

openeogeotrellis/deploy/submit_batch_job.sh
64:   backend_assembly=https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.7-openeo.jar

geotrellis-extensions:

scripts/get-jars.py
45:        "https://artifactory.vgt.vito.be/libs-snapshot-public/org/openeo/geotrellis-extensions/2.3.0_2.12-SNAPSHOT/geotrellis-extensions-2.3.0_2.12-SNAPSHOT.jar"

scripts/submit.sh
16:extensions=https://artifactory.vgt.vito.be/libs-snapshot-public/org/openeo/geotrellis-extensions/2.2.0-SNAPSHOT/geotrellis-extensions-2.2.0-SNAPSHOT.jar

scripts/submit-batch-test.sh
16:extensions=https://artifactory.vgt.vito.be/libs-snapshot-public/org/openeo/geotrellis-extensions/2.2.0-SNAPSHOT/geotrellis-extensions-2.2.0-SNAPSHOT.jar

scripts/submit-local.sh
20:extensions=https://artifactory.vgt.vito.be/libs-snapshot-public/org/openeo/geotrellis-extensions/2.2.0-SNAPSHOT/geotrellis-extensions-2.2.0-SNAPSHOT.jar

benchmarks/benchmark.sh
15:extensions=https://artifactory.vgt.vito.be/libs-snapshot-public/org/openeo/geotrellis-extensions/2.2.0-SNAPSHOT/geotrellis-extensions-2.2.0-SNAPSHOT.jar

tests/pre_test.sh
7:curl -L -O -C - https://artifactory.vgt.vito.be/libs-snapshot-public/org/openeo/geotrellis-extensions/2.3.0_2.12-SNAPSHOT/geotrellis-extensions-2.3.0_2.12-SNAPSHOT.jar

Obviously this is very annoying to maintain (illustrated by the different versions present here)

soxofaan commented 2 years ago

can't we just define a simple text file with single source of truth references (in the style of python's requirements.txt), which can then be consumed from the various places we need (bash scripts, python scripts, ...)