search

Open-EO / openeo-gfmap

Generic framework for EO mapping applications building on openEO
Apache License 2.0
4 stars 0 forks source link

JobManager: store joblist on http url as Parquet #107

Open jdries opened 1 month ago

jdries commented 1 month ago

Job manager csv file is now written somewhere on a posix filesystem, making it relatively hard to access from anywhere.

Eventually, something like this should end up in the APEx 'user workspace', which actually will be an object storage bucket.

For now however, I propose a simply upload to artifactory, which is also quite easy: https://gist.github.com/pgorsira/ca978d06b90f9019bedb

I also propose the use of Parquet, which is a lot more efficient and smaller then csv.

(Fully converting jobmanager to parquet is another issue, this one allows us to do followup prototyping tasks without that effort.)

jdries commented 1 month ago

Extra: jobtracker uses multiple files, so we should use partitioned geoparquet files on collection id.