search

gbif / pipelines

Pipelines for data processing (GBIF and LivingAtlases)
Apache License 2.0
40 stars 28 forks source link

K8s: Improve the file cooking process for identifiers and the interpretation stages #1055

Closed muttcg closed 2 months ago

muttcg commented 2 months ago

For large datasets, identifiers and interpretation (some other?) stages can generate a large number of relatively small files, which directly impact the performance of the system.

Can be linked to: #1048