javier-molina
commented
3 years ago Respective changes for Dave and Simon already reviewed.
@sbearcsiro I missed yesterday this is a PR not an issue so I will leave it here for you to either close or merge as needed.
This pull request is adding additional features to the image service to support the pipelines based ingestion. See AtlasOfLivingAustralia/la-pipelines#6
This pull request includes the following:
Async web services for upload of zipped AVRO files for bulk ingestion of images from pipelines with persistent queue
Services to check status of an upload (to allow for external polling)
Admin UI for monitoring the progress of ingestion
Formalisation of metadata fields supported for update with the addition of standard dublin core terms in use by GBIF for multimedia (eg. audience, contributor, publisher, created, type)
Export web services to allow for fast retrieval of a list of images stored for a dataset/dataResourceUid
Admin UI
Through the UI, and administrator can:
Enable/disable processing of uploads Reload a AVRO file Clear current queue Additional features The following can now be adjusted via the setting table:
Number of background threads used for thumbnailing can now be adjusted via the setting Number of threads used to download images from a data publisher Throttling of download threads
Other changes:
Removal of staging logic as the UI was removed several versions ago. Purge job for batch file uploads to remove completed jobs from database after a number of days (number of days is configurable in database).