Async web services for upload of zipped AVRO files for bulk ingestion of images from pipelines with persistent queue
Services to check status of an upload (to allow for external polling)
Admin UI for monitoring the progress of ingestion
Formalisation of metadata fields supported for update with the addition of standard dublin core terms in use by GBIF for multimedia (eg. audience, contributor, publisher, created, type)
Export web services to allow for fast retrieval of a list of images stored for a dataset/dataResourceUid
Admin UI
Through the UI, and administrator will be able to:
Enable/disable processing of uploads
Reload an AVRO file
Clear current queue
Additional features
In addition, the following can need to be adjusted via the setting table:
Number of background threads used for thumbnailing can now be adjusted via the setting
Number of threads used to download images from a data publisher
Throttling of download threads
Other changes required:
Removal of staging logic as the UI was removed several versions ago.
Purge job for batch file uploads to remove completed jobs from database after a number of days (number of days is configurable in database).
Additional features to the image service to support the pipelines based ingestion.
See https://github.com/AtlasOfLivingAustralia/la-pipelines/issues/6
Thist includes the following:
Admin UI for monitoring the progress of ingestion
Formalisation of metadata fields supported for update with the addition of standard dublin core terms in use by GBIF for multimedia (eg.
audience,contributor,publisher,created,type)Export web services to allow for fast retrieval of a list of images stored for a dataset/dataResourceUid
Admin UI
Through the UI, and administrator will be able to:
Additional features
In addition, the following can need to be adjusted via the
settingtable:thumbnailingcan now be adjusted via thesettingOther changes required: