AtlasOfLivingAustralia / image-service

Image repository and tiling services

https://images.ala.org.au

0 stars 17 forks source link

Feature/pipelines bulk services #118

Closed djtfmartin closed 3 years ago

djtfmartin commented 3 years ago

AVRO upload services for use with pipelines.

This pull request is adding additional features to the image service to support the pipelines based ingestion. See https://github.com/AtlasOfLivingAustralia/la-pipelines/issues/6

This pull request includes the following:

Async web services for upload of zipped AVRO files for bulk ingestion of images from pipelines with persistent queue
Services to check status of an upload (to allow for external polling)

Screen Shot 2020-11-05 at 2 56 00 pm

Admin UI for monitoring the progress of ingestion
Formalisation of metadata fields supported for update with the addition of standard dublin core terms in use by GBIF for multimedia (eg. audience, contributor, publisher, created, type)
Export web services to allow for fast retrieval of a list of images stored for a dataset/dataResourceUid

Screen Shot 2020-11-05 at 2 55 41 pm

Admin UI

Screen Shot 2020-11-05 at 2 44 35 pm

Through the UI, and administrator can:

Enable/disable processing of uploads
Reload a AVRO file
Clear current queue

Additional features

The following can now be adjusted via the setting table:

Number of background threads used for thumbnailing can now be adjusted via the setting
Number of threads used to download images from a data publisher
Throttling of download threads

Screen Shot 2020-11-05 at 2 49 28 pm

Other changes:

Removal of staging logic as the UI was removed several versions ago.
Purge job for batch file uploads to remove completed jobs from database after a number of days (number of days is configurable in database).

djtfmartin commented 3 years ago

this has been superceded by other PRs