search

AtlasOfLivingAustralia / data-management

Data management issue tracking
7 stars 0 forks source link

Update original filename image url for Questagame #770

Open patkyn opened 3 years ago

patkyn commented 3 years ago

There are about 252,689 records in DB with http://api.questagame.com in original_filename for dr1902. Because, the raw csv file from Questagame are sent with 'https://api.questagame.com', pipelines treat these as new images due to url change and exported to image-service as new images to upload. Large number of images in the image-service queue had caused some performance issues in image-service.

sadeghim commented 2 years ago

Maybe this AtlasOfLivingAustralia/image-service#164 can help to solve the problem in future.

patkyn commented 2 years ago

As communicated with Questagame, the move to switch to https is a once-off effort and it won't be switched back and forth. However, Questagame also indicated that there will be a move to switch to Amazon S3 some time in future and this will be communicated to us before they switch as they're aware of this issue that we are facing.

Updated backend postgres with a script that does the batch updates of 500 records it takes about 10 minutes to execute to update 250k records for dr1902