Problem:
Whenever a major image provider (like iNat) changes a common part of their image URLs like protocol, domain or path causes image-service to start downloading all of them and then compare the hash code of the image with the image DB and find if they are duplicate or not and this can take a significant time from image-service and block the other loads.
Suggestion:
Implement groups of regular expressions for each DR that can extract the IDs of images prior to download and match it with the existing image URLs for that data resource in the database. Matched URLs will be added as alternative URL without downloading the images.
Non matched URLs will go through and they are needed to be downloaded.
This will eliminate the need to download all the images for most of the cases.
Problem: Whenever a major image provider (like iNat) changes a common part of their image URLs like protocol, domain or path causes image-service to start downloading all of them and then compare the hash code of the image with the image DB and find if they are duplicate or not and this can take a significant time from image-service and block the other loads.
Suggestion: Implement groups of regular expressions for each DR that can extract the IDs of images prior to download and match it with the existing image URLs for that data resource in the database. Matched URLs will be added as alternative URL without downloading the images. Non matched URLs will go through and they are needed to be downloaded. This will eliminate the need to download all the images for most of the cases.