do not store the full image in the database

[eroux]

The current recipe stores the full images (in base64) in the database, but we don't want to do that for two reasons:

• it will make the database too big relatively quick
• it looses the information about file names, so we can't really crop the images once they're annotated

For that the stream could just use s3 URIs with a temporary token. That means that the image URIs will stop working at some point, but this will preserve the s3 key and we can reconstruct new s3 URIs later on

[Zakongjampa]

For this, the prodigy will accept an argument when we start the prodigy. We can pass --remove-base64 which will not save the base64 data of the image in the database. I hope this is helpful sir.

[eroux]

well, but then what will prodigy store? If it doesn't store the image url on s3, how can we reconstruct it from what's in the database?

[ta4tsering]

issue resolved, below is the log image using sudo journalctl -u prodigy_bdrc_crop_images.service

One Example of yielded dict to the stream as below: {image: https://s3.amazonaws.com/image-processing.bdrc.io/NLM1/W2KG208129/sources-web/W2KG208129-I2KG208175/I2KG2081750001.tif_19.png?AWSAccessKeyId=AKIASPOFTMDCENMI&Signature=p5C5atPCHw1SsyElCgh0ZFm00T4%3D&Expires=1674025970}

p.s : Image URL ExipresIn=36000 or in 10 hours

[eroux]

thanks! It looks good, is there a way to make it (much) longer than that, something like a year? If not we'll have to make a script that goes through the database and updates the URL of the images

[ta4tsering]

I don't know the upper limit to the ExpiresIn, I will check and update to the upper limit.

[ta4tsering]

As per this documentation Documentation Link nothing is mentioned about the maximum value for the ExpiresIn. But then as per this Documentation Documentation link it says maximum value is 7 days. But I put the ExpiresIn as one year or 31536000 seconds and restarted the server, to see if one year works and It works fine for now.

[eroux]

ok great! Let's create one Python script (independent from Prodigy) that does the following:

• connects to a sqlite database that is passed as an argument
• reads all lines and each time it spots an s3 url with a token, it updates the token and saves the database

and then an "update-database.sh" script that:

• stops the Prodigy service
• launches the Python script on the database
• starts the Prodigy service again

[ta4tsering]

sure, will look into it

[ta4tsering]

here is the python script to update the s3_url or to update to extend the URL's ExpiresIn. update_s3_url.py

below is the test, in the test, I am assertting the status_code of the response of the new url. test_update_s3_url.py

and finally the update-database.sh

[eroux]

this is good! some remarks:

• the test file should use the code in update_s3_url.py and not duplicate it, otherwise it doesn't really test the file
• before updating the token, the code should check if the URL is indeed an s3 url, there are cases where the url will just be a iiif url, no need to try to update the token in that case

[ta4tsering]

okay, I will make those changes