Images missing for multiple NHMD Pinned Insects & Herbarium exports - Githubissues

NHMDenmark / Pinned-Insects-workstation

Work on the Pinned Insects workstation and workflow for mass digitisation in Denmark (DaSSCo)

0 stars 0 forks source link

Images missing for multiple NHMD Pinned Insects & Herbarium exports #113

Open beckerah opened 2 weeks ago

beckerah commented 2 weeks ago

Description: While checking exports for NHMD Pinned Insects, I came across a few exports that I couldn't easily find the images for. After some research, it seems these specimens were digitized while the ingestion client was producing errors and not correctly ingesting images. I read through the slack messages (channel: ingestion_client_users) concerning the issue, and it sounds like the images were all successfully ingested July 29 - 30, 2024. Looking through those image folders, I was able to find some but not all of the images. I am still unable to locate images for the following exports:

NHMD_PinnedInsects_20240722_14_41_JMJ_original_copy.csv (Barcodes: 1739481-1751790)
NHMD_PinnedInsects_20240724_15_59_MJG_original.csv (Barcodes: 1740064-1740140)

The first export has some data in it that needs to be checked against the image so I'm unable to proceed with that until we locate the images.

Next steps: I need to ask Khaled and Bhupjit if they have any leads as to where the images could be.

beckerah commented 1 week ago

Images for this export were found:

NHMD_PinnedInsects_20240724_15_59_MJG_original.csv (Barcodes: 1740064-1740140)

Still missing images for:

NHMD_PinnedInsects_20240722_14_41_JMJ_original_copy.csv (Barcodes: 1739481-1751790)

beckerah commented 1 week ago

Images for both exports have now been located.

However, I am having trouble locating images for:

NHMD_PinnedInsects_20240716_14_47_RL_processed.tsv (254 out of 348 images missing, others in a HERB folder)
NHMD_PinnedInsects_20240718_11_57_AI_processed.tsv (77 out of 123 images missing)

beckerah commented 1 week ago

I'm also having trouble finding images to match NHMD Herbarium exports from July. I've now got a script running to match barcodes to GUIDs in a database. It will take a few days to go through all the folders and add everything to the database but once it's finished, we should be able to see where all of the images are (assuming they exist on the N drive.)

beckerah commented 1 week ago

I am still finding exports that I cannot locate matching images for. These were all from the end of July 2024. Pip suggested I ask Khaled and/or Bhupjit if they have any ideas about where these images could be. After gaining their insight, I will either:

hold off on processing this data for specify until we can locate the images
divide the spreadsheets and only import the records we're certain are accurate

It's possible that these specimens may need to be re-imaged.

beckerah commented 1 week ago

I was able to resume using an old script that matches image GUIDs with their respective barcodes in a database table. It will take a while to go through all the folders but already I've located some of the missing images using the database.

Additionally, Rebekka found some images on the WORKHERB0003 workstation from July and August that were not ingested due to errors. She is ingesting them now. Hopefully some of the missing herbarium images are in that set.