IIIF ingest ("Picturae")

[pdaengeli]

Goal

Ingest of tif images from resstore.unibe.ch/boga_herbarium_bernense/Picturae to iiif.ub.unibe.ch.

Target structure:

Sample requests (collection slug + file name):

• https://iiif.ub.unibe.ch/image/v3/boga/Sheet-0056800.tif/info.json
• https://iiif.ub.unibe.ch/image/v3/boga/Sheet-0056800.tif/full/800,/0/default.jpg

Process flow

• rsync with --include-from include.txt --exclude '*' to selectively copy from resstore to NAS
• restructure folders from boxes to sheets (core command: for file in **/*.tif; do dir=$(echo $file | cut -d. -f1); mkdir -p $dir; mv $file $dir; done)
• EDIT: split by sheet number
• bagit: sudo bagit.py --contact-name 'XY' --md5 bogabag_00XXX
• Stage at UB share (sudo rsync -r bogabag_00* /volume1/docker/delivery/)
• Import task
• clean-up (rm -r /volume1/docker/delivery/_processed/bogabag_00XXX) + clean-up scratch area

Progress control

Fetching to NAS and restructuring

• [x] ├── 2022_10_03 (bogabag_00000-000219)
• [x] ├── 2022_10_04 (bogabag_00021-00046)
• [x] ├── 2022_10_05 (bogabag_00046-00074)
• [x] ├── 2022_10_06 (bogabag_00074-00105)
• [x] ├── 2022_10_07 (bogabag_00105-00132)
• [x] ├── 2022_10_10 (up to here, BOX-0000158 (Sheet-0015350.tif), done, but in differing structure; bogabag_00132-00164)
• [x] ├── 2022_10_11 (bogabag_00164-00205)
• [x] ├── 2022_10_12 (bogabag_00205-00238)
• [x] ├── 2022_10_13 (bogabag_00238-00272)
• [x] ├── 2022_10_14 (bogabag_00272-00306)
• [x] ├── 2022_10_17 (bogabag_00306-00336)
• [x] ├── 2022_10_18 (bogabag_00336-00366)
• [x] ├── 2022_10_19 (bogabag_00366-00440)
• [x] ├── 2022_10_20 (bogabag_00440-00445)
• [x] ├── 2022_10_21 (bogabag_00445-00464)
• [x] ├── 2022_10_24 (bogabag_00464-00497)
• [x] ├── 2022_10_25 (bogabag_00497-00536)
• [x] └── 2022_10_26 (bogabag_00536-00568)

IIIF ingest

• [x] bogabag_0000 (990 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0001 (835 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0002 (999 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0003 (944 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0004 (984 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0005 (999 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0006 (994 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0007 (995 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0008 (994 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0009 (990 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0010 (996 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0011 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [ ] bogabag_0012
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0013
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0014
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0015
  • [x] ready on delivery --> permit overwriting of existing files when launching import task
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0016
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0017
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0018
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0019
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0020
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0021
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0022
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0023
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0024
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [ ] bogabag_0025
  • [x] ready on delivery
  • [x] iiif task successful
  • [ ] deleted from _processed
• [x] bogabag_0026 (995 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0027 (998 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0028 (998 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0029 (983 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0030 (998 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0031 (999 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0032 (994 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0033 (982 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0034 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0035 (993 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0036 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0037 (999 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0038 (984 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0039 (926 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0040 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0041 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0042 (995 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0043 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0044 (999 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0045 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0046 (997 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0047 (993 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0048 (995 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0049 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0050 (999 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0051 (999 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0052 (999 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0053 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0054 (1000 tif files locally and remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0055 (999 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed
• [x] bogabag_0056 (874 tif files; diff local remote :white_check_mark: )
  • [x] ready on delivery
  • [x] iiif task successful
  • [x] deleted from _processed

[pdaengeli]

The IIIF server isnn't coping well with the bags created so far, likely due to their size.

size

``` su -dh bogabag_00536-00568 525G bogabag_00536-00568 su -dh bogabag_00497-00536 632G bogabag_00497-00536 ```

Thus adjusting the strategy:

• initial preparation steps based on day-dated Picturae folders
• bags based on max 1000 sheets (e.g. bogabag_0056 containing Sheet-0056000 through Sheet-0056877)

The first try was successful and took ca 4 hours for just the IIIF ingest process (bogabag_0056)

[pdaengeli]

• [ ] get counts per sheet group on resstore (find . -name 'Sheet-0056*.tif' | wc -l) and compare completeness of the ingest (delivery counts are given above) NB: needs to run over all files as there is a structural mismatch Edit: doing this as diffs of file lists as it is more reliable/informative than counts

commands

``` # list remote files ssh dh@130.92.252.28 find /tmp/boga_herbarium_bernense/Picturae -name \'Sheet-0027*.tif\' -printf \'%f\\n\' | sort > comparison/tif-list-0027-remote.txt # list delivery files find ../_processed/bogabag_0028/data/herb-specs/ -name 'Sheet-0027*.tif' -printf '%f\n' | sort > comparison/tif-list-0028-local.txt # diff diff comparison/tif-list-0028-remote.txt comparison/tif-list-0028-local.txt > comparison/comparison-0028.txt # check diff cat comparison/comparison-0028.txt # number of remote files cat comparison/tif-list-0028-remote.txt | wc -l ```

[pdaengeli]

Completeness check

• get list of resstore files:

  ssh dh@130.92.252.28 find /tmp/boga_herbarium_bernense/Picturae -name  \'Sheet-00*.tif\' -printf \'%f\\n\' | sort > comparison/tif-list-all-remote.txt

• get metadata from IIIF server:

• sort column G and paste contents to comparison-tif-list-all-local.txt

• diff comparison/tif-list-all-remote.txt comparison/tif-list-all-local.txt > comparison/comparison-all.txt

Result

A clean diff :)

Thus skipping remaining tests and closing here.

---

alternative approach

iterate over remote list and query http status with a command such as ``` curl -s -o /dev/null -I -w "%{http_code}" https://iiif.ub.unibe.ch/image/v3/boga/Sheet-0000001.tif/info.json ```