IIIF collection import dies halfway through

benwbrum commented 1 year ago

This IIIF collection is failing for JHU: https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/collection

The manifests imported successfully are these:

https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/AB142/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Add12042/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Add31840/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Add42133/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/AlbiRochegude103/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Amiens437/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arras845/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arras897/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal2872/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal2989/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal3336/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal3337/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal3338/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal3339/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal5209/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal5210/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal5226/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal5871/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal6045/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Arsenal6818/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/AssembleeNationale1230/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/AugsburgI423/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/AugsburgI424/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Besancon553/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Valencia387/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Bodmer79/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Chalon-sur-Saone33/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Chalons-en-Champagne270/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/CodPoetPhil6/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/CodGall17/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/CodGall80/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/CoxMacro/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Dijon525/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Dijon526/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Douce195/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Douce332/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Egerton1069/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Egerton2022/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Egerton881/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/ENSBAMas81/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Ferrell/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12587/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12588/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12589/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12590/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12591/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12592/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12593/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12594/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12595/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12596/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais12786/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1462/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais15109/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1558/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1559/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1560/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1561/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1562/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1563/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1564/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1565/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1566/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1567/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1568/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1569/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1570/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1571/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1572/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1573/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1574/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1575/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1576/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais1665/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19137/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19153/manifest
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19154/manifest

Notably, the last work created in the collection does not have an associated sc_manifest, so we must be dying while importing it:

{"id"=>32038375,
 "title"=>"Français 19155",
 "description"=>
  "Bibliothèque nationale de France, fr. 19155Bibliothèque nationale de France, fr. 19155\n" +
  "<br /><br />\n",
 "created_on"=>Wed, 02 Nov 2022 06:24:20 UTC +00:00,
 "owner_user_id"=>32009650,
 "restrict_scribes"=>false,
 "transcription_version"=>14,
 "physical_description"=>nil,
 "document_history"=>nil,
 "permission_description"=>nil,
 "location_of_composition"=>nil,
 "author"=>nil,
 "transcription_conventions"=>nil,
 "collection_id"=>32000369,
 "scribes_can_edit_titles"=>false,
 "supports_translation"=>false,
 "translation_instructions"=>nil,
 "pages_are_meaningful"=>true,
 "ocr_correction"=>false,
 "slug"=>"francais-19155",
 "picture"=>nil,
 "featured_page"=>32690410,
 "identifier"=>nil,
 "next_untranscribed_page_id"=>32690405,
 "original_metadata"=>
  "[{\"label\":\"numberOfIllustrations\",\"value\":\"0\"},{\"label\":\"date\",\"value\":\"15th century\"},{\"label\":\"commonName\",\"value\":\"Français 19155\"},{\"label\":\"website\",\"value\":\"\"},{\"label\":\"origin\",\"value\":\"France\"},{\"label\":\"yearEnd\",\"value\":\"1500\"},{\"label\":\"repository\",\"value\":\"Bibliothèque nationale de France\"},{\"label\":\"title\",\"value\":\"Roman de la Rose\"},{\"label\":\"type\",\"value\":\"manuscript\"},{\"label\":\"currentLocation\",\"value\":\"Paris\"},{\"label\":\"numberOfPages\",\"value\":\"146\"},{\"label\":\"material\",\"value\":\"parchment\"},{\"label\":\"width\",\"value\":\"206\"},{\"label\":\"dimensionUnits\",\"value\":\"mm\"},{\"label\":\"shelfmark\",\"value\":\"fr. 19155\"},{\"label\":\"yearStart\",\"value\":\"1400\"},{\"label\":\"height\",\"value\":\"276\"},{\"label\":\"dimensions\",\"value\":\"206mm x 276mm\"}]",
 "genre"=>nil,
 "source_location"=>nil,
 "source_collection_name"=>nil,
 "source_box_folder"=>nil,
 "in_scope"=>true,
 "editorial_notes"=>nil,
 "document_date"=>nil,
 "uploaded_filename"=>nil,
 "metadata_description"=>nil,
 "metadata_description_version_id"=>nil,
 "description_status"=>"undescribed",
 "searchable_metadata"=>
  "0\n" +
  "\n" +
  "15th century\n" +
  "\n" +
  "Français 19155\n" +
  "\n" +
  "\n" +
  "\n" +
  "France\n" +
  "\n" +
  "1500\n" +
  "\n" +
  "Bibliothèque nationale de France\n" +
  "\n" +
  "Roman de la Rose\n" +
  "\n" +
  "manuscript\n" +
  "\n" +
  "Paris\n" +
  "\n" +
  "146\n" +
  "\n" +
  "parchment\n" +
  "\n" +
  "206\n" +
  "\n" +
  "mm\n" +
  "\n" +
  "fr. 19155\n" +
  "\n" +
  "1400\n" +
  "\n" +
  "276\n" +
  "\n" +
  "206mm x 276mm\n" +
  "\n" +
  "Français 19155\n" +
  "\n" +
  "Bibliothèque nationale de France, fr. 19155Bibliothèque nationale de France, fr. 19155\n" +
  "<br /><br />\n" +
  "\n" +
  "\n" +
  "true",
 "recipient"=>nil}

The last work has 15 pages, the last of which does not have an associated sc_canvas record, so we are likely dying while importing that canvas:

https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/binding.frontcover/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/frontmatter.pastedown/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/frontmatter.flyleaf.01r/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/frontmatter.flyleaf.01v/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/001r/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/001v/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/002r/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/002v/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/003r/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/003v/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/004r/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/004v/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/005r/canvas
https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/005v/canvas

In that case, it looks like we're likely to be failing on the manifest https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/manifest on the sixteenth canvas (presumably named https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/006r/canvas)

Let's look at this canvas:


            {
               "@id":"https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/006r/canvas",
               "@type":"sc:Canvas",
               "label":"6r",
               "thumbnail":{
                  "@id":"https://image.library.jhu.edu/iiif/rose%2fFrancais19155%2fFrancais19155.006r/full/!128,128/0/default.jpg",
                  "service":{
                     "@context":"http://iiif.io/api/image/2/context.json",
                     "@id":"https://image.library.jhu.edu/iiif/rose%2fFrancais19155%2fFrancais19155.006r",
                     "profile":"http://iiif.io/api/image/2/level2.json"
                  }
               },
               "height":3361,
               "width":2447,
               "images":[
                  {
                     "@id":"https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/006r;image/annotation",
                     "@type":"oa:Annotation",
                     "label":"6r",
                     "resource":{
                        "@id":"https://image.library.jhu.edu/iiif/rose%2fFrancais19155%2fFrancais19155.006r/full/full/0/default.jpg",
                        "@type":"dctypes:Image",
                        "format":"image/jpeg",
                        "width":2447,
                        "height":3361,
                        "service":{
                           "@context":"http://iiif.io/api/image/2/context.json",
                           "@id":"https://image.library.jhu.edu/iiif/rose%2fFrancais19155%2fFrancais19155.006r",
                           "profile":"http://iiif.io/api/image/2/level2.json"
                        },
                        "label":"6r"
                     },
                     "motivation":"sc:painting",
                     "on":"https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/006r/canvas"
                  }
               ],
               "otherContent":[
                  {
                     "@id":"https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/006r;all/annotations",
                     "@type":"sc:AnnotationList",
                     "label":"6r"
                  }
               ]
            },

It looks like these canvases already have annotation lists, although they seem to be empty:

{
   "@context":"http://iiif.io/api/presentation/2/context.json",
   "@id":"https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/006r;all/annotations",
   "@type":"sc:AnnotationList",
   "label":"6r annotations",
   "description":"Annotation list for everything on page 6r",
   "resources":[

   ]
}

benwbrum commented 1 year ago

This manifest imports successfully on a local machine. Attempting a collection next.

benwbrum commented 1 year ago

No exception is thrown in the logs on production:

fromthepage@fromthepage:~/deployment/releases/20221025204017/public/imports$ tail 32000369_iiif.log
[74/161] attempting https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19137/manifest
Français 19137 has been imported

[75/161] attempting https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19153/manifest
Français 19153 has been imported

[76/161] attempting https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19154/manifest
Français 19154 has been imported

[77/161] attempting https://jdm.library.jhu.edu/iiif-pres-dlmm/rose/Francais19155/manifest

We did deploy a new release on 2022-11-02 19:53:26 GMT. It seems possible that this could have interfered with the rake import process, as the previous file location used for redirecting the rake task output (symlink from ~/deployment/releases/20221025204017/public/imports/32000369_iiif.log to ~/deployment/current/public/imports/32000369_iiif.log) changed. I'm pretty sure that's not how UNIX filehandles work, however. I'll try moving that file in development during a long-running import (if this process gets past Français 19155 successfully) to test that hypothesis.

benwbrum commented 1 year ago

I've determined the following:

If our rake task encounters an exception, it does appear in the log, so it appears that an exception is not the problem
Moving the location of the log file does not kill the task

Is it possible that something else in the OS is killing the long-running task, several hours in?

benwbrum / fromthepage

IIIF collection import dies halfway through #3381