search

LD4P / sinopia_exporter

A simple CLI tool for exporting RDF from a Sinopia group
Apache License 2.0
0 stars 0 forks source link

exporter chokes on some resources #49

Closed jmartin-sul closed 4 years ago

jmartin-sul commented 4 years ago

for example:

$ ./bin/export ucdavis
sinopia_exporter v0.3.0
beginning export of RDF from group: ucdavis
Sinopia Server client lazily instantiated.  base URL: https://trellis.sinopia.io
error saving resource ucdavis/afbe09ad-e05e-4463-91ac-3607a54b1800 to ./exported_rdf/ucdavis_2019-10-28T23:23:05.213Z : TypeError: data.map is not a function
    at Function.exports.convertToType (/Users/user/software_dev_projects/sinopia_exporter/node_modules/sinopia_server/src/ApiClient.js:531:23)
    at Function.exports.constructFromObject (/Users/user/software_dev_projects/sinopia_exporter/node_modules/sinopia_server/src/model/SinopiaResource.js:75:34)
    at Function.exports.convertToType (/Users/user/software_dev_projects/sinopia_exporter/node_modules/sinopia_server/src/ApiClient.js:527:23)
    at exports.deserialize (/Users/user/software_dev_projects/sinopia_exporter/node_modules/sinopia_server/src/ApiClient.js:350:20)
    at /Users/user/software_dev_projects/sinopia_exporter/node_modules/sinopia_server/src/ApiClient.js:473:30
    at Request.callback (/Users/user/software_dev_projects/sinopia_exporter/node_modules/superagent/lib/node/index.js:814:12)
    at /Users/user/software_dev_projects/sinopia_exporter/node_modules/superagent/lib/node/index.js:1036:18
    at IncomingMessage.<anonymous> (/Users/user/software_dev_projects/sinopia_exporter/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at Stream.emit (events.js:210:5)
    at Stream.EventEmitter.emit (domain.js:476:20)
completed export of ucdavis at 2019-10-28T23:23:05.500Z
finished export of RDF from group: ucdavis

the resource from the above error: https://trellis.sinopia.io/repository/ucdavis/afbe09ad-e05e-4463-91ac-3607a54b1800

$ curl -H 'Accept: application/ld+json' https://trellis.sinopia.io/repository/ucdavis/afbe09ad-e05e-4463-91ac-3607a54b1800 | jq
{
  "@id": "",
  "@type": "schema:Thing",
  "description": {
    "@language": "en",
    "@value": "Thing of Things"
  },
  "identifier": {
    "@language": "en",
    "@value": "ark:/86290/d7654e"
  },
  "hasResourceTemplate": "sinopia:resourceTemplate:schema:Thing",
  "@context": {
    "description": {
      "@id": "http://schema.org/description"
    },
    "identifier": {
      "@id": "http://schema.org/identifier"
    },
    "hasResourceTemplate": {
      "@id": "http://sinopia.io/vocabulary/hasResourceTemplate"
    },
    "schema": "http://schema.org/",
    "owl": "http://www.w3.org/2002/07/owl#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "skos": "http://www.w3.org/2004/02/skos/core#",
    "memento": "http://mementoweb.org/ns#",
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "acl": "http://www.w3.org/ns/auth/acl#",
    "geo": "http://www.w3.org/2003/01/geo/wgs84_pos#",
    "dc11": "http://purl.org/dc/elements/1.1/",
    "as": "https://www.w3.org/ns/activitystreams#",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "ldp": "http://www.w3.org/ns/ldp#",
    "time": "http://www.w3.org/2006/time#",
    "prov": "http://www.w3.org/ns/prov#",
    "dc": "http://purl.org/dc/terms/"
  }
}
justinlittman commented 4 years ago

The resources that succeeded had an @graph, as in:

{
    "@graph": [{
        "@id": "",
        "@type": "http://id.loc.gov/ontologies/bibframe/Instance",
        "adminMetadata": "_:b1",
        "carrier": "http://id.loc.gov/vocabulary/carriers/nc",
        "date": {
            "@language": "en",
            "@value": "2012"
        },
        "dimensions": {
            "@language": "en",
            "@value": "23 cm"
        },
        "editionStatement": {
            "@language": "ko",
            "@value": "1판"
        },
        "extent": "_:b3",
        "identifiedBy": ["_:b0", "_:b2", "_:b5", "_:b6"],
        "instanceOf": "_:b9",
        "issuance": "http://id.loc.gov/vocabulary/issuance/mono",
        "media": "http://id.loc.gov/vocabulary/mediaTypes/n",
        "provisionActivity": "_:b7",
        "provisionActivityStatement": {
            "@language": "ko",
            "@value": "서울시 : 소와당"
        },
        "responsibilityStatement": {
            "@language": "ko",
            "@value": "李 成美 지음"
        },
        "title": ["_:b4", "_:b8"],
        "hasResourceTemplate": "ld4p:RT:bf2:Monograph:Instance"
    }, {
        "@id": "_:b0",
        "@type": "http://id.loc.gov/ontologies/bibframe/Lccn",
        "value": {
            "@language": "en",
            "@value": "2012389510"
        }
    }, {
        "@id": "_:b1",
        "@type": "http://id.loc.gov/ontologies/bibframe/AdminMetadata",
        "encodingLevel": "http://id.loc.gov/vocabulary/menclvl/f",
        "creationDate": {
            "@language": "en",
            "@value": "2019-10-16"
        },
        "descriptionAuthentication": "http://id.loc.gov/vocabulary/marcauthen/lccopycat-nm",
        "descriptionConventions": "http://id.loc.gov/vocabulary/descriptionConventions/rda",
        "descriptionLanguage": "http://id.loc.gov/vocabulary/languages/eng"
    }, {
        "@id": "_:b10",
        "@type": "http://id.loc.gov/ontologies/bibframe/Title",
        "mainTitle": {
            "@language": "en",
            "@value": "Ŏjin ŭigwe wa misulsa"
        }
    }, {
        "@id": "_:b11",
        "@type": "http://id.loc.gov/ontologies/bibframe/VariantTitle",
        "mainTitle": {
            "@language": "ko",
            "@value": "어진 의궤 와 미술사"
        }
    }, {
        "@id": "_:b12",
        "@type": "http://id.loc.gov/ontologies/bibframe/SupplementaryContent",
        "label": {
            "@language": "en",
            "@value": "Includes bibliographical references (pages 357-368) and index"
        }
    }, {
        "@id": "_:b13",
        "@type": "Agent",
        "Agent": "Sowadang"
    }, {
        "@id": "_:b14",
        "@type": "http://id.loc.gov/ontologies/bibframe/Source",
        "label": {
            "@language": "en",
            "@value": "OCLC"
        }
    }, {
        "@id": "_:b15",
        "@type": "http://id.loc.gov/ontologies/bibframe/Source",
        "label": {
            "@language": "en",
            "@value": "Alma"
        }
    }, {
        "@id": "_:b2",
        "@type": "http://id.loc.gov/ontologies/bibframe/Local",
        "source": "_:b15",
        "value": {
            "@language": "en",
            "@value": "990036657290403126"
        }
    }, {
        "@id": "_:b3",
        "@type": "http://id.loc.gov/ontologies/bibframe/Extent",
        "label": {
            "@language": "en",
            "@value": "407 pages"
        }
    }, {
        "@id": "_:b4",
        "@type": "http://id.loc.gov/ontologies/bibframe/ParallelTitle",
        "mainTitle": {
            "@language": "en",
            "@value": "Ŏjin ŭigwe wa misulsa"
        },
        "subtitle": {
            "@language": "en",
            "@value": "Chosŏn kugwang ch'osanghwa ŭi chejak kwa mosa"
        }
    }, {
        "@id": "_:b5",
        "@type": "http://id.loc.gov/ontologies/bibframe/Local",
        "source": "_:b14",
        "value": {
            "@language": "en",
            "@value": "776115565"
        }
    }, {
        "@id": "_:b6",
        "@type": "http://id.loc.gov/ontologies/bibframe/Isbn",
        "value": {
            "@language": "en",
            "@value": "9788993820676"
        }
    }, {
        "@id": "_:b7",
        "@type": "http://id.loc.gov/ontologies/bibframe/Publication",
        "agent": "_:b13",
        "date": {
            "@language": "en",
            "@value": "2012"
        },
        "place": "http://id.loc.gov/authorities/names/n79066627"
    }, {
        "@id": "_:b8",
        "@type": "http://id.loc.gov/ontologies/bibframe/Title",
        "mainTitle": {
            "@language": "ko",
            "@value": "어진 의궤 와 미술사"
        },
        "subtitle": {
            "@language": "ko",
            "@value": "조선 국왕 초상화 의 제작 과 모사"
        }
    }, {
        "@id": "_:b9",
        "@type": "http://id.loc.gov/ontologies/bibframe/Work",
        "colorContent": "http://id.loc.gov/vocabulary/mcolor/mul",
        "content": "http://id.loc.gov/vocabulary/contentTypes/txt",
        "geographicCoverage": "http://id.loc.gov/authorities/names/n79109033",
        "illustrativeContent": "http://id.loc.gov/vocabulary/millus/ill",
        "language": "http://id.loc.gov/vocabulary/languages/kor",
        "originDate": {
            "@language": "en",
            "@value": "2012"
        },
        "originPlace": "http://id.loc.gov/authorities/names/n79066627",
        "subject": ["http://id.loc.gov/authorities/subjects/sh85105117", "http://id.loc.gov/authorities/subjects/sh85072427", "http://id.loc.gov/authorities/subjects/sh2008000386", "http://id.loc.gov/authorities/subjects/sh85096774", "http://id.loc.gov/authorities/subjects/sh85033585"],
        "supplementaryContent": "_:b12",
        "temporalCoverage": {
            "@language": "en",
            "@value": "Chosŏn dynasty, 1392-1910"
        },
        "title": ["_:b10", "_:b11"]
    }],
    "@context": {
        "carrier": {
            "@id": "http://id.loc.gov/ontologies/bibframe/carrier",
            "@type": "@id"
        },
        "identifiedBy": {
            "@id": "http://id.loc.gov/ontologies/bibframe/identifiedBy",
            "@type": "@id"
        },
        "adminMetadata": {
            "@id": "http://id.loc.gov/ontologies/bibframe/adminMetadata",
            "@type": "@id"
        },
        "responsibilityStatement": {
            "@id": "http://id.loc.gov/ontologies/bibframe/responsibilityStatement"
        },
        "editionStatement": {
            "@id": "http://id.loc.gov/ontologies/bibframe/editionStatement"
        },
        "issuance": {
            "@id": "http://id.loc.gov/ontologies/bibframe/issuance",
            "@type": "@id"
        },
        "date": {
            "@id": "http://id.loc.gov/ontologies/bibframe/date"
        },
        "extent": {
            "@id": "http://id.loc.gov/ontologies/bibframe/extent",
            "@type": "@id"
        },
        "provisionActivityStatement": {
            "@id": "http://id.loc.gov/ontologies/bibframe/provisionActivityStatement"
        },
        "title": {
            "@id": "http://id.loc.gov/ontologies/bibframe/title",
            "@type": "@id"
        },
        "dimensions": {
            "@id": "http://id.loc.gov/ontologies/bibframe/dimensions"
        },
        "provisionActivity": {
            "@id": "http://id.loc.gov/ontologies/bibframe/provisionActivity",
            "@type": "@id"
        },
        "media": {
            "@id": "http://id.loc.gov/ontologies/bibframe/media",
            "@type": "@id"
        },
        "hasResourceTemplate": {
            "@id": "http://sinopia.io/vocabulary/hasResourceTemplate"
        },
        "instanceOf": {
            "@id": "http://id.loc.gov/ontologies/bibframe/instanceOf",
            "@type": "@id"
        },
        "colorContent": {
            "@id": "http://id.loc.gov/ontologies/bibframe/colorContent",
            "@type": "@id"
        },
        "content": {
            "@id": "http://id.loc.gov/ontologies/bibframe/content",
            "@type": "@id"
        },
        "originPlace": {
            "@id": "http://id.loc.gov/ontologies/bibframe/originPlace",
            "@type": "@id"
        },
        "subject": {
            "@id": "http://id.loc.gov/ontologies/bibframe/subject",
            "@type": "@id"
        },
        "illustrativeContent": {
            "@id": "http://id.loc.gov/ontologies/bibframe/illustrativeContent",
            "@type": "@id"
        },
        "temporalCoverage": {
            "@id": "http://id.loc.gov/ontologies/bibframe/temporalCoverage"
        },
        "language": {
            "@id": "http://id.loc.gov/ontologies/bibframe/language",
            "@type": "@id"
        },
        "originDate": {
            "@id": "http://id.loc.gov/ontologies/bibframe/originDate"
        },
        "supplementaryContent": {
            "@id": "http://id.loc.gov/ontologies/bibframe/supplementaryContent",
            "@type": "@id"
        },
        "geographicCoverage": {
            "@id": "http://id.loc.gov/ontologies/bibframe/geographicCoverage",
            "@type": "@id"
        },
        "mainTitle": {
            "@id": "http://id.loc.gov/ontologies/bibframe/mainTitle"
        },
        "subtitle": {
            "@id": "http://id.loc.gov/ontologies/bibframe/subtitle"
        },
        "value": {
            "@id": "http://www.w3.org/1999/02/22-rdf-syntax-ns#value"
        },
        "place": {
            "@id": "http://id.loc.gov/ontologies/bibframe/place",
            "@type": "@id"
        },
        "agent": {
            "@id": "http://id.loc.gov/ontologies/bibframe/agent",
            "@type": "@id"
        },
        "label": {
            "@id": "http://www.w3.org/2000/01/rdf-schema#label"
        },
        "descriptionLanguage": {
            "@id": "http://id.loc.gov/ontologies/bibframe/descriptionLanguage",
            "@type": "@id"
        },
        "descriptionConventions": {
            "@id": "http://id.loc.gov/ontologies/bibframe/descriptionConventions",
            "@type": "@id"
        },
        "descriptionAuthentication": {
            "@id": "http://id.loc.gov/ontologies/bibframe/descriptionAuthentication",
            "@type": "@id"
        },
        "creationDate": {
            "@id": "http://id.loc.gov/ontologies/bibframe/creationDate"
        },
        "encodingLevel": {
            "@id": "http://id.loc.gov/ontologies/bflc/encodingLevel",
            "@type": "@id"
        },
        "Agent": {
            "@id": "http://id.loc.gov/ontologies/bibframe/Agent"
        },
        "source": {
            "@id": "http://id.loc.gov/ontologies/bibframe/source",
            "@type": "@id"
        },
        "schema": "http://schema.org/",
        "owl": "http://www.w3.org/2002/07/owl#",
        "xsd": "http://www.w3.org/2001/XMLSchema#",
        "skos": "http://www.w3.org/2004/02/skos/core#",
        "memento": "http://mementoweb.org/ns#",
        "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
        "acl": "http://www.w3.org/ns/auth/acl#",
        "geo": "http://www.w3.org/2003/01/geo/wgs84_pos#",
        "dc11": "http://purl.org/dc/elements/1.1/",
        "as": "https://www.w3.org/ns/activitystreams#",
        "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
        "ldp": "http://www.w3.org/ns/ldp#",
        "time": "http://www.w3.org/2006/time#",
        "prov": "http://www.w3.org/ns/prov#",
        "dc": "http://purl.org/dc/terms/"
    }
}
justinlittman commented 4 years ago

According to our Swagger, @graph is required for a resource: https://github.com/LD4P/sinopia_server/blob/master/swagger.yaml#L1095

jmartin-sul commented 4 years ago

manually tested by exporting rdf from the ucdavis group, confirmed that all resources downloaded without error (including the one mentioned specifically in this issue).