Create initial delimited import template

[mcritchlow]

Descriptive summary

Create an initial csv import template to be used for initial tool R&D.

Focusing on:

1. hierarchical components
2. linked records
3. CSV vs TSV vs Excel - is there a preferred format that will make life easier for DOMM and Dev?

Deliver:

• An initial template
• An initial set of unit and acceptance tests

Dependency

111

Related work

Ticket migrated from https://github.com/ucsdlib/dams5-cc-pilot/issues/39

[ghost]

@mcritchlow funny you say that, we were just talking about using exactMatch when we are very confident.

For targeting ORCID vs other properties in the display, can we use the base URL, e.g., http://orcid.org ?

[lsitu]

@mcritchlow Yes, I am concerning about it either. I think it'll be nice if the ORCID can have its own predicate so that we can grab it for special handling.

[remerjohnson]

@lsitu @mcritchlow the makers a fabio have an ontology called "scoro": http://www.sparontologies.net/ontologies/scoro

One of the predicates is scoro:hasORCID

We could put our property within Agent as hasORCID or identifier:orcid and point to that predicate?

[ghost]

@lsitu I have some details in the new template to work out, but a draft version for discussion is available at: https://docs.google.com/a/ucsd.edu/spreadsheets/d/1EhMWaDFsBZDhp-BU-5LtOI0tECDx2EtAZ28iZk-Ih58/edit?usp=sharing

Cells in yellow are to be determined, so you can ignore them for now. The header list is vertical so I can show you the definitions and sample values. The final result will be a horizontal headers like the old ingest spreadsheet.

The tab "CV values" contains values allowed in controlled value columns. As in the old ingest spreadsheet, the values are labels that should be converted upon ingest. For example: CC-BY = http://creativecommons.org/licenses/by/4.0/ I have added the matching values for convenience, but you can also find them on the "dams4.2 data dictionary" Google sheet.

The tab "Formatted key-value rules" are the values allowed within { } for single-cell multi-property strings. I don;t think we need individual validation for each { } column (date, relatedResource). I expect that the keys will be applicable any time we use the { } format.

There will be more agent:role columns added. I have to get these from our other documentation.

Let me know if you have questions. If we have agreed on most things, I can turn this into a .xlsx template for your development testing.

[lsitu]

@GregReser The ingest template looks nice. Thank you very much.

• For CVs controls, is it possible to use exact CV without conversions or not?
• For key/value inside {}, Is there a generic way to identify all the key/values pairs or not? For example, a comma, a while space etc. I see you mention the "Formatted key-value rules" and it could be any keys, right?
• For linking of the range ucsd:Agent, what's the look up rules for agent types like Agent:Person, Agent:Group and Agent:Organization, just using the label and ignore all other fields?
• In "Ingest Instructions" for IDs, it looks like the value "convert to: urn:x-public-id:accession:[cell value]" is a little misleading since prefix "urn:x-public-id:accession" could be vary for each identifier type. Maybe using something like "urn:[x-experimental-id]:[URN Prefix]:[cell value]" is better?

[ghost]

@lsitu

• CVs can only use the short label not the conversion
• For key/value inside {} - at this time, the only valid keys are the ones listed in "Formatted key-value rules". key/values pairs are separated by white space. Would it work to consider the white space+key as a reliable separator? Thinking ahead, if we have a white space in a value, will the parsing still work?
• For linking of the range ucsd:Agent - lookup label and alternateLabel only. Is that what you mean?
• I changed the "Ingest Instructions" for ID the correct prefix to indicate public or private and I added the URN Prefix. Is that useful?

[lsitu]

@GregReser :

• CVs can only use the short label not the conversion *I mean values in CVs for validation should be matched what are allowed to be used in the Excel spreadsheet for ingest. Another way that we can do is to retrieve the CVs from the local authority repository. But we may need to define those CVS in label/url pairs and use either the label or the url in the Excel spreadsheet. What do you think?
• For key/value inside {} - at this time, the only valid keys are the ones listed in "Formatted key-value rules". key/values pairs are separated by white space. Would it work to consider the white space+key as a reliable separator? Thinking ahead, if we have a white space in a value, will the parsing still work? *Hmm, lots of concerns. I am worrying about the "description" text that could be anything, and the space+key could be anything that need to validate, right?
• For linking of the range ucsd:Agent - lookup label and alternateLabel only. Is that what you mean? *Yes. I think this is just something that we may need to know for questions like: What is a duplicate record? How those alternateLabel and other fields will be used? Is it possible to have the same label for different type of agents or not?
• I changed the "Ingest Instructions" for ID the correct prefix to indicate public or private and I added the URN Prefix. Is that useful? *Yes, it's clear now and I think it will help others understand it. Thanks.

[lsitu]

@GregReser @remerjohnson For key/value inside {} , maybe we can just define a separator char, something like comma, semicolon etc. that are used less often. If we need to use it in side a "description" text, we can escape the char with \ just like what we are currently doing with pie ( | ). Once we can determine the key/value in pairs, then we can just split them for the key and value. What do you think?

[ghost]

@lsitu

• CVs For simplicity and readability, only the label should be valid in the Excel spreadsheet. You have a method now that says the spreadsheet label for language "eng - English" is valid. On ingest the value "eng" is used in the object build. This is the same idea.
• For key/value inside {} - maybe space+key is not the right way to talk about it. Basically, we have a quasi JSON structure, except we have no quotes around the value and no comma between the key:value pairs. This makes it challenging to parse the contents of {}. Before we say that we have to use a JSON structure, let's see if we can make the simpler form work.
  • Our keys are limited. Right now there are only three valid keys - begin: end: uri: Provided these key substrings are not found in any values within {}, can the keys serve as separators? For example, if the substring "end:" is found, then the substring that follows (until "}" or another valid key) is the value associated with end: As I write this, it seems complicated and may not work. The benefit would be the parsing could be used in future multi property cells.
  • An alternative would be to have parsing rules for each multiple properties column. We know what to expect in each column. Right now we have two columns like this:
  • related resource we already know because we are using it in the old spreadsheet. Just modify the parsing for the use of "@{uri:value}. I mean the value = the substring after "{uri:" and before "}"
  • date will always have the same format @{begin:DateTime end:DateTime} so the separator will always be " end:"
  • Another alternative would be to use JSON. My fear there is that the quotes around values might cause trouble with how quotes are handled elsewhere. For instance, if we use CVS files for ingest. Is this a concern?

[lsitu]

@GregReser Just a couple of little questions regarding the two issues above: For CVs, maybe we just need the instructions to build the values for each field and ingest it into the Fedora. For example, with "eng", how would you like the uri to be?

For key/value inside {} , don't you expect our program to report any wrong keys? I think we can just make it work anyway. But with the simple key/value cases that you have now, if we can say white space is the separator for multiple key/value pairs, then it'll become much more general, thoughts?

[ghost]

@lsitu if we need a reliable separator that could be used for future cases, then I want to use " | ". Whitespace is too generic and common in values. I know we use " | " to indicate multiple values in other cells, but I think the use inside {} is similar: we are separating multiple key:value pairs.

We could use another delimiter, e.g., "; " within {}, but anything we think of might be contained in another string somewhere in the spreadsheet. At least with " | " we are aware of it's use as a separator.

[ghost]

@lsitu For CV values build values

• I have added the full URIs for language.
• copyright jurisdiction uses a literal string and will be a two character code as shown.

Are there any others that need clarification?

[ghost]

@lsitu For linking of the range ucsd:Agent - lookup label and alternateLabel

Your questions are very good and difficult to answer. Here's my first thoughts:

• What is a duplicate record?
  • By running the new authority import with pre-ingest validation (issue 48) we should be able to resolve duplicate records. Duplicates might still slip through though.
  • For the Object ingest Agent matching (this issue)
  • Can the pre-ingest validation report multiple string matches? If so, we could resolve them before final ingest
  • On final ingest, the last match could be used, but the report could indicate which other matches were found.
• How those alternateLabel and other fields will be used?
  • They will be used for search ("Central Library" is the same as "Geisel Library"), possibly, maybe, hopefully
  • They will be used for display and web linking in the UI, possibly, maybe, hopefully
• Is it possible to have the same label for different type of agents or not?
  • If we are going to rely on string matches, labels will have to be unique. We will have to manage this by pre-ingest testing and through periodic authority record analysis (run a report to search for duplicates).

[lsitu]

@GregReser Thanks for clarifying all these. For linking of the range ucsd:Agent - lookup label and alternateLabel *****If we lookup records with label and alternateLabel, then we need both the label and alternateLabel in the spreadsheet, and we'll allow different combination of label and alternateLabel of ucsd:Agent records to be created. This means, the label won't be unique, which will be conflicting with your assumption for the last question below.

• What is a duplicate record?

  • By running the new authority import with pre-ingest validation (issue 48) we should be able to resolve duplicate records. Duplicates might still slip through though. *****I think determine a matched record (or duplicate records) is important as you know we'll exclude duplicate records from created. And if we found any duplicate records for some reason, we need to fix our tool as a bug and clean up the metadata at the meantime.
  • For the Object ingest Agent matching (this issue)
  • Can the pre-ingest validation report multiple string matches? If so, we could resolve them before final ingest *****Yes, we can report multiple string matches in the pre-ingest validation.
  • On final ingest, the last match could be used, but the report could indicate which other matches were found. *****I think we can report it in the ingest log.

• How those alternateLabel and other fields will be used?

  • They will be used for search ("Central Library" is the same as "Geisel Library"), possibly, maybe, hopefully
  • They will be used for display and web linking in the UI, possibly, maybe, hopefully

• Is it possible to have the same label for different type of agents or not?

  • If we are going to rely on string matches, labels will have to be unique. We will have to manage this by pre-ingest testing and through periodic authority record analysis (run a report to search for duplicates). *****If we lookup records with label and alternateLabel, then label won't be unique.

[lsitu]

@GregReser For key/value pairs in {}, I think we cannot use pie ( | ) as the separator since we've used it to delimit multiple values, unless we can clarify either key/value pairs or multiple values will be allow in a cell, but not both. It seems like "; " is a better fit than the space. If anyone don't like it, let's move forward with the origin plan and we'll make it work for now then. Thanks.

[ghost]

@lsitu For key/value pairs in {}, you may be right, but if we use "; " as a separator within {} won't that cause a problem when we have "; " as part of a string in other cells? In other words, we would treat "; " as a separator only some times.

[lsitu]

@GregReser No, the separator for key/value pairs in {} only affects the values inside the {}. It'll be safe to use it anywhere else in the spreadsheet. For example, if we choose to use ";" as the key/value pairs separator, we can still use it intact outside of the {} without escaping in the current cell or any other cells.

[lsitu]

For example, if we choose "," as separator, the following cell values are valid: Jan. 1st, 1925 Jan. 1st, 1925@{begin:1925-01-01, end:1925-08-03} Jan. 1st, 1925@{begin:1925-01-01, end:1925-08-03} | Jan. 1st, 1925

[ghost]

@lsitu I assume you are thinking that a multi property cell could contain more than one value? You want to be able to split the multiples first then parse the key:values inside {} Example date:issued April 8, 1908 @{begin:1908; end:1908} | June 30, 1935 @{begin:1935; end:1935}

OK, let's use "; " Since we are changing this, how about using key=value for readability? : and ; look very similar. I'm taking inspiration for the Dublin Core formatted text string spec: http://dublincore.org/documents/dcmi-dcsv/ Our use is a little different.

Example: April 8, 1908 @{begin=1908; end=1908}

[ghost]

@lsitu I missed your example above showing " | " when I was writing my comment. I had not planned on allowing multiple values in a {} cell, which is why I didn't see a problem. I guess we should allow multiple values, since that is our general rule. So yes, let's use "; " inside {}

[lsitu]

@GregReser Since we are not following the JSON format for key/value pairs in {}, it sounds good to use ";" as key/value pairs separator and and use "=" as key/value delimiter. I think your example looks good: April 8, 1908 @{begin=1908; end=1908}

[ghost]

@lsitu Great, I'll update the template.

[ghost]

@lsitu Matching records by string - Right, we don't want duplicate ARKs for the same Agent, Place, or Concept. It seems that we want to lookup all labels and alternateLabels to make sure there is not a match or possible match. If we only search labels, we might miss a match. The possible matches will require that a person makes the final decision.

In the ingest spreadsheet, I don't think we can include both label and alternateLabel partly because we wouldn't know what the alternates are - we only know what the data provider gave us. If they gave us "Reser, Greg" and there is an ARK with alternateLabel "Reser, Greg" and label "Reser, Gregory" we want the report to say: possible match found with label "Reser, Gregory". These would be flagged for human review. Would it possible to make something like this work?

[lsitu]

@GregReser Maybe we can do something with non-work records pre-validation in Support batch ingest of non-Work records to identify all the records with the same label? Once an authority record with a label and an alternateLabel is identified to be used, then we have to include both the label and alternateLabel in the ingest spreadsheet. If we only include the label in the ingest spreadsheet, we'll never know which record is used for ingest if there are several authority records that have the same label exists.

[ghost]

@lsitu Perhaps handle all the records with the same label or alternateLabel in the non-work (authority) records pre-validation and only match label in the object ingest. This puts the burden on us to do the proper pre-ingest validation and conflict resolution, but it makes the final ingest easier. Adding alternateLabels to the final ingest spreadsheet would be messy, so it would be better to avoid it.

On final ingest, we should get a report of all new ARKs generated (as we do now) and we should add the Class and label that was created. These could be reviewed manually if anything looks wrong.

[lsitu]

@GregReser Yes, I agree. So in the final ingest, only the label will be used in the spread sheet we will lookup and use any authority record that matched the label in the spreadsheet. I think we can extend it to use the @{} syntax to include the known alternateLabel later when it's necessary. Does it sounds good?

[ghost]

@lsitu Sounds good!

[mcritchlow]

@GregReser @lsitu - I know we've shifted focus to other things lately, but I wanted to circle back to this effort and see if it would be possible to articulate what remaining gaps we have for this ticket, whether that work can be broken down into individual tickets, etc. Curious what you both think, thanks.

[ghost]

@mcritchlow I think these are the remaining gaps:

• resolve relatedResource predicate(s) and Class #58
• resolve rights properties
  • do we need copyright status or can it be inferred from rights URI?
• resolve Matt's question about identifier:roger record (should it be more generic, e.g., "identifier:catalog_record"
• add remaining agent predicates (it's a long list and I was waiting because it will make the template harder to read).

@lsitu In the object_import_template, I've been highlighting changes in yellow to make sure they you see them. I suggest you remove the yellow when you have implemented the change. This way we will all know what remains to be updated.

Anything else?

[lsitu]

@GregReser It looks like there could be more issues. See my comments in the attached Excel spreadsheet in https://github.com/ucsdlib/horton/issues/69#issuecomment-291249356. And also, we may need instructions for assembling the title column fields into a big title string, right?

[ghost]

@lsitu good question about title. Are you thinking of how we currently assemble title, part name, part number, subtitle in mads:authoritativeLabel? I don't think we are doing this in D5.

[lsitu]

Yes. When I first see those bunch of title fields. Then I think it'll be just the UI displaying issue that we need to talk about later. Thanks.

[ghost]

You're right, it will be a messy display issue :)

[lsitu]

@GregReser It looks like we can't use property name "visibility" (row #134), which is hold as keyword in Hyrax. The following error is thrown from Hyrax: ArgumentError: visibility is a keyword and not an acceptable property name. /Users/lsitu/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/active-fedora-11.0.0.rc7/lib/active_fedora/attributes.rb:127:in define_active_triple_accessor' /Users/lsitu/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/active-fedora-11.0.0.rc7/lib/active_fedora/attributes.rb:118:inproperty' ...

Do you have any thoughts to avoid it?

[ghost]

@remerjohnson @arwenhutt Should we change visibility and visibility expiration to: rights override rights override expiration

[ghost]

@lsitu Let's go with: rights override rights override expiration

[lsitu]

Sure and it's done. Thanks Greg.

[gamontoya]

Note: Waiting on FileUse modeling

[gamontoya]

Defer for Batch/Import/Export Working Group.