Create ISO19115-3 reader for MDTranslator

[btylerburton]

User Story

In order to transform an ISO-19115 source to DCAT-US, datagovteam needs to work with the MDTranslator team to implement an ISO-19115 reader for the MDTranslator.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• [ ] GIVEN that the work to implement an ISO-19115 reader has been performed WHEN I provide a valid source to the MDTranslator THEN I expect to receive a valid DCAT-US transformation back from the service

ISO to DCATUS Progress

(source)

• [x] Title
• [x] Description
• [x] Keyword
• [x] Modified
• [x] Publisher
• [x] ContactPoint
• [x] AccessLevel
• [x] Identifier
• [x] Distribution
• [x] Rights
• [x] Spatial
• [x] Temporal
• [x] License
• [x] Issued
• [x] Language
• [x] DescribedBy
• [x] IsPartOf
• [x] Theme
• [x] References
• [x] LandingPage
• [x] SystemOfRecords
• [x] DescribedByType
• [x] AccrualPeriodicity
• [x] PrimaryITInvestmentUII

program code and bureau code aren't required in the non-federal version of DCATUS. there's also no mapping from iso to this so we're skipping them.

• [ ] ProgramCode
• [ ] BureauCode

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

MDTranslator already has an ISO writer, so the work of implementing a reader will involve porting that transformer from writer to reader, applying adequate testing, and then confirming that it works by testing it on a live source.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch ( deployment )

• [ ] Implement an ISO-19115 reader in the MDTranslator repo
• [ ] integrate feature into transformation rails app
• [ ] Build the new rails app and deploy it on cloud.gov
• [ ] Validate that it works by passing it an ISO-19115 source and confirming you receive DCAT-US JSON back.

Sketch ( feature merge with upstream )

• [x] lint/formatting
  • [x] update rubocop rules as necessary to conform to upstream/existing codebase
  • [x] apply rules, fix issues, disable rules when necessary
• [ ] anything else?....

Sketch ( feature high-level )

• [ ] obtain/create iso19115-3 xml containing all possible information acceptable by that schema
• [ ] run test against this xml checking all information is present in mdjson ( internal ) object then write to iso19115-3 xml. the output xml should be identical to the input ( no transformation is needed )

Sketch ( feature low-level )

• a way to partition this ticket out is via the properties of the mdtranslator internal object used in the iso 19115-3 writer. the idea is to populate these properties with information from the input iso 19115-3 file
• each property generally represents some unit of work and would be merged into the feature branch
• this list is subject to change as more work presents itself

Resources

• md viewer

What mdjson/internal object properties need to be populated?

• [x] :schema
• [ ] :contacts
• [ ] :metadata
  • [ ] :metadataInfo
  • [x] :metadataIdentifier
  • [x] :parentMetadata
  • [x] :defaultMetadataLocale
  • [x] :otherMetadataLocales
  • [ ] :metadataContacts
    • [x] :roleName
    • [ ] :roleExtents
    • [ ] :description
    • [ ] :geographicExtents
      • [ ] :description
      • [ ] :containsData
      • [ ] :identifier
      • [ ] :boundingBox
      • [ ] :geographicElements
      • [ ] :nativeGeoJson
      • [ ] :computedBbox
    • [ ] :temporalExtents
    • [ ] :verticalExtents
    • [ ] :parties
  • [ ] :metadataDates
  • [ ] :metadataLinkages
  • [ ] :metadataConstraints
  • [ ] :metadataMaintenance
  • [ ] :alternateMetadataReferences
  • [ ] :metadataStatus
  • [ ] :extensions
  • [ ] :resourceInfo
  • [ ] :lineageInfo
  • [ ] :distributorInfo
  • [ ] :associatedResources
  • [ ] :additionalDocuments
  • [ ] :funding
  • [ ] :dataQuality
• [ ] :dataDictionaries
• [ ] :metadataRepositories

anything less and we risk being confused by data not being transformed simply because it's not accounted for

[rshewitt]

i'd like to do this work if it's cool with the team

[rshewitt]

dcatus writer branch remains as a draft pr with 185 commits and 248 file changes going to develop. the latest release branch (2.19.0) was merged into the dcatus writer branch 3 weeks ago.

[rshewitt]

iso_19115_3 writer

[rshewitt]

the mdtranslate transformation process involves 3 schema's.

• the input file/schema ( 1 )
• the mdtranslator internal schema ( 2 )
• the destination schema ( 3 )

since there's an existing iso 19115-3 writer that means schema 2 -> 3 exists in some fashion. what we're trying to do is get from schema 1 -> 2. i wouldn't be surprised if it's not that simple though and more work may need to be done on 2 -> 3

[rshewitt]

iso reader first commit. I successfully ran a transformation using

bundle exec mdtranslator translate [FILE] -r=iso19115_3 -w=dcat_us

where [FILE] = xml. linking on gdrive because github doesn't support .xml attachments in comments apparently.

there's no information actually being transformed so that needs to be done. the only thing that it does is read the xml and prepare the internal metadata object ( mentioned in my previous comment as schema 2 ) with enough information for the process to complete.

[rshewitt]

existing work on iso 19115-3 reader

[rshewitt]

1410 files inspected, 15857 offenses detected, 2618 offenses autocorrectable

there's too many things to fix when linting the entire repo. i'm not great at bulk text editing either so i'm just going to apply the linting to our addition(s)

[rshewitt]

schema resources

• https://github.com/ISO-TC211/XML/tree/master/schemas.isotc211.org/19115/-3
• https://schemas.isotc211.org/19115/

[rshewitt]

mdtranslator has processing flags in the output. each of these have a pass and message flag ( e.g. readerStructurePass & readerStructureMessags )

• readerStructure ( log level = error )
  • Metadata input file was not provided
  • Reader was not specified
  • XML file is not well formed
  • mdJson reader: schema definition is missing
• readerValidation ( log level = error )
  • FGDC file did not contain a tag
  • Requested reader [reader] cannot be found.
• readerExecution ( log level = warning & error. mostly warning )
  • FGDC reader: Conversion of dateTime string to object failed
• writer ( log level = warning, error, or notice )
  • Requested writer [writer] cannot be found.
  • date is missing

[rshewitt]

discussed previous prs ( 07/03/24 )

[rshewitt]

discussed previous prs ( 07/17/24 )

[Bagesary]

Refactoring is in progress.

[rshewitt]

a rule i'm implementing in my next minor refactor is explicitly indicating whether something is optional or required and providing the schema element as evidence. for example,

# :title (required)
# <element name="title" type="gco:CharacterString_PropertyType">
title = xCitation.xpath(@@titleXPath)[0]
if title.nil?
   msg = 'WARNING: ISO19115-3 reader: element \'cit:title\' '\
      'is missing in \'cit:CI_Citation\''
   hResponseObj[:readerExecutionMessages] << msg
   hResponseObj[:readerExecutionPass] = false
   return nil
end

cit:title is a required element within the cit:CI_Citation indicated by the schema element provided as a comment. when an xml element doesn't indicate a minimum number of occurrences (e.g. minOccurs="0") the default is 1 (source) meaning it's required. providing the schema element as a comment provides evidence as to why something is optional/required rather than just assuming it's correct.

[rshewitt]

we're prioritizing the information dcat needs instead of making the iso reader feature complete. this means the iso19115-3 reader will store information in the mbJson according to how the dcatus writer wants it. it's not considering how other writers want the data to be so this feature is rigid but fits our needs. @Bagesary

[rshewitt]

here's the exhaustive list of "sections" (i.e. modules) for the dcatus writer. there's 29. this doesn't reflect anything within the files themselves.

• [ ] dcat_us_access_level.rb

  • mapping: mco:MD_SecurityConstraints & mco:MD_LegalConstraints -> access level
  • dcatus expects the necessary data to be at this location mbJSON[:metadata][:resourceInfo][:constraints]
  • that hash contains either legal or security constraints which determine whether the access level is public, non-public, or restricted public.
  • writing to iso19115-3, the constraint data is expected to be at this location mbJSON[:metadata][:metadataInfo][:metadataConstraints]. it's not in resourceInfo but metadataInfo. this suggests a duplication of data in different places to ensure writes work correctly.
  • PR

• [ ] dcat_us_access_url.rb

  • mapping: cit:CI_OnlineResource

• [ ] dcat_us_accrualPeriodicity.rb

• [ ] dcat_us_bureau_code.rb

• [ ] dcat_us_contact_point.rb

• [ ] dcat_us_dcat_us.rb

• [ ] dcat_us_described_by.rb

• [ ] dcat_us_described_by_type.rb

• [ ] dcat_us_description.rb

• [ ] dcat_us_distribution.rb

• [ ] dcat_us_download_url.rb

• [ ] dcat_us_identifier.rb

• [ ] dcat_us_is_part_of.rb

• [ ] dcat_us_issued.rb

• [ ] dcat_us_keyword.rb

• [ ] dcat_us_landing_page.rb

• [ ] dcat_us_language.rb

• [ ] dcat_us_license.rb

• [ ] dcat_us_media_type.rb

• [ ] dcat_us_modified.rb

• [ ] dcat_us_primaryITInvestmentUII.rb

• [ ] dcat_us_program_code.rb

• [ ] dcat_us_publisher.rb

• [ ] dcat_us_references.rb

• [ ] dcat_us_rights.rb

• [ ] dcat_us_spatial.rb

• [ ] dcat_us_system_of_records.rb

• [ ] dcat_us_temporal.rb

• [ ] dcat_us_theme.rb

[Bagesary]

@rshewitt to bring Chris and Jonathan up to speed on the roadblocks

[rshewitt]

some refactoring that might be needed as we further test the iso reader include:

• updating the return value of the unpack functions to the associated internal metadata object rather than nil. this is in response to my conversation with johnathan.
• update xml paths to point to the correct information.

• removing the initial check for the metadata block the unpack functions are responsible for parsing. here's an example of what that looks like. This is a pattern followed in many of the modules.

  # ...
  def self.unpack(args)
    # MD Medium (required)
    xMedium = xParent.xpath(@@mediumXPath)[0]
  
    if xMedium.nil?
       msg = "WARNING: ISO19115-3 reader: element \'mrd:MD_Medium\' '\
          'is missing in #{xParent.name}"
       hResponseObj[:readerExecutionMessages] << msg
       hResponseObj[:readerExecutionPass] = false
       return nil
    end
  # ...

all of these are straight forward and not time consuming to resolve

[rshewitt]

LandingPage, SystemOfRecords, DescribedByType, AccrualPeriodicity, & PrimaryITInvestmentUII are done in my recent work (haven't pushed to remote) which puts us at 80% complete. the remaining properties require further investigation into how contacts are managed.

:contacts is a root level property in the internal object but contact information in the iso document occurs as either an individual or organization which falls within a responsible party which the mdb:contact element is. so when processing individual/organization elements the contact information within it needs to bubble up to the root level. there's helper functions which deal with contacts.

[rshewitt]

there's no evaluation of required fields in the dcatus writer. since every writer has that kind of functionality i would assume that work still need to be done.

[rshewitt]

test of full translation of iso19115-3 to dcatus!

[Bagesary]

Reid will review this ticket to check for outstanding work and create appropriate new tickets. This ticket must be linked to all the children tickets

[Bagesary]

@rshewitt will check with @jbrown-xentity if all the outstanding tickets have been created and included in the Harvestor document. Then this ticket is good to close.

[Bagesary]

Harvestor document : https://docs.google.com/document/d/1RMHSXSTm2ndr_jAEDHsoT1aRV6Ig4Oq2vzT4Uj6NOxk/edit#heading=h.78eh6qtk0voh