search

pod4lib / aggregator

POD Aggregator, f.k.a. the POD Data Lake
https://pod.stanford.edu
Apache License 2.0
9 stars 3 forks source link

MARC::XMLWriter will generate invalid XML documents. #882

Open corylown opened 2 years ago

corylown commented 2 years ago

MARC::XMLWriter is used by MarcRecordWriterService and OaiMarcRecordWriterService to generate MARC XML documents from contributor supplied MARC records. The supplied MARC records can include invalid byte sequences, non-printing control characters, and other anomalies that are invalid in XML.

This ticket is a placeholder for determining what, if anything, we ought to do about this.

Interestingly, because we're using Nokogiri to produce OAI-PMH responses from documents containing XML record snippets, the OAI feed is valid XML because Nokogiri appears to remove data that would be invalid in XML. However, MARC XML formatted normalized full dumps and deltas served via the web app and ResourceSync may be invalid.

JohnMarkOckerbloom commented 2 years ago

Do we know how Nokogiri handles the invalid data? (Do they just remove invalid characters, or do they do something more than that?) Might not hurt for us to do the same thing when normalizing, if that's easily done; it'd be a straightforward way to bring the different MARC XML views into sync.

corylown commented 2 years ago

In the case I observed Nokogiri removed the invalid bytes.