crkn-rcdr / cihm-metadatabus

Documentation and Docker build environment for key portions of the metadata bus.
BSD 2-Clause "Simplified" License
0 stars 1 forks source link

dmdtask: Storing wrong metadata into Swift. #48

Closed RussellMcOrmond closed 2 years ago

RussellMcOrmond commented 2 years ago

We are trying to diagnose an issue where the wrong metadata is being stored in Swift during large bulk updates.

RussellMcOrmond commented 2 years ago

I ran a tool that looked at the array, and saw what I believe was the correct metadata.

3374d92f3e55b10348c3c13c26a105fd has 23512 items
3374d92f3e55b10348c3c13c26a105fd has 23512 dmd entries
3374d92f3e55b10348c3c13c26a105fd has 23512 flattenentries
Found qmma.10001 at 10000
$VAR1 = {
          'message' => '',
          'shouldStore' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),
          'id' => 'qmma.10001',
          'stored' => $VAR1->{'shouldStore'},
          'label' => 'Esnagami Lake, Ontario',
          'found' => $VAR1->{'shouldStore'},
          'output' => 'issueinfo',
          'parsed' => $VAR1->{'shouldStore'}
        };
$VAR2 = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<issueinfo xmlns=\"http://canadiana.ca/schema/2012/xsd/issueinfo\"><series>NTSSNRC</series><title>Esnagami Lake, Ontario</title><sequence>10001</sequence><language>eng</language><language>fra</language><published>1974</published><pubstatement>Ottawa: Dept. of Energy, Mines and Resources, Surveys and Mapping Branch, 1974</pubstatement><source>McGill University Library and Archives</source><identifier>Maps of the National Topographic System of Canada, 1:50,000\x{a0} ; x042L07</identifier><note>Edition: 1 MCE</note><note>Year Valid: 1962</note><note>Scale: 1:50,000</note><note>Style: Multicoloured</note><note>Universal Transverse Mercator [UTM] grid Zone 16</note><note>https://doi.org/10.5683/SP3/I5L5CL</note><note>http://geo.scholarsportal.info/#r/details/_uri\@=NTS50K042L07_1974ed1mceTIFF&amp;_add:true</note></issueinfo>
";
$VAR3 = {
          'pubmax' => '1974-12-31T23:59:59.999Z',
          'no' => [
                    'Edition: 1 MCE',
                    'Year Valid: 1962',
                    'Scale: 1:50,000',
                    'Style: Multicoloured',
                    'Universal Transverse Mercator [UTM] grid Zone 16',
                    'https://doi.org/10.5683/SP3/I5L5CL',
                    'http://geo.scholarsportal.info/#r/details/_uri@=NTS50K042L07_1974ed1mceTIFF&_add:true'
                  ],
          'pu' => [
                    'Ottawa: Dept. of Energy, Mines and Resources, Surveys and Mapping Branch, 1974'
                  ],
          'identifier' => [
                            'Maps of the National Topographic System of Canada, 1:50,000 ; x042L07'
                          ],
          'lang' => [
                      'eng',
                      'fra'
                    ],
          'ti' => [
                    'Esnagami Lake, Ontario'
                  ],
          'pubmin' => '1974-01-01T00:00:00.000Z',
          'no_source' => [
                           'McGill University Library and Archives'
                         ]
        };

Looked it up: https://access.canadiana.ca/object/edit/69429/m0s756d5v22b

Looked into Swift:

Object: 69429/m0s756d5v22b/dmdISSUEINFO.xml    
 875 bytes  |    Type:  xml  |    Etag: c213eea940b0773e9a6f0616a367d286
 Last modified (in object store): Tue Aug 16 2022 13:53:22 GMT-0400 (EDT)

Downloaded, and it is the wrong XML.

 <?xml version="1.0" encoding="UTF-8"?>
<issueinfo xmlns="http://canadiana.ca/schema/2012/xsd/issueinfo"><series>NTSSNRC</series><title>Toronto Lake, Ontario</title><sequence>9996</sequence><language>eng</language><language>fra</language><published>1970</published><pubstatement>Ottawa: Dept. of Energy, Mines and Resources, Surveys and Mapping Branch, 1970</pubstatement><source>McGill University Library and Archives</source><identifier>Maps of the National Topographic System of Canada, 1:50,000  ; x042L05</identifier><note>Edition: 1 MCE, Provisional</note><note>Year Valid: 1960</note><note>Scale: 1:50,000</note><note>Style: Multicoloured</note><note>Universal Transverse Mercator [UTM] grid Zone 16</note><note>https://doi.org/10.5683/SP3/1N1KZV</note><note>http://geo.scholarsportal.info/#r/details/_uri@=NTS50K042L05_1970ed1mceTIFF&amp;_add:true</note></issueinfo>