Closed evacougnon closed 1 year ago
Can't replicate in prod. Left note in testrail to check on this next systest
reopening as still happening with portal systest version 4.42.102
Issue still happening
What version of the aodn portal are you using? 4.42.104
Which browser are you using? Chrome
Since there's nothing else on list for tonight and tomorrow Friday 28/7, I start to pick on this one (which was originally allocated to iteration 256. @utas-raymondng
@vietnguyengit Thanks
can't reproduce locally for the particular UUID mentioned
I put a console.log()
into the method at : https://github.com/aodn/aodn-portal/blob/378f5a921b3d646e1d777454b6b21806a32e2562/web-app/js/portal/cart/DownloadPanelItemTemplate.js#L235
here what's I can see (locally).
I believe for some reason, that particular UUID collection object, value of linkFiles
variable is not assigned on systest environment, do you think there could be any conflicts happening when doing portal systest procedure @craigrose ?
Issue still happening @vfisaac
Shows error when trying to download, see screenshot
What version of the aodn portal are you using? Portal v4.42.104
Which browser are you using? Chrome
A diff check shows Geonetwork metadata files of the mentioned record between systest and prod are different (missed those extra links in systest Geonetwork record)
So my conclusion is, this is not an issue to deal with the portal, more likely with the Geonetwork side, possibly a glitch during the harvesting process?
https://www.diffchecker.com/5bEe1mij/
systest on the left - prod on the right
note to myself:
https://github.com/aodn/harvesters/tree/master/workspace/SOOP_BA https://github.com/aodn/chef-private/blob/7bc7103bef2f7725e2b7fde8088e636bc38a50c7/bin/talend-helper.rb#L93C1-L105C4
https://deploy.aodn.org.au/job/cloudformation_deploy/14526/console
TASK [geonetwork3_config : add harvesters and populate metadata catalogue] *****
[0;33mchanged: [localhost] => (item={'name': 'Catalogue Portal for systest', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'Catalogue Portal for systest', 'host': 'https://catalogue-portal.aodn.org.au/geonetwork', 'mefFormatFull': 'false', 'createRemoteCategory': 'true', 'xslfilter': 'transform_19115_3_testing_urls'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 0 0 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
So @atkinsn confirmed that stuff in catalogue stacks is manually created. My theory for this issue now is:
(1) Somehow, details of IMOS - Bio-Acoustic Ships of Opportunity (BA SOOP) sub-facility
metadata record are out of sync between catalogue-portal-systest
and prod catalogue portal.
(2) The deployment of portal systest fetches data from the catalogue portal systest stack, while prod fetches data from the prod catalogue portal, which is why that record on portal systest
doesn't have the additional links as in portal prod
.
To fix this, simply add those 3 missing additional links to the metadata record of IMOS - Bio-Acoustic Ships of Opportunity (BA SOOP) sub-facility
on catalogue-portal-systest
and rebuild/deploy the portal systest.
and again, I strongly believe this is not anything to deal with the portal codebase.
I'll need another confirmation whether it is safe to click the redeploy button for the portal systest at the moment @vfisaac
Those 3 links relate to 3 directly uploaded files. Looks like they are getting stripped out in the harvest process.
Those 3 links relate to 3 directly uploaded files. Looks like they are getting stripped out in the harvest process.
thanks @atkinsn
the Jenkins deploy logs between systest
and prod
of the step called "add harvesters and populate metadata catalogue" (which probably declared here: https://github.com/aodn/cloud-deploy/blob/e5f5aafb906c4e4664f1db068b883c7be5fb8e49/ansible/roles/geonetwork3_config/tasks/manage.yml#L181)
are very different
TASK [geonetwork3_config : add harvesters and populate metadata catalogue] *****
[0;33mchanged: [localhost] => (item={'name': 'Catalogue Portal for systest', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'Catalogue Portal for systest', 'host': 'https://catalogue-portal.aodn.org.au/geonetwork', 'mefFormatFull': 'false', 'createRemoteCategory': 'true', 'xslfilter': 'transform_19115_3_testing_urls'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 0 0 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
TASK [geonetwork3_config : add harvesters and populate metadata catalogue] *****
[0;33mchanged: [localhost] => (item={'name': 'External - AAD', 'harvester_data': {'node': {'@type': 'webdav', 'site': {'name': 'External - AAD', 'url': 'https://data.aad.gov.au/sg/api/waf/aadc-metadata/iso-19115-1-aodn/', 'icon': 'AAD_logo.gif'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active', 'validate': 'false', 'recurse': 'false', 'subtype': 'waf'}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'linkage-updater?pattern=[http://geoserver-123.aodn.org.au&replacement=http://geoserver-portal.aodn.org.au](http://geoserver-123.aodn.org.au%26replacement%3Dhttp//geoserver-portal.aodn.org.au)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': '%Sea Temperature Observing System%'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:AIMS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS: Microdebris contamination in Australian coastal and marine waters', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS: Microdebris contamination in Australian coastal and marine waters', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'linkage-updater?pattern=[http://geoserver-123.aodn.org.au&replacement=http://geoserver-portal.aodn.org.au](http://geoserver-123.aodn.org.au%26replacement%3Dhttp//geoserver-portal.aodn.org.au)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': 'Microdebris contamination in Australian coastal and marine waters'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:IMOS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS: Great Barrier Reef Genomics Database: Seawater Illumina Reads', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS: Great Barrier Reef Genomics Database: Seawater Illumina Reads', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'linkage-updater?pattern=[http://geoserver-123.aodn.org.au&replacement=http://geoserver-portal.aodn.org.au](http://geoserver-123.aodn.org.au%26replacement%3Dhttp//geoserver-portal.aodn.org.au)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': 'Great Barrier Reef Genomics Database: Seawater Illumina Reads'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:IMOS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS: Northern Australia Automated Marine Weather and Oceanographic Stations', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS: Northern Australia Automated Marine Weather and Oceanographic Stations', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'linkage-updater?pattern=[http://geoserver-123.aodn.org.au&replacement=http://geoserver-portal.aodn.org.au](http://geoserver-123.aodn.org.au%26replacement%3Dhttp//geoserver-portal.aodn.org.au)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': '%Northern Australia Automated Marine Weather and Oceanographic Stations%'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:AIMS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS: Great Barrier Reef Marine Monitoring Program for Inshore Water Quality', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS: Great Barrier Reef Marine Monitoring Program for Inshore Water Quality', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'modify-urls', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': 'Great Barrier Reef Marine Monitoring Program for Inshore Water Quality - Vertical Profiles of Conductivity Temperature and Depth (CTD)'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:AIMS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A ADCP Data Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A ADCP Data Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'fe20ec11-ad5b-425f-82dd-6b00f448eb1a'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Catch Operations Data', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Catch Operations Data', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'c836f971-991e-10ec-e043-0200007f2db9'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Current Meter Mooring Data Overview (1983 - 1998)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Current Meter Mooring Data Overview (1983 - 1998)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'dfbb5340-9119-4559-8712-a53630acfbb0'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Marine National Facility Voyage Tracks Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Marine National Facility Voyage Tracks Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'c836a6dc-eba7-28c3-e043-0200007fdbdb'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A CTD Data Overview (1982 - present)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A CTD Data Overview (1982 - present)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'c1229b01-41c7-08bf-e044-00144f7bc0f4'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Hydrology Data Overview (1942 - present)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Hydrology Data Overview (1942 - present)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'c1225108-233d-74e9-e044-00144f7bc0f4'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A RV Franklin XBT Data Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A RV Franklin XBT Data Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': '033a1ddd-b206-09d0-e053-08114f8c3b40'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A RV Investigator XBT Data Overview (2014 onwards)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A RV Investigator XBT Data Overview (2014 onwards)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'dfd2c49d-d469-4691-9d10-ac154fdd4985'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO RV Investigator ISAR/SST Sea Surface Temperature Data Overview (2014 onwards)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO RV Investigator ISAR/SST Sea Surface Temperature Data Overview (2014 onwards)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 0 0 ? * *', 'oneRunOnly': 'false', 'status': 'active', 'validate': 'false'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'bdf91f86-2968-4711-873e-2761383bb207'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A RV Southern Surveyor XBT Data Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A RV Southern Surveyor XBT Data Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active', 'validate': 'false'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'd5a640e7-c695-23e1-e043-08114f8c7b58'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Sediment Sampling Sites Overview (1986 to present)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Sediment Sampling Sites Overview (1986 to present)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': '82fa2ab2-2338-4505-b917-b400406b7c7c'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Wildlife Observations Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Wildlife Observations Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': '11d4b1d8-c03e-4e37-8ecc-b085ca80e7df'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'GA Catalogue MH370 Phase 1 150m Bathymetry datasets', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'GA Catalogue MH370 Phase 1 150m Bathymetry datasets', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'https://ecat.ga.gov.au/geonetwork/srv/eng/csw?service=CSW&request=GetCapabilities&acceptVersions=2.0.2', 'icon': 'GA_logo.gif', 'xslfilter': 'ga', 'outputSchema': 'http://www.isotc211.org/2005/gmd'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': 'd887e71a-71dc-4851-94a9-920f7b7cc7e5'}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'GA Catalogue Australian Marine Sediments (MARS) Database', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'GA Catalogue Australian Marine Sediments (MARS) Database', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'https://ecat.ga.gov.au/geonetwork/srv/eng/csw?service=CSW&request=GetCapabilities&acceptVersions=2.0.2', 'icon': 'GA_logo.gif', 'xslfilter': 'ga', 'outputSchema': 'http://www.isotc211.org/2005/gmd'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': 'a05f7892-eef2-7506-e044-00144fdd4fa6'}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - IMAS: AODN Portal IMAS', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - IMAS: AODN Portal IMAS', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://metadata.imas.utas.edu.au/geonetwork/srv/eng/csw-AODN-Portal-IMAS?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'IMAS_logo.gif', 'xslfilter': 'portal-linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'IMOS Catalogue', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'IMOS Catalogue', 'host': 'https://catalogue-imos.aodn.org.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'modify-urls', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 0 0 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'anyField': '_cat', 'anyValue': 'portal:*'}}, 'ifRecordExistAppendPrivileges': 'true', 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - NIWA - GN3 upgrade pending', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - NIWA - GN3 upgrade pending', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'https://catalogue.aodn.org.au/geonetwork/srv/eng/csw-NIWA?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'NIWA_logo.gif', 'xslfilter': 'replace', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'inactive'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - IMAS: AODN Portal OEH', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - IMAS: AODN Portal OEH', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://metadata.imas.utas.edu.au/geonetwork/srv/eng/csw-AODN-Portal-OEH?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'NSW_GOV_logo.gif', 'xslfilter': 'portal-linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:NSW_GOV'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - IMAS: AODN Portal UWA', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - IMAS: AODN Portal UWA', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://metadata.imas.utas.edu.au/geonetwork/srv/eng/csw-AODN-Portal-UWA?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'UWA_logo.png', 'xslfilter': 'portal-linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
It is obvious that there were a lot more things that happened in the prod
deployment at that particular step. I understand that, however, I doubt if the stuff in Catalogue Portal for systest
(probably declared here: https://github.com/aodn/cloud-deploy/blob/master/sample-config/data_bags/geonetwork3/imos_webapps_geonetwork_harvesters/portal_catalogue_systest.json) works 100% correctly as it should.
Futher investigation notes:
There are 2 json files defining IMOS Catalogue
not sure both of them are used or one is outdated:
xslfilter
to be modify-urls
name
under site
section Catalogue-imos portal records
seems to be not found anywhere in the logs trail above, it has the same xslfilter
filter as Catalogue Portal for systest
which is transform_19115_3_testing_urls
(defined here: https://github.com/aodn/cloud-deploy/blob/master/sample-config/ebprep_conf/templates/geonetwork3/transform_19115_3_testing_urls.xsl.template)The json defining Catalogue Portal for systest
here:
xslfilter
of this one is correct? should it be modify-urls
? it also doesn't have "outputSchema": "http://standards.iso.org/iso/19115/-3/mdb/2.0"
set as in IMOS Catalogue
well, I will run up 2 cloud deploy stacks to compare the changes then draw another conclusion
ahhh, worked to the bottom of this, I noticed systest fetching data from https://catalogue-portal.aodn.org.au
and prod
fetching data from https://catalogue-imos.aodn.org.au/
which the metadata record of the same collection are different from here!
https://catalogue-portal.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82 (you can't find the 3 additional links from this one)
but you should be able to find them here: https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82
final PR to fix this stuff: https://github.com/aodn/cloud-deploy/pull/1336
@vfisaac @Chetnamann @evacougnon heads up, that I updated systest harvester to test the investigation theory above, details below.
The IMOS harvester on systest catalogue has been updated to reflect the configs in prod catalogue, I can't seem to be able wiping out the existing records harvested with the old configs. Tried overwriting UUID but not seem working as well, maybe needs a server restart?
Anyhow, I cloned the available systest harvester and particular filter for `IMOS - Bio-Acoustic Ships of Opportunity (BA SOOP) sub-facility" collection only and allow to create new UUID for the same collection (because there is one that already exists that I can't delete or overwrite - there is now 2 x IMOS - Bio-Acoustic Ships of Opportunity (BA SOOP) sub-facility availabile on catalogue systest :warning: - one with missing additional links (old one), one has everything (new one)).
A new metadata record available here: https://catalogue-systest.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/70172727-3425-4e90-bf4c-07191cb601ac
and it has those 3 additional links.
on the portal systest, you would see those 3 links show up in step 3:
https://portal-systest.aodn.org.au/search?uuid=70172727-3425-4e90-bf4c-07191cb601ac
by the above, I think I can draw a conclusion for the root of this issue...mismatching harvester's configs on prod and systest, in particular...fetching data from different sources.
Try to refresh the catalogue systest deployment and re-harvest the records with updated harvester (wipe out existing records (I can't, maybe someone else can) or destroy portal systest stacks and redeploy them ?), systest metadata records should be harvested by the harvester with the same source as configured for prod catalogue to rectify this issue.
OR
No need to update configs and synchronise the source geonetwork for the systest harvester to match with prod. Simply fix the missing links from here (maybe manually?): https://catalogue-portal.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82
@vietnguyengit have a look at https://github.com/aodn/cloud-deploy/blob/b9dacbe5850c49ae8924fa0b640597f73101c629/sample-config/ebprep_conf/templates/geonetwork3/modify-urls.xsl.template
the problem might be there. this is the only place metadata gets changed
Thanks @craigrose , there is a PR created before the one you wanted a revert to update the configs only, rather than changing from catalogue-portal
to catalogue-imos
https://github.com/aodn/cloud-deploy/pull/1334
It however, doesn't have any effects wanted, 3 additional links still missing.
So I guess, the target to fix this problem is not about catalogue-systest
but in catalogue-portal
, additional links missing there already.
Somehow, someone or something ... made the additional links appear back in the metadata record at: https://catalogue-portal.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82
I didn't make any changes to the IMOS harvester configs
@craigrose @atkinsn
now they disappears again @craigrose
ok to document what has been discussed in Slack,
the additional links not actually appeared and disappeared by magic in catalogue_portal
stack
metadata record: https://catalogue-portal.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82
if logged in under imos_admin
account, the links show up when viewing the metadata record.
but unauthenticated will not see them!
logged results on the left, unauthenticated on the right:
we have tried clearing cache of both GN and local browsers, including deleting the harvested records and reharvesting them, no changes really.
then we looked into the database, and the additional links do exist at database table.
i searched in geonetwork portal 3 database on prod database node, with query:
select * from metadata where uuid='8edf509b-1481-48fd-b9c5-b95b42247f82';
but it's not nice to look at the lengthy text on terminal windows so I extract the contents to a text file with:
COPY(select * from metadata where uuid='8edf509b-1481-48fd-b9c5-b95b42247f82') TO '/tmp/soop_ba_db.txt' WITH CSV HEADER;
attached a copy of what needs to be checked here:
and those 3 links are in the extracted contents, so not caching issue but more likely permissions set to those 3 additional links
@vietnguyengit - Regrouping for the next round ...
Issue description
I have determined that the links are missing in the harvested records for cases where the protocol is downloadother
and/or when the source file to download is NOT located at the catalogue we are harvesting from. Since catalogue-portal is harvesting from catalogue-imos the links are present because the file is located at catalogue-imos. But in these cases the file does not get downloaded to catalogue-portal (this is correct). The subsequent harvest from catalogue-portal to catalogue-systest sees that the file is not present at catalogue-portal so excludes the link. This applies to all records with these types of links - ie it is not limited to 8edf509b-1481-48fd-b9c5-b95b42247f82
.
Possible next steps:
Conclusion
The download protocols which are available to GN are specified in the schema-plugins. downloadother is not in iso19115-3.2018 schema plugin. This absence can be seen in the associated resources
panel which leads to it being impossible to edit (or create) such download links there. Harvesting is also using the schema plugin to validate and create the links in the harvested record. This is probably where it is going wrong.
We could look at updating the schema-plugin, but that would be a significant amount of work and there are still uncertaintities that this will completely resolve the issue. Since this will only be an issue in systest I don't think that is worth the effort.
My suggestion is to create a second harvester for systest that will harvest the records directly from catalogue-imos. The harvester will need to overwrite records that are all ready harvested from catalogue-portal.
Some notes that may or may not be related:
transform_19115_3_testing_urls
. We checked the operation of the transform on the raw XML from catalogue-portal and catalogue-imos and it is working as expected.fix up
transform when editing records. This should not affect harvesting but we checked that by turning off the option and reharvesting.downloadother
protocol.There will be other legacy download protocols that may also be, being excluded. These originate from the ANZMEST days. The Portal code has been set up to recognise them where appropriate.
From a report run in 2015, this is the list of all protocols that were present in metadata records.
WWW:DOWNLOAD-1.0-http--download WWW:DOWNLOAD-1.0-http—downloadother WWW:LINK-1.0-http—downloaddata WWW:LINK-1.0-http--link WWW:LINK-1.0-http--metadata-URL (not relevant anymore) WWW:LINK-1.0-http—related
DoDed
This issue tracker is only for AODN Portal issues.
Test of reference: https://aodn.testrail.com/index.php?/tests/view/84237
Steps to reproduce the issue
Expected behaviour
should list 4 links (see screenshot below taken from production that is working fine)
Actual behaviour
Only one link "view metadata link"
Screenshots.
Screenshot for production:![Screenshot from 2023-02-23 08-52-06](https://user-images.githubusercontent.com/5467402/220767841-cd705c67-1aa5-4346-bb52-ef80bc1191f0.png)
What version of the aodn portal are you using?
4.42.97
Which browser are you using?
Chrome
Any extra information that might be useful in the debugging process.
This is working as expected on prod
PR @vietnguyengit
https://github.com/aodn/cloud-deploy-config/pull/822 https://github.com/aodn/cloud-deploy/pull/1347
DOD @utas-raymondng