Portal systest - additinoal link on step 3 not displayed

evacougnon commented 1 year ago

This issue tracker is only for AODN Portal issues.

Test of reference: https://aodn.testrail.com/index.php?/tests/view/84237

Steps to reproduce the issue

Expected behaviour

should list 4 links (see screenshot below taken from production that is working fine)

Actual behaviour

Only one link "view metadata link"

Screenshots.

Screenshot for production: Screenshot from 2023-02-23 08-52-06

What version of the aodn portal are you using?

4.42.97

Which browser are you using?

Chrome

Any extra information that might be useful in the debugging process.

This is working as expected on prod

PR @vietnguyengit

https://github.com/aodn/cloud-deploy-config/pull/822 https://github.com/aodn/cloud-deploy/pull/1347

DOD @utas-raymondng

craigrose commented 1 year ago

Can't replicate in prod. Left note in testrail to check on this next systest

evacougnon commented 1 year ago

reopening as still happening with portal systest version 4.42.102

Chetnamann commented 1 year ago

Issue still happening

What version of the aodn portal are you using? 4.42.104

Which browser are you using? Chrome

vietnguyengit commented 1 year ago

Since there's nothing else on list for tonight and tomorrow Friday 28/7, I start to pick on this one (which was originally allocated to iteration 256. @utas-raymondng

utas-raymondng commented 1 year ago

@vietnguyengit Thanks

vietnguyengit commented 1 year ago

can't reproduce locally for the particular UUID mentioned

vietnguyengit commented 1 year ago

I put a console.log() into the method at : https://github.com/aodn/aodn-portal/blob/378f5a921b3d646e1d777454b6b21806a32e2562/web-app/js/portal/cart/DownloadPanelItemTemplate.js#L235

here what's I can see (locally).

I believe for some reason, that particular UUID collection object, value of linkFiles variable is not assigned on systest environment, do you think there could be any conflicts happening when doing portal systest procedure @craigrose ?

Chetnamann commented 1 year ago

Issue still happening @vfisaac

Shows error when trying to download, see screenshot

What version of the aodn portal are you using? Portal v4.42.104

Which browser are you using? Chrome

vietnguyengit commented 1 year ago

A diff check shows Geonetwork metadata files of the mentioned record between systest and prod are different (missed those extra links in systest Geonetwork record)

So my conclusion is, this is not an issue to deal with the portal, more likely with the Geonetwork side, possibly a glitch during the harvesting process?

https://www.diffchecker.com/5bEe1mij/

systest on the left - prod on the right

vietnguyengit commented 1 year ago

note to myself:

https://github.com/aodn/harvesters/tree/master/workspace/SOOP_BA https://github.com/aodn/chef-private/blob/7bc7103bef2f7725e2b7fde8088e636bc38a50c7/bin/talend-helper.rb#L93C1-L105C4

https://github.com/aodn/cloud-deploy/blob/master/sample-config/ebprep_conf/templates/geonetwork3/linkage-updater.xsl.template

https://deploy.aodn.org.au/job/cloudformation_deploy/14526/console

TASK [geonetwork3_config : add harvesters and populate metadata catalogue] *****
[0;33mchanged: [localhost] => (item={'name': 'Catalogue Portal for systest', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'Catalogue Portal for systest', 'host': 'https://catalogue-portal.aodn.org.au/geonetwork', 'mefFormatFull': 'false', 'createRemoteCategory': 'true', 'xslfilter': 'transform_19115_3_testing_urls'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 0 0 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m

vietnguyengit commented 1 year ago

So @atkinsn confirmed that stuff in catalogue stacks is manually created. My theory for this issue now is:

(1) Somehow, details of IMOS - Bio-Acoustic Ships of Opportunity (BA SOOP) sub-facility metadata record are out of sync between catalogue-portal-systest and prod catalogue portal. (2) The deployment of portal systest fetches data from the catalogue portal systest stack, while prod fetches data from the prod catalogue portal, which is why that record on portal systest doesn't have the additional links as in portal prod.

To fix this, simply add those 3 missing additional links to the metadata record of IMOS - Bio-Acoustic Ships of Opportunity (BA SOOP) sub-facility on catalogue-portal-systest and rebuild/deploy the portal systest.

vietnguyengit commented 1 year ago

and again, I strongly believe this is not anything to deal with the portal codebase.

vietnguyengit commented 1 year ago

I'll need another confirmation whether it is safe to click the redeploy button for the portal systest at the moment @vfisaac

atkinsn commented 1 year ago

Those 3 links relate to 3 directly uploaded files. Looks like they are getting stripped out in the harvest process.

vietnguyengit commented 1 year ago

Those 3 links relate to 3 directly uploaded files. Looks like they are getting stripped out in the harvest process.

thanks @atkinsn

vietnguyengit commented 1 year ago

the Jenkins deploy logs between systest and prod of the step called "add harvesters and populate metadata catalogue" (which probably declared here: https://github.com/aodn/cloud-deploy/blob/e5f5aafb906c4e4664f1db068b883c7be5fb8e49/ansible/roles/geonetwork3_config/tasks/manage.yml#L181)

are very different

systest: https://deploy.aodn.org.au/job/cloudformation_deploy/14526/console

TASK [geonetwork3_config : add harvesters and populate metadata catalogue] *****
[0;33mchanged: [localhost] => (item={'name': 'Catalogue Portal for systest', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'Catalogue Portal for systest', 'host': 'https://catalogue-portal.aodn.org.au/geonetwork', 'mefFormatFull': 'false', 'createRemoteCategory': 'true', 'xslfilter': 'transform_19115_3_testing_urls'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 0 0 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m

prod: https://deploy.aodn.org.au/job/cloudformation_deploy/14070/console

TASK [geonetwork3_config : add harvesters and populate metadata catalogue] *****
[0;33mchanged: [localhost] => (item={'name': 'External - AAD', 'harvester_data': {'node': {'@type': 'webdav', 'site': {'name': 'External - AAD', 'url': 'https://data.aad.gov.au/sg/api/waf/aadc-metadata/iso-19115-1-aodn/', 'icon': 'AAD_logo.gif'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active', 'validate': 'false', 'recurse': 'false', 'subtype': 'waf'}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'linkage-updater?pattern=[http://geoserver-123.aodn.org.au&replacement=http://geoserver-portal.aodn.org.au](http://geoserver-123.aodn.org.au%26replacement%3Dhttp//geoserver-portal.aodn.org.au)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': '%Sea Temperature Observing System%'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:AIMS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS: Microdebris contamination in Australian coastal and marine waters', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS: Microdebris contamination in Australian coastal and marine waters', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'linkage-updater?pattern=[http://geoserver-123.aodn.org.au&replacement=http://geoserver-portal.aodn.org.au](http://geoserver-123.aodn.org.au%26replacement%3Dhttp//geoserver-portal.aodn.org.au)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': 'Microdebris contamination in Australian coastal and marine waters'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:IMOS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS: Great Barrier Reef Genomics Database: Seawater Illumina Reads', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS: Great Barrier Reef Genomics Database: Seawater Illumina Reads', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'linkage-updater?pattern=[http://geoserver-123.aodn.org.au&replacement=http://geoserver-portal.aodn.org.au](http://geoserver-123.aodn.org.au%26replacement%3Dhttp//geoserver-portal.aodn.org.au)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': 'Great Barrier Reef Genomics Database: Seawater Illumina Reads'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:IMOS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS: Northern Australia Automated Marine Weather and Oceanographic Stations', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS: Northern Australia Automated Marine Weather and Oceanographic Stations', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'linkage-updater?pattern=[http://geoserver-123.aodn.org.au&replacement=http://geoserver-portal.aodn.org.au](http://geoserver-123.aodn.org.au%26replacement%3Dhttp//geoserver-portal.aodn.org.au)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': '%Northern Australia Automated Marine Weather and Oceanographic Stations%'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:AIMS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - AIMS: Great Barrier Reef Marine Monitoring Program for Inshore Water Quality', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'External - AIMS: Great Barrier Reef Marine Monitoring Program for Inshore Water Quality', 'host': 'https://geonetwork.apps.aims.gov.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'modify-urls', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 30 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'title': 'Great Barrier Reef Marine Monitoring Program for Inshore Water Quality - Vertical Profiles of Conductivity Temperature and Depth (CTD)'}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:AIMS'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A ADCP Data Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A ADCP Data Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'fe20ec11-ad5b-425f-82dd-6b00f448eb1a'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Catch Operations Data', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Catch Operations Data', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'c836f971-991e-10ec-e043-0200007f2db9'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Current Meter Mooring Data Overview (1983 - 1998)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Current Meter Mooring Data Overview (1983 - 1998)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'dfbb5340-9119-4559-8712-a53630acfbb0'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Marine National Facility Voyage Tracks Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Marine National Facility Voyage Tracks Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'c836a6dc-eba7-28c3-e043-0200007fdbdb'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A CTD Data Overview (1982 - present)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A CTD Data Overview (1982 - present)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'c1229b01-41c7-08bf-e044-00144f7bc0f4'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Hydrology Data Overview (1942 - present)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Hydrology Data Overview (1942 - present)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'c1225108-233d-74e9-e044-00144f7bc0f4'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A RV Franklin XBT Data Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A RV Franklin XBT Data Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': '033a1ddd-b206-09d0-e053-08114f8c3b40'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A RV Investigator XBT Data Overview (2014 onwards)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A RV Investigator XBT Data Overview (2014 onwards)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'dfd2c49d-d469-4691-9d10-ac154fdd4985'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO RV Investigator ISAR/SST Sea Surface Temperature Data Overview (2014 onwards)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO RV Investigator ISAR/SST Sea Surface Temperature Data Overview (2014 onwards)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 0 0 ? * *', 'oneRunOnly': 'false', 'status': 'active', 'validate': 'false'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'bdf91f86-2968-4711-873e-2761383bb207'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A RV Southern Surveyor XBT Data Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A RV Southern Surveyor XBT Data Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active', 'validate': 'false'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': 'd5a640e7-c695-23e1-e043-08114f8c7b58'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Sediment Sampling Sites Overview (1986 to present)', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Sediment Sampling Sites Overview (1986 to present)', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': '82fa2ab2-2338-4505-b917-b400406b7c7c'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - CSIRO O and A Wildlife Observations Overview', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - CSIRO O and A Wildlife Observations Overview', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://www.marlin.csiro.au/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'CSIRO_logo.gif', 'xslfilter': 'linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': '11d4b1d8-c03e-4e37-8ecc-b085ca80e7df'}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'GA Catalogue MH370 Phase 1 150m Bathymetry datasets', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'GA Catalogue MH370 Phase 1 150m Bathymetry datasets', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'https://ecat.ga.gov.au/geonetwork/srv/eng/csw?service=CSW&request=GetCapabilities&acceptVersions=2.0.2', 'icon': 'GA_logo.gif', 'xslfilter': 'ga', 'outputSchema': 'http://www.isotc211.org/2005/gmd'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': 'd887e71a-71dc-4851-94a9-920f7b7cc7e5'}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'GA Catalogue Australian Marine Sediments (MARS) Database', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'GA Catalogue Australian Marine Sediments (MARS) Database', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'https://ecat.ga.gov.au/geonetwork/srv/eng/csw?service=CSW&request=GetCapabilities&acceptVersions=2.0.2', 'icon': 'GA_logo.gif', 'xslfilter': 'ga', 'outputSchema': 'http://www.isotc211.org/2005/gmd'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': 'a05f7892-eef2-7506-e044-00144fdd4fa6'}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - IMAS: AODN Portal IMAS', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - IMAS: AODN Portal IMAS', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://metadata.imas.utas.edu.au/geonetwork/srv/eng/csw-AODN-Portal-IMAS?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'IMAS_logo.gif', 'xslfilter': 'portal-linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'IMOS Catalogue', 'harvester_data': {'node': {'@type': 'geonetwork', 'site': {'name': 'IMOS Catalogue', 'host': 'https://catalogue-imos.aodn.org.au/geonetwork', 'createRemoteCategory': 'true', 'mefFormatFull': 'false', 'xslfilter': 'modify-urls', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 0 0 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'anyField': '_cat', 'anyValue': 'portal:*'}}, 'ifRecordExistAppendPrivileges': 'true', 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - NIWA - GN3 upgrade pending', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - NIWA - GN3 upgrade pending', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'https://catalogue.aodn.org.au/geonetwork/srv/eng/csw-NIWA?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'NIWA_logo.gif', 'xslfilter': 'replace', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'inactive'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - IMAS: AODN Portal OEH', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - IMAS: AODN Portal OEH', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://metadata.imas.utas.edu.au/geonetwork/srv/eng/csw-AODN-Portal-OEH?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'NSW_GOV_logo.gif', 'xslfilter': 'portal-linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}, 'categories': [{'category': {'@id': 'portal:NSW_GOV'}}]}}})[0m
[0;33mchanged: [localhost] => (item={'name': 'External - IMAS: AODN Portal UWA', 'harvester_data': {'node': {'@type': 'csw', 'site': {'name': 'External - IMAS: AODN Portal UWA', 'account': {'use': 'false', 'username': {}, 'password': {}}, 'capabilitiesUrl': 'http://metadata.imas.utas.edu.au/geonetwork/srv/eng/csw-AODN-Portal-UWA?request=GetCapabilities&service=CSW&acceptVersions=2.0.2', 'icon': 'UWA_logo.png', 'xslfilter': 'portal-linkage-updater', 'outputSchema': 'http://standards.iso.org/iso/19115/-3/mdb/2.0'}, 'content': {'validate': 'false', 'importxslt': 'none'}, 'options': {'every': '0 45 23 ? * *', 'oneRunOnly': 'false', 'status': 'active'}, 'searches': {'search': {'CreationDate': {'value': None}, 'GeographicDescriptionCode': {'value': None}, 'OperatesOn': {'value': None}, 'Modified': {'value': None}, 'DistanceUOM': {'value': None}, 'Operation': {'value': None}, 'ResourceIdentifier': {'value': None}, 'Format': {'value': None}, 'Identifier': {'value': None}, 'Language': {'value': None}, 'ServiceType': {'value': None}, 'OrganisationName': {'value': None}, 'KeywordType': {'value': None}, 'AnyText': {'value': None}, 'PublicationDate': {'value': None}, 'AlternateTitle': {'value': None}, 'Abstract': {'value': None}, 'HasSecurityConstraints': {'value': None}, 'Title': {'value': None}, 'CouplingType': {'value': None}, 'TopicCategory': {'value': None}, 'ParentIdentifier': {'value': None}, 'Subject': {'value': None}, 'ResourceLanguage': {'value': None}, 'TempExtent_end': {'value': None}, 'ServiceTypeVersion': {'value': None}, 'Type': {'value': None}, 'RevisionDate': {'value': None}, 'OperatesOnName': {'value': None}, 'Denominator': {'value': None}, 'DistanceValue': {'value': None}, 'TempExtent_begin': {'value': None}, 'OperatesOnIdentifier': {'value': None}, 'SpecificationDate': {'value': None}, 'AccessConstraints': {'value': None}, 'ResponsiblePartyRole': {'value': None}, 'Degree': {'value': None}, 'Lineage': {'value': None}, 'OnlineResourceMimeType': {'value': None}, 'ConditionApplyingToAccessAndUse': {'value': None}, 'Date': {'value': None}, 'MetadataPointOfContact': {'value': None}, 'OnlineResourceType': {'value': None}, 'Relation': {'value': None}, 'SpecificationDateType': {'value': None}, 'Classification': {'value': None}, 'OtherConstraints': {'value': None}, 'SpecificationTitle': {'value': None}, 'bbox-xmin': {'value': None}, 'bbox-ymin': {'value': None}, 'bbox-xmax': {'value': None}, 'bbox-ymax': {'value': None}}}, 'privileges': {'group': {'@id': '1', 'operation': [{'@name': 'view'}, {'@name': 'dynamic'}, {'@name': 'featured'}]}}}}})[0m

It is obvious that there were a lot more things that happened in the prod deployment at that particular step. I understand that, however, I doubt if the stuff in Catalogue Portal for systest (probably declared here: https://github.com/aodn/cloud-deploy/blob/master/sample-config/data_bags/geonetwork3/imos_webapps_geonetwork_harvesters/portal_catalogue_systest.json) works 100% correctly as it should.

vietnguyengit commented 1 year ago

Futher investigation notes:

There are 2 json files defining IMOS Catalogue not sure both of them are used or one is outdated:

https://github.com/aodn/cloud-deploy/blob/master/sample-config/data_bags/geonetwork3/imos_webapps_geonetwork_harvesters/portal_catalogue_imos.json: this one seems to be the one used in the above logs, it has xslfilter to be modify-urls
https://github.com/aodn/cloud-deploy/blob/master/sample-config/data_bags/geonetwork3/imos_webapps_geonetwork_harvesters/catalogue_imos.json : this one has name under site section Catalogue-imos portal records seems to be not found anywhere in the logs trail above, it has the same xslfilter filter as Catalogue Portal for systest which is transform_19115_3_testing_urls (defined here: https://github.com/aodn/cloud-deploy/blob/master/sample-config/ebprep_conf/templates/geonetwork3/transform_19115_3_testing_urls.xsl.template)

The json defining Catalogue Portal for systest here:

https://github.com/aodn/cloud-deploy/blob/master/sample-config/data_bags/geonetwork3/imos_webapps_geonetwork_harvesters/portal_catalogue_systest.json#L11C23-L11C53: I suspect if xslfilter of this one is correct? should it be modify-urls? it also doesn't have "outputSchema": "http://standards.iso.org/iso/19115/-3/mdb/2.0" set as in IMOS Catalogue

vietnguyengit commented 1 year ago

PR: https://github.com/aodn/cloud-deploy/pull/1334

vietnguyengit commented 1 year ago

well, I will run up 2 cloud deploy stacks to compare the changes then draw another conclusion

vietnguyengit commented 1 year ago

ahhh, worked to the bottom of this, I noticed systest fetching data from https://catalogue-portal.aodn.org.au

and prod fetching data from https://catalogue-imos.aodn.org.au/

which the metadata record of the same collection are different from here!

https://catalogue-portal.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82 (you can't find the 3 additional links from this one)

but you should be able to find them here: https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82

vietnguyengit commented 1 year ago

final PR to fix this stuff: https://github.com/aodn/cloud-deploy/pull/1336

vietnguyengit commented 1 year ago

@vfisaac @Chetnamann @evacougnon heads up, that I updated systest harvester to test the investigation theory above, details below.

The IMOS harvester on systest catalogue has been updated to reflect the configs in prod catalogue, I can't seem to be able wiping out the existing records harvested with the old configs. Tried overwriting UUID but not seem working as well, maybe needs a server restart?

Anyhow, I cloned the available systest harvester and particular filter for `IMOS - Bio-Acoustic Ships of Opportunity (BA SOOP) sub-facility" collection only and allow to create new UUID for the same collection (because there is one that already exists that I can't delete or overwrite - there is now 2 x IMOS - Bio-Acoustic Ships of Opportunity (BA SOOP) sub-facility availabile on catalogue systest :warning: - one with missing additional links (old one), one has everything (new one)).

A new metadata record available here: https://catalogue-systest.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/70172727-3425-4e90-bf4c-07191cb601ac

and it has those 3 additional links.

on the portal systest, you would see those 3 links show up in step 3:

https://portal-systest.aodn.org.au/search?uuid=70172727-3425-4e90-bf4c-07191cb601ac

by the above, I think I can draw a conclusion for the root of this issue...mismatching harvester's configs on prod and systest, in particular...fetching data from different sources.

Next step (possibly not in this ticket scope, at least we know why now?)

Try to refresh the catalogue systest deployment and re-harvest the records with updated harvester (wipe out existing records (I can't, maybe someone else can) or destroy portal systest stacks and redeploy them ?), systest metadata records should be harvested by the harvester with the same source as configured for prod catalogue to rectify this issue.

OR

No need to update configs and synchronise the source geonetwork for the systest harvester to match with prod. Simply fix the missing links from here (maybe manually?): https://catalogue-portal.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82

craigrose commented 1 year ago

@vietnguyengit have a look at https://github.com/aodn/cloud-deploy/blob/b9dacbe5850c49ae8924fa0b640597f73101c629/sample-config/ebprep_conf/templates/geonetwork3/modify-urls.xsl.template

the problem might be there. this is the only place metadata gets changed

vietnguyengit commented 1 year ago

Thanks @craigrose , there is a PR created before the one you wanted a revert to update the configs only, rather than changing from catalogue-portal to catalogue-imos

https://github.com/aodn/cloud-deploy/pull/1334

It however, doesn't have any effects wanted, 3 additional links still missing.

So I guess, the target to fix this problem is not about catalogue-systest but in catalogue-portal , additional links missing there already.

https://catalogue-portal.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82

vietnguyengit commented 1 year ago

Somehow, someone or something ... made the additional links appear back in the metadata record at: https://catalogue-portal.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82

I didn't make any changes to the IMOS harvester configs

@craigrose @atkinsn

vietnguyengit commented 1 year ago

now they disappears again @craigrose

vietnguyengit commented 1 year ago

ok to document what has been discussed in Slack,

the additional links not actually appeared and disappeared by magic in catalogue_portal stack

metadata record: https://catalogue-portal.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/8edf509b-1481-48fd-b9c5-b95b42247f82

if logged in under imos_admin account, the links show up when viewing the metadata record.

but unauthenticated will not see them!

logged results on the left, unauthenticated on the right:

we have tried clearing cache of both GN and local browsers, including deleting the harvested records and reharvesting them, no changes really.

then we looked into the database, and the additional links do exist at database table.

i searched in geonetwork portal 3 database on prod database node, with query:

select * from metadata where uuid='8edf509b-1481-48fd-b9c5-b95b42247f82';

but it's not nice to look at the lengthy text on terminal windows so I extract the contents to a text file with:

COPY(select * from metadata where uuid='8edf509b-1481-48fd-b9c5-b95b42247f82') TO '/tmp/soop_ba_db.txt' WITH CSV HEADER;

attached a copy of what needs to be checked here:

extracted_gn_soop_ba.txt

and those 3 links are in the extracted contents, so not caching issue but more likely permissions set to those 3 additional links

craigrose commented 1 year ago

@vietnguyengit - Regrouping for the next round ...

Issue description

I have determined that the links are missing in the harvested records for cases where the protocol is downloadother and/or when the source file to download is NOT located at the catalogue we are harvesting from. Since catalogue-portal is harvesting from catalogue-imos the links are present because the file is located at catalogue-imos. But in these cases the file does not get downloaded to catalogue-portal (this is correct). The subsequent harvest from catalogue-portal to catalogue-systest sees that the file is not present at catalogue-portal so excludes the link. This applies to all records with these types of links - ie it is not limited to 8edf509b-1481-48fd-b9c5-b95b42247f82.

Possible next steps:

Verify that this file/link checking is what is actually happening. To do this deploy a geonetwork with a copy of the record and the file. Mess with the protocol and endpoint.
Can we set up harvesting so that the harvesting follows the link to check for the file rather than just check of catalogue-portal?
Or - harvest the affected records from catalogue-imos directly?
Are Xlink settings having an impact?

Conclusion

The download protocols which are available to GN are specified in the schema-plugins. downloadother is not in iso19115-3.2018 schema plugin. This absence can be seen in the associated resources panel which leads to it being impossible to edit (or create) such download links there. Harvesting is also using the schema plugin to validate and create the links in the harvested record. This is probably where it is going wrong.

We could look at updating the schema-plugin, but that would be a significant amount of work and there are still uncertaintities that this will completely resolve the issue. Since this will only be an issue in systest I don't think that is worth the effort.

My suggestion is to create a second harvester for systest that will harvest the records directly from catalogue-imos. The harvester will need to overwrite records that are all ready harvested from catalogue-portal.

Some notes that may or may not be related:

Harvested records are transformed by transform_19115_3_testing_urls. We checked the operation of the transform on the raw XML from catalogue-portal and catalogue-imos and it is working as expected.
Geonetwork transforms XML using templates to display the metadata. Looking at the raw XML from the database we can see that the links are missing there.
GN uses a fix up transform when editing records. This should not affect harvesting but we checked that by turning off the option and reharvesting.
GN uses caches to cache display (there is also the browser cache). Clearing these made no difference.
At one point the problem seemed to be caused by permissions (logged in etc.). @vietnguyengit saw this but @craigrose could not duplicate.
Harvesting the record directly from catalogue-imos is no problem - the links are there.
The links that are missing have their endpoints (files) always located on catalogue-imos and use the downloadother protocol.

atkinsn commented 1 year ago

There will be other legacy download protocols that may also be, being excluded. These originate from the ANZMEST days. The Portal code has been set up to recognise them where appropriate.

From a report run in 2015, this is the list of all protocols that were present in metadata records.

WWW:DOWNLOAD-1.0-http--download WWW:DOWNLOAD-1.0-http—downloadother WWW:LINK-1.0-http—downloaddata WWW:LINK-1.0-http--link WWW:LINK-1.0-http--metadata-URL (not relevant anymore) WWW:LINK-1.0-http—related

utas-raymondng commented 1 year ago

DoDed

aodn / aodn-portal