search

magda-io / magda

A federated, open-source data catalog for all your big data and small data
https://magda.io
Apache License 2.0
495 stars 93 forks source link

Registry API dereference parameter not work for in cluster `global` mode #2383

Closed t83714 closed 4 years ago

t83714 commented 4 years ago

Describe the bug

Registry API dereference parameter not work for in cluster global mode.

It seems we introduced a new tenant ID -1 for global mode and we use this for indexer indexing requests.

However, it seemsdereference parameter will be ignored if tenant ID is -1 (works fine for 0).

e.g. distributions aspects will be left with a list of ids. This will stop indexer from indexing any records.

To Reproduce

  1. Port forward the registry API (as request goes through the gateway will have tenant ID overwritten )
  2. Make GET request http://localhost:6102/v0/records?aspect=dcat-dataset-strings&optionalAspect=dataset-distributions&optionalAspect=source&optionalAspect=temporal-coverage&optionalAspect=dataset-publisher&optionalAspect=dataset-quality-rating&optionalAspect=dataset-format&optionalAspect=publishing&optionalAspect=spatial-coverage&optionalAspect=dataset-access-control&dereference=true&start=0&limit=1 with header: X-Magda-Tenant-Id=-1
  3. The returned response shows the distributions aspects contains ids only instead of the actual aspect data.

e.g.:

{
    "hasMore": true,
    "nextPageToken": "194075",
    "records": [
        {
            "aspects": {
                "source": {
                    "id": "neii",
                    "name": "National Environmental Information Infrastructure",
                    "type": "csw-dataset",
                    "url": "http://neii.bom.gov.au/services/catalogue/csw?service=CSW&version=2.0.2&request=GetRecordById&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&id=34b50bab-8124-4c78-a1d1-a0a83601ec56"
                },
                "dcat-dataset-strings": {
                    "contactPoint": "Australian Bureau of Meteorology, climatedata@bom.gov.au",
                    "description": "These datasets are long term averages of solar radiation at the surface over the Australian land mass. Applications of these data include solar energy, agriculture, building thermal design and water balance modelling. Climatologies are given for two radiation parameters: the global horizontal exposure, which is the total amount of solar energy falling on a horizontal surface over a time interval; and the direct normal exposure which is the total of the component of radiation from the sun’s disk on a plane perpendicular to the beam. Climatologies of daily exposure are given as an annual average and as a set of twelve monthly averages. Climatologies of the diurnal cycle are given as monthly averages of hourly exposures through the day. These data sets are derived from 23 years (1990 - 2012) of data from satellites operated by Japan Meteorological Agency and the US National Oceanographic & Atmospheric Administration. Data are updated infrequently.  NEII",
                    "keywords": [
                        "WMS",
                        "OPeNDAP",
                        "Gridded",
                        "Satellite",
                        "Solar",
                        "Radiation",
                        "Meteorology",
                        "National Environmental Information Infrastructure",
                        "neii.gov.au"
                    ],
                    "languages": [
                        "eng"
                    ],
                    "modified": "2008-01-01T12:00:00",
                    "publisher": "Australian Bureau of Meteorology",
                    "spatial": "POLYGON((112.025 -43.975, 153.975 -43.975, 153.975 -10.025, 112.025 -10.025, 112.025 -43.975))",
                    "themes": [],
                    "title": "Australian Gridded Solar Climatology Web Data Services"
                },
                "dataset-publisher": {
                    "publisher": {
                        "aspects": {
                            "organization-details": {
                                "name": "Australian Bureau of Meteorology",
                                "title": "Australian Bureau of Meteorology"
                            },
                            "source": {
                                "id": "neii",
                                "name": "National Environmental Information Infrastructure",
                                "type": "csw-organization",
                                "url": "http://neii.bom.gov.au/services/catalogue/csw"
                            }
                        },
                        "id": "org-neii-Australian Bureau of Meteorology",
                        "name": "Australian Bureau of Meteorology"
                    }
                },
                "dataset-distributions": {
                    "distributions": [
                        "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-0",
                        "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-1",
                        "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-2",
                        "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-3",
                        "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-4",
                        "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-5",
                        "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-6"
                    ]
                },
                "dataset-quality-rating": {
                    "dataset-linked-data-rating": {
                        "score": 0,
                        "weighting": 1
                    }
                }
            },
            "id": "ds-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56",
            "name": "Australian Gridded Solar Climatology Web Data Services",
            "sourceTag": "166b0e41-5fa5-4cd8-b879-b421b699645b",
            "tenantId": 0
        }
    ]
}

Expected behavior

The response should be:

{
    "hasMore": true,
    "nextPageToken": "194075",
    "records": [
        {
            "aspects": {
                "source": {
                    "id": "neii",
                    "name": "National Environmental Information Infrastructure",
                    "type": "csw-dataset",
                    "url": "http://neii.bom.gov.au/services/catalogue/csw?service=CSW&version=2.0.2&request=GetRecordById&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&id=34b50bab-8124-4c78-a1d1-a0a83601ec56"
                },
                "dcat-dataset-strings": {
                    "contactPoint": "Australian Bureau of Meteorology, climatedata@bom.gov.au",
                    "description": "These datasets are long term averages of solar radiation at the surface over the Australian land mass. Applications of these data include solar energy, agriculture, building thermal design and water balance modelling. Climatologies are given for two radiation parameters: the global horizontal exposure, which is the total amount of solar energy falling on a horizontal surface over a time interval; and the direct normal exposure which is the total of the component of radiation from the sun’s disk on a plane perpendicular to the beam. Climatologies of daily exposure are given as an annual average and as a set of twelve monthly averages. Climatologies of the diurnal cycle are given as monthly averages of hourly exposures through the day. These data sets are derived from 23 years (1990 - 2012) of data from satellites operated by Japan Meteorological Agency and the US National Oceanographic & Atmospheric Administration. Data are updated infrequently.  NEII",
                    "keywords": [
                        "WMS",
                        "OPeNDAP",
                        "Gridded",
                        "Satellite",
                        "Solar",
                        "Radiation",
                        "Meteorology",
                        "National Environmental Information Infrastructure",
                        "neii.gov.au"
                    ],
                    "languages": [
                        "eng"
                    ],
                    "modified": "2008-01-01T12:00:00",
                    "publisher": "Australian Bureau of Meteorology",
                    "spatial": "POLYGON((112.025 -43.975, 153.975 -43.975, 153.975 -10.025, 112.025 -10.025, 112.025 -43.975))",
                    "themes": [],
                    "title": "Australian Gridded Solar Climatology Web Data Services"
                },
                "dataset-publisher": {
                    "publisher": {
                        "aspects": {
                            "organization-details": {
                                "name": "Australian Bureau of Meteorology",
                                "title": "Australian Bureau of Meteorology"
                            },
                            "source": {
                                "id": "neii",
                                "name": "National Environmental Information Infrastructure",
                                "type": "csw-organization",
                                "url": "http://neii.bom.gov.au/services/catalogue/csw"
                            }
                        },
                        "id": "org-neii-Australian Bureau of Meteorology",
                        "name": "Australian Bureau of Meteorology"
                    }
                },
                "dataset-distributions": {
                    "distributions": [
                        {
                            "aspects": {
                                "csw-distribution": {},
                                "dataset-format": {
                                    "confidenceLevel": 70,
                                    "format": "WMS"
                                },
                                "dcat-distribution-strings": {
                                    "accessURL": "http://neii.bom.gov.au/services/solarclim/wms/data/monClim_gloHorExp1Hou.nc?service=WMS&version=1.3.0&request=GetCapabilities",
                                    "description": "Solar Climatology - Monthly-hourly climatology of hourly exposure (Global Horizontal Exposure) - Web Map Service",
                                    "format": "OGC:WMS-1.3.0-http-get-capabilities",
                                    "title": "Solar Climatology - Monthly-hourly climatology of hourly exposure (Global Horizontal Exposure) - Web Map Service"
                                },
                                "source": {
                                    "id": "neii",
                                    "name": "National Environmental Information Infrastructure",
                                    "type": "csw-distribution",
                                    "url": "http://neii.bom.gov.au/services/catalogue/csw?service=CSW&version=2.0.2&request=GetRecordById&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&id=34b50bab-8124-4c78-a1d1-a0a83601ec56"
                                },
                                "source-link-status": {
                                    "httpStatusCode": 200,
                                    "status": "active"
                                }
                            },
                            "id": "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-0",
                            "name": "Solar Climatology - Monthly-hourly climatology of hourly exposure (Global Horizontal Exposure) - Web Map Service"
                        },
                        {
                            "aspects": {
                                "csw-distribution": {},
                                "dataset-format": {
                                    "confidenceLevel": 70,
                                    "format": "WMS"
                                },
                                "dcat-distribution-strings": {
                                    "accessURL": "http://neii.bom.gov.au/services/solarclim/wms/data/monClim_dirNorExp1Hou.nc?service=WMS&version=1.3.0&request=GetCapabilities",
                                    "description": "Solar Climatology - Monthly-hourly climatology of hourly exposure (Direct Normal Exposure) - Web Map Service",
                                    "format": "OGC:WMS-1.3.0-http-get-capabilities",
                                    "title": "Solar Climatology - Monthly-hourly climatology of hourly exposure (Direct Normal Exposure) - Web Map Service"
                                },
                                "source": {
                                    "id": "neii",
                                    "name": "National Environmental Information Infrastructure",
                                    "type": "csw-distribution",
                                    "url": "http://neii.bom.gov.au/services/catalogue/csw?service=CSW&version=2.0.2&request=GetRecordById&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&id=34b50bab-8124-4c78-a1d1-a0a83601ec56"
                                },
                                "source-link-status": {
                                    "httpStatusCode": 200,
                                    "status": "active"
                                }
                            },
                            "id": "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-1",
                            "name": "Solar Climatology - Monthly-hourly climatology of hourly exposure (Direct Normal Exposure) - Web Map Service"
                        },
                        {
                            "aspects": {
                                "csw-distribution": {},
                                "dataset-format": {
                                    "confidenceLevel": 70,
                                    "format": "WMS"
                                },
                                "dcat-distribution-strings": {
                                    "accessURL": "http://neii.bom.gov.au/services/solarclim/wms/data/monClim_gloHorExp1Day.nc?service=WMS&version=1.3.0&request=GetCapabilities",
                                    "description": "Solar Climatology - Monthly climatology of daily exposure (Global Horizontal Exposure) - Web Map Service",
                                    "format": "OGC:WMS-1.3.0-http-get-capabilities",
                                    "title": "Solar Climatology - Monthly climatology of daily exposure (Global Horizontal Exposure) - Web Map Service"
                                },
                                "source": {
                                    "id": "neii",
                                    "name": "National Environmental Information Infrastructure",
                                    "type": "csw-distribution",
                                    "url": "http://neii.bom.gov.au/services/catalogue/csw?service=CSW&version=2.0.2&request=GetRecordById&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&id=34b50bab-8124-4c78-a1d1-a0a83601ec56"
                                },
                                "source-link-status": {
                                    "httpStatusCode": 200,
                                    "status": "active"
                                }
                            },
                            "id": "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-2",
                            "name": "Solar Climatology - Monthly climatology of daily exposure (Global Horizontal Exposure) - Web Map Service"
                        },
                        {
                            "aspects": {
                                "csw-distribution": {},
                                "dataset-format": {
                                    "confidenceLevel": 70,
                                    "format": "WMS"
                                },
                                "dcat-distribution-strings": {
                                    "accessURL": "http://neii.bom.gov.au/services/solarclim/wms/data/monClim_dirNorExp1Day.nc?service=WMS&version=1.3.0&request=GetCapabilities",
                                    "description": "Solar Climatology - Monthly climatology of daily exposure (Direct Normal Exposure) - Web Map Service",
                                    "format": "OGC:WMS-1.3.0-http-get-capabilities",
                                    "title": "Solar Climatology - Monthly climatology of daily exposure (Direct Normal Exposure) - Web Map Service"
                                },
                                "source": {
                                    "id": "neii",
                                    "name": "National Environmental Information Infrastructure",
                                    "type": "csw-distribution",
                                    "url": "http://neii.bom.gov.au/services/catalogue/csw?service=CSW&version=2.0.2&request=GetRecordById&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&id=34b50bab-8124-4c78-a1d1-a0a83601ec56"
                                },
                                "source-link-status": {
                                    "httpStatusCode": 200,
                                    "status": "active"
                                }
                            },
                            "id": "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-3",
                            "name": "Solar Climatology - Monthly climatology of daily exposure (Direct Normal Exposure) - Web Map Service"
                        },
                        {
                            "aspects": {
                                "csw-distribution": {},
                                "dataset-format": {
                                    "confidenceLevel": 70,
                                    "format": "WMS"
                                },
                                "dcat-distribution-strings": {
                                    "accessURL": "http://neii.bom.gov.au/services/solarclim/wms/data/annClim_gloHorExp1Day.nc?service=WMS&version=1.3.0&request=GetCapabilities",
                                    "description": "Solar Climatology - Annual climatology of daily exposure (Global Horizontal Exposure) - Web Map Service",
                                    "format": "OGC:WMS-1.3.0-http-get-capabilities",
                                    "title": "Solar Climatology - Annual climatology of daily exposure (Global Horizontal Exposure) - Web Map Service"
                                },
                                "source": {
                                    "id": "neii",
                                    "name": "National Environmental Information Infrastructure",
                                    "type": "csw-distribution",
                                    "url": "http://neii.bom.gov.au/services/catalogue/csw?service=CSW&version=2.0.2&request=GetRecordById&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&id=34b50bab-8124-4c78-a1d1-a0a83601ec56"
                                },
                                "source-link-status": {
                                    "httpStatusCode": 200,
                                    "status": "active"
                                }
                            },
                            "id": "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-4",
                            "name": "Solar Climatology - Annual climatology of daily exposure (Global Horizontal Exposure) - Web Map Service"
                        },
                        {
                            "aspects": {
                                "csw-distribution": {},
                                "dataset-format": {
                                    "confidenceLevel": 70,
                                    "format": "WMS"
                                },
                                "dcat-distribution-strings": {
                                    "accessURL": "http://neii.bom.gov.au/services/solarclim/wms/data/annClim_dirNorExp1Day.nc?service=WMS&version=1.3.0&request=GetCapabilities",
                                    "description": "Solar Climatology - Annual climatology of daily exposure (Direct Normal Exposure) - Web Map Service",
                                    "format": "OGC:WMS-1.3.0-http-get-capabilities",
                                    "title": "Solar Climatology - Annual climatology of daily exposure (Direct Normal Exposure) - Web Map Service"
                                },
                                "source": {
                                    "id": "neii",
                                    "name": "National Environmental Information Infrastructure",
                                    "type": "csw-distribution",
                                    "url": "http://neii.bom.gov.au/services/catalogue/csw?service=CSW&version=2.0.2&request=GetRecordById&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&id=34b50bab-8124-4c78-a1d1-a0a83601ec56"
                                },
                                "source-link-status": {
                                    "httpStatusCode": 200,
                                    "status": "active"
                                }
                            },
                            "id": "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-5",
                            "name": "Solar Climatology - Annual climatology of daily exposure (Direct Normal Exposure) - Web Map Service"
                        },
                        {
                            "aspects": {
                                "csw-distribution": {},
                                "dataset-format": {
                                    "confidenceLevel": 90,
                                    "format": "XML"
                                },
                                "dcat-distribution-strings": {
                                    "accessURL": "http://neii.bom.gov.au/services/solarclim/catalog.xml",
                                    "description": "Solar Climatology - Solar Climatology data products - OPeNDAP (XML)",
                                    "format": "WWW:LINK-1.0-http--opendap",
                                    "title": "Solar Climatology - Solar Climatology data products - OPeNDAP (XML)"
                                },
                                "source": {
                                    "id": "neii",
                                    "name": "National Environmental Information Infrastructure",
                                    "type": "csw-distribution",
                                    "url": "http://neii.bom.gov.au/services/catalogue/csw?service=CSW&version=2.0.2&request=GetRecordById&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&id=34b50bab-8124-4c78-a1d1-a0a83601ec56"
                                },
                                "source-link-status": {
                                    "httpStatusCode": 200,
                                    "status": "active"
                                }
                            },
                            "id": "dist-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56-6",
                            "name": "Solar Climatology - Solar Climatology data products - OPeNDAP (XML)"
                        }
                    ]
                },
                "dataset-quality-rating": {
                    "dataset-linked-data-rating": {
                        "score": 0,
                        "weighting": 1
                    }
                }
            },
            "id": "ds-neii-34b50bab-8124-4c78-a1d1-a0a83601ec56",
            "name": "Australian Gridded Solar Climatology Web Data Services",
            "sourceTag": "166b0e41-5fa5-4cd8-b879-b421b699645b",
            "tenantId": 0
        }
    ]
}
AlexGilleran commented 4 years ago

@t83714 is this fixed by #2367 ?

t83714 commented 4 years ago

Closed as it's fixed via https://github.com/magda-io/magda/pull/2367