Closed steveoni closed 3 months ago
The new dcat is tested with this rest api https://ws-data.nisra.gov.uk/public/api.restful/PxStat.Data.Cube_API.ReadDataset/CCMSOA/JSON-stat/2.0/en
and here is the harvester: https://ni.dev.datopian.com/harvest/edit/ws-data-rest
and dataset generated: https://ni.dev.datopian.com/dataset/ccmsoa
Hi @steveoni We are missing the following metadata in the harvested resource: There is a last updated on the source that is not showing in the harvested dataset
We also have the info in terms of
Topic category: Economic and Labour Market Statistics (in ODNI: https://admin.opendatani.gov.uk/group/economy)
Frequency of update: Monthly
contact: name and email and phone is in the source
Is this metadata not captured in the API?
How can we populate this as well.
ALSO note that for this source, the harvester would need to run monthly
In terms of the csv resource it looks good
Hi @steveoni We are missing the following metadata in the harvested resource: There is a last updated on the source that is not showing in the harvested dataset
![]()
We also have the info in terms of Topic category: Economic and Labour Market Statistics (in ODNI: https://admin.opendatani.gov.uk/group/economy) Frequency of update: Monthly contact: name and email and phone is in the source
Is this metadata not captured in the API?
How can we populate this as well.
ALSO note that for this source, the harvester would need to run monthly
@SusanBot85 the screenshot of the last updated
is for the general dataset not for a specific resource. So if dataset is harvested, it shows the created date
as the day harvested and updated datae will show when next it is updated. thats my guess.
also if you want the last updated
to be attached to each resources, thats possible. but this can me noted for the next round of work
but this can me noted for the next round of work
What does this mean? How will it change the scope?
Also what is your feedback in terms of pulling in the additional info that is present on the source: Topic category: Economic and Labour Market Statistics (in ODNI: https://admin.opendatani.gov.uk/group/economy) Frequency of update: Monthly contact: name and email and phone is in the source
cc @gabrielailieva @anuveyatsu
@SusanBot85
Topic category: Economic and Labour Market Statistics
this contact name not Topic category
for Frequency and contact name
this already applied with a fixed value (OSNI Mapping Helpdesk
) this follow the same pattern they use for their other DCAT harvester we have.
but for a reason is not getting reflected, so thats a bug to look into.
So for fast feeback i will update this manually so we can send message to the client. and if they approve of it, fixing the bug and other udpate will be part of the main work
Also question for the client
1) what field in the API should we use for Topic category
2) what field should we use for keyword
3) should we use the contact name and address provided in the API
@SusanBot85
The client wants to add a new dataset catalog to the portal and the current harvester does not support this new data-catalog, hence a need to add a new harvester implementation
Assumption
We are working on the assumption that we will harvest all the public datasets on the NISRA platform
Acceptance
Task
Analysis
Analysis done by ismail: https://hackmd.io/@susanbot/rJPIlAx7T
Esitmate
From the Ismail analysis he said 3-4.5 days, I will estimate 1 week with a little buffer, might be less.
Setting up the environment locally and ensuring harvester works might take time and also mapping the user data with ckan can take time.
Expected completion date : latest 1st April 2024.