search

GSS-Cogs / family-covid-19

0 stars 0 forks source link

NISRA-Weekly-deaths-Year-NI #15

Open ajtucker opened 4 years ago

Tracey-B commented 4 years ago

BA Tech Spec Review 09/07/2020 Only tables 1-7 (deaths registered) have been included in the stage 1 spec, consider if tables 8-11 (death occurrences)should also be included.

LPerryman commented 4 years ago

Data split into 2, Registered deaths and date of death. Only Registered deaths are uploaded to PMD4 at the moment due to current limitations of the system. Needs to be changed back when PMD4 is capable of loading more than one dataset per pipeline

ajtucker commented 4 years ago

Running csv2rdf on covid-19-death-occurrences-observations.csv produces an empty file. I guess we need the next step GSS-Cogs/gss-utils#59

LPerryman commented 4 years ago

To get both datasets through to PMD had to change the Unit column values to Count when some should be Cumulative Count for the Date of Death dataset

LPerryman commented 4 years ago

Had some problems with duplicate rows. Made changes to both datasets but removed the Cumulative Count values altogether in the DATE OF DEATH dataset to get it published on PMD4 but these can be derived from the main data anyway so is not a major loss. Both datasets now published in PMD4 so moving this issue to the 'To Review' column

ajtucker commented 4 years ago

I've added a mapping for the data markers as we now have the "GSS Harmonised Symbols" codelist back.

However, running the update now shows an error with duplicate observations again.

ajtucker commented 4 years ago

Adding a label to pipelines that have the float64 temporary workaround.

Note that float64 isn't a valid XML Scheme datatype, it should be double (64 bit float) or ideally decimal (no limit on decimal places).

LPerryman commented 4 years ago

Stage 2 spec has been completed (for a second time). Waiting to find out exactly what the hyphens '-' mean but data can be transformed in the mean time but don't publish.

ajtucker commented 4 years ago

In the current data, the counts are showing up as decimals with a .0 on the end and in the RDF are all typed as strings, so we have to coerce them into xsd:decimals before we can add them up.

We should ensure that they're all integer counts.

LPerryman commented 4 years ago

Email from NISRA about '-' in data values

Hi David (Hull) Thank you for the reminder – your email got lost in a sea of emails and I haven’t had a chance to do my usual review of emails to check what’s outstanding! The ‘–‘ represents no covid cases and that is the same for all tables, so for example in Table 6 the ‘–‘ in Antrim & Newtownabbey on Week 38 just means that no cases were recorded. You are right though, that is not consistent with other tables where zeros are recorded! I hadn’t even noticed that! Claire (claire.rocks@nisra.gov.uk)

david-hull commented 4 years ago

Both datasets checked. All looks fine except for know issues (error accessing landing page from PMD, etc.) and filters are not working.