search

akarlinsky / world_mortality

World Mortality Dataset: international data on all-cause mortality.
MIT License
282 stars 56 forks source link

Data Sources for Ireland #30

Open pb-white opened 1 year ago

pb-white commented 1 year ago

Hi,

I'm contacting you from the Department of Health in Ireland in relation to the data source quoted for Ireland which appears to be incorrect. For example the link provided for 2015-2019 (https://data.cso.ie/table/RIP02) does not contain any data pre-December 2019 when the web-scraped series from rip.ie began. Also for 2020, final official deaths data is now available, this should be used rather than the experiment web-scraped series for 2020.

The more appropriate sources are

Ireland (monthly) 2015 - 2020: Ireland Central Statistics Office (CSO) - Vital Statistics Annual Reports 2015-2020: https://data.cso.ie/table/VSD39

2021: Ireland Central Statistics Office (CSO) - Measuring Mortality Using Public Data Sources: https://data.cso.ie/table/RIP02

For 2022 onwards, weekly data based on the web-scarped series from rip.ie is available from Eurostat https://ec.europa.eu/eurostat/databrowser/view/demo_r_mwk_ts

We have also recently submitted these corrections to these sources to the WHO in relation to their ongoing consultation on their excess mortality estimates.

Pauline White

s-eiermann commented 1 year ago

Could you also provide pre-2015 data please?

akarlinsky commented 1 year ago

Thank you, we will update accordingly in next push.

On data sources: yes, this is a mistake in the links. Our data for 2015 to 2019 is official monthly death counts from CSO as distributed by EuroStat (https://ec.europa.eu/eurostat/databrowser/bookmark/dce42618-58fe-406c-b8e3-36d681fb4c9a?lang=en).

We will now switch to VSD39 for 2015-2020.

dkobak commented 1 year ago

Ariel, after the today's update, the README links to https://data.cso.ie/table/RIP02 twice, and does not link to https://data.cso.ie/table/VSD39. Is it a mistake?

akarlinsky commented 1 year ago

Yes I forgot to update the readme. I'm on a conference/trip in the next few days but probably can update the readme sometime later today.

On Sun, 26 Feb 2023, 00:43 Dmitry Kobak, @.***> wrote:

Ariel, after the today's update, the README links to https://data.cso.ie/table/RIP02 twice, and does not link to https://data.cso.ie/table/VSD39. Is it a mistake?

— Reply to this email directly, view it on GitHub https://github.com/akarlinsky/world_mortality/issues/30#issuecomment-1445220666, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALRQ66LXZZRCSIJUWOURGITWZKDKTANCNFSM6AAAAAAVDAUNFU . You are receiving this because you commented.Message ID: @.***>

pb-white commented 1 year ago

Hi thanks for responding to the query.

1) Pre-2015 data: The Table at this link https://data.cso.ie/table/VSD39 provides data from 2000 - 2020

We'll keep an eye out for the next update to the README links.

akarlinsky commented 1 year ago

readme updated in commit https://github.com/akarlinsky/world_mortality/commit/262a59cfd09bcfe7f2a1269373335c7aa77a2ac1.

@pb-white you can close this issue if everything is ok with you. Thanks again!

pb-white commented 1 year ago

Hi Thank you for following up on those updates. We are happy to close the issue now.

Just to flag that final official data on deaths by month for 2021 will be available at the link https://data.cso.ie/table/VSD39 in October of this year. When this becomes available, we'd ask that it replace the current preliminary web-scraped data for 2021. We'll be in contact when published. Thanks

akarlinsky commented 1 year ago

Of course. In all countries we constantly revise and use the most up-to-date and complete version of data.

Very happy to see that CSO Ireland is doing this. Many thanks.

akarlinsky commented 1 year ago

Dear Pauline, can you please send me an email? karlinsky@gmail.com

Thank you

R-Witzbold commented 1 week ago

Hi Ariel und Dimitry,

Please be advised, the death counts (2015-2021) in VSD39 from CSO from which you are calculating weekly totals for Ireland are INCOMPLETE. These totals exclude "late registered deaths", i.e. deaths registered after the CSO publishes its Vital Statistics Annual Report. For example: 2019 - over 900 deaths have since been added to total number of deaths occurring in VSD39 (which you are using), i.e. a difference ~3%

2007-2020 REVISED TOTALS are now readily available by MONTH from CSO https://data.cso.ie/table/VSM01 VSM01: Revised Deaths Occurring (Number): by Age Group, Sex, Month Notes: "This table was created to take account of late registered deaths and update figures published at the time of Annual Reports. These figures will be revised over time to take account of further late registrations...."

For 2021 onward, the Eurostat weekly data scraped from RIP.ie are also an undercount - i.e. it is not a question of the figures "might" change, but rather they "will increase" once registrations become more complete. CSO has estimated the scraped totals from RIP.ie are ultimately as much as 3~5% below actual. For example: 2020 - current shortfall between scraped weekly totals and revised figure is ~4%. 2022 - current shortfall is only ~1% (but will increase as more late registrations added)

Therefore, I suggest you comment the more recent (2021-2024) Irish data as "incomplete" instead of "preliminary".

Kind regards, Rainer

akarlinsky commented 1 week ago

@pb-white How do you think is best for us to proceed here?

R-Witzbold commented 6 days ago

@akarlinsky it depends on what quality you want:

Regarding Ms White's Statistics and Analytics Unit (Irish Dept. of Health), I can tell you they seemingly do not have a policy of updating their statistical outputs with revised and more complete deaths figures. I.e. almost all official statistics from Irish Dept. of Health are based on the incomplete figures released by the Central Statistics Office as "final", 22 months after year's end. (These are the totals you have been using from https://data.cso.ie/table/VSD39)

Your paper on International Completeness of Death Registration shows your awareness around the issue. For example, in Ireland the percentage completeness of the "final" figures had been falling 2011-2019 from ~99% to ~97%. Of course, this becomes more significant when estimating excess mortality and also when analysing trends in younger age groups as "late registered deaths" are disproportionately younger.