Updates on ODF & open standards for publishing

[edent]

Just a quick update on the adoption of ODF (and other open formats) for publishing on GOVUK. Attached is a CSV showing how many open format and closed form documents have uploaded over the last two years.

Important notes:

• Departments can "dual-publish" - some will upload an ODS spreadsheet and also an XLS version. Or an ODT and a PDF. This is driven by their customer need.
• Open formats here include ODF, PDF, CSV, plain text & plain XML.
• Closed formats here include docx, doc, xlsx, ppt, pptx and similar.

Total attachments

This shows the trends in publishing over the last two years. Each data point is how many attachments were published that month.

Observations

• Publishers are increasing publishing directly as HTML. This has lead to the general decline in new attachments.
• Open Formats are now more popular than closed formats.
• In total, ~40% of all published documents are in an open format.

Open Vs Closed

This graph shows the top 50 Departments by number of attachments. There is a long-tail which is not included in this image, but which is included in the data. (Click the image for a larger version.)

Observations

• Very few departments only publish closed format documents.
• Most departments publish a mix - this may be due to dual publishing.
• There are many departments who publish the majority of their content in open formats.

Conclusions

It is encouraging to see open formats gain in popularity - although there is still some way to go. We are making changes to the publishing process to make it clear that open formats must be published.

User feedback has generally been good - although some users with specific software needs still struggle with ODS.

Data

Here is a CSV of the data: Open Vs Closed format attachments by organisation and month.csv.zip

[RKHayden]

Is there similar data for downloads of documents? That would show to what extent the supply of documents in an open format is serving a user need. That would particularly informative for those cases where organisations dual-publish in open and closed formats.

[edent]

I don't know if we routinely collect that data. Let me check and get back to you.

[edent]

@RKHayden Here's some data we've collected regarding downloads of documents.

• This covers 2018-01-01 to 2018-03-31
• It is based on Google Analytics reporting on number of times a download link was clicked
• This should be thought of as indicative data. I've rounded the number to avoid the suggestion that this is precise data
• This doesn't specifically look at dual publishing
• As you can see, PDF dominates. I've included a graph showing the open/closed/other split if you ignore PDF.

Extension	Unique Files	Total Downloads
pdf	121,000	21,930,000
csv	5,000	722,000
doc	4,000	674,000
xlsx	6,000	671,000
docx	3,000	372,000
ods	4,000	293,000
xls	5,000	273,000
zip	1,300	221,000
odt	600	114,000
xlsm	300	67,000
rtf	300	48,000
pptx	100	11,000
jpg	200	8,000
ppt	100	5,000
png	80	5,000
eps	30	2,000
txt	70	1,500
gif	30	600
xml	60	600
xsd	40	400
xlt	2	100
odp	10	100
ps	4	10
rdf	10	10
dot	1	3

CSV is the most popular data format. Anecdotally, we know that XLS/X/M files are often used as ersatz data entry forms. The .DOC files mostly appear to be older content which haven't been updated in a while. .ODT is used about a third of the time compared to .DOCX.

The .ZIP files sometimes contain open formats, sometimes closed. The RTF content mostly appears to be legacy forms.

We are working on:

1. Finding which publishers don't offer a choice of open formats
2. Finding owners of high-traffic legacy content to see if it can be updated

[RKHayden]

@edent Interesting to see that, assuming there were no files that were never downloaded and thus excluded from the Google Analytics data, .ods is being made available much more than .odt (and that's despite .csv being an alternative open format for tabular data): 600 .odt unique files versus 7,000 .doc and .docx unique files whereas there are 4,000 .ods unique files versus 11,000 .xls and .xlsx unique files.

The following analysis may not be wholly valid (it may be skewed by the popularity of particular pages), but seems to indicate that the .odt formats does well where it is available. The .ods format, although much more widely available on gov.uk, seems less popular.

Extension	Unique Files	Total Downloads	Average downloads per unique file
xlsm	300	67,000	223
odt	600	114,000	190
pdf	121,000	21,930,000	181
zip	1,300	221,000	170
doc	4,000	674,000	169
rtf	300	48,000	160
csv	5,000	722,000	144
docx	3,000	372,000	124
xlsx	6,000	671,000	112
pptx	100	11,000	110
ods	4,000	293,000	73
eps	30	2,000	67
png	80	5,000	63
xls	5,000	273,000	55
ppt	100	5,000	50
xlt	2	100	50
jpg	200	8,000	40
txt	70	1,500	21
gif	30	600	20
xml	60	600	10
xsd	40	400	10
odp	10	100	10
dot	1	3	3
ps	4	10	3
rdf	10	10	1

As an aside, I wonder if there is a presentation issue on pages where downloads are offered. Just browsing through pages of one department (BEIS) I notice that, except for PDF, the open formats appear labelled as an 'alternative format' and below the proprietary format. And there's nothing to explain where the 'alternative format' might be usable. (For example this page). The link that's provided to Wikipedia is unlikely to help many users.

[edent]

Thanks - we're also working on updated guidance, so people know what the file formats are. I'll leave this issue open for now - but might also do a blog post about it.

[edent]

Updated guidance has been published on https://www.gov.uk/guidance/how-to-publish-on-gov-uk/creating-and-updating-pages#adding-attachments

Makes it clear that only open formats are acceptable. We're now monitoring and will see if it makes a difference.

[KatieAtherton]

Hi Terence,

We (MoJ PDC) have sent out the email this morning to colleagues to move to publishing spreadsheets (other than pivot tables) in .ods, from their next publication, off the back of the cross-govt meeting we attended.

There is still a little hesitation about the issue that .ODS is not compatible with a lot of software (e.g. apple), so I was wondering whether there are any updates on this. I had in my notes that the guidance would assist the user with any compatible software they can use as per their platform, (but then what if they needed to send the document to someone on a different platform?), GDS were in talks with Apple themselves, and that potentially, we could upload in .ODS and GDS can do the conversion (e.g. to .xls) for us.

We have considered publishing in both ODS and XLS formats but to do so would introduce additional QA work and additional risk of errors between the documents.

Let me know if it's easier to chat by email/phone.

MoJ PDC

[edent]

Hi @KatieAtherton. We will shortly be publishing an improved list of software that people can use to view/edit ODS on their computers, phones, and tablets. I hope that will be this week.

We continue to chat with Apple to see if Numbers can support ODS. If not, users will have to use alternate software. This is the same situation as MS Windows where people have to install an office app to view and edit documents.

[edent]

@KatieAtherton just a quick update. We're changing the links which appear under ODF so they point to the guidance, not Wikipedia - see https://github.com/alphagov/whitehall/issues/3962

The updated guidance is coming soon. @Lawrence-G is finishing it up now.

[edent]

We've blogged about this at https://gdstechnology.blog.gov.uk/2018/04/27/open-document-format-in-government-an-update/

[fweng322]

Keep going! The most important thing is to communicate with users so that they can understand why to use ODF. We've had maybe more than 100 big and small sessions with end users. IMO such communication would work. People would be easier to accept this policy and less resistance when learning and using tools like LibreOffice.

[edent]

A quick update. Will post in more details when we have more data. Specifically number of files uploaded.

• This covers 2018-05-15 to 2018-08-15
• It is based on Google Analytics reporting on number of times a download link was clicked
• This should be thought of as indicative data. I've rounded the number to avoid the suggestion that this is precise data
• This doesn't specifically look at dual publishing
• This cannot be directly compared to the previous figures. A slightly different methodology was used (I wanted to use R rather than Python). Slightly different time frames used as well - certain files get more downloads at specific times of year.
• Remember, indicative data.

Extension	Unique Files	Total Downloads
pdf	153,000	22,789,000
doc	4,000	772,000
csv	5,000	684,000
xlsx	6,000	604,000
docx	3,000	403,000
ods	4,000	338,000
xls	5,000	230,000
zip	1,000	198,000
odt	1,000	181,000
xlsm	300	64,000
pptx	100	19,000
rtf	300	18,000
jpg	200	8,000
ppt	100	4,000
png	100	3,000
txt	70	1,000
eps	30	1,000
xml	50	1,000
gif	30	900
xsd	40	500
odp	6	100
xlt	2	20
ps	8	20
od	1	10
rdf	3	10

[stevenjmesser]

Do we know the direction of travel for open formats? I can see that CSV stayed the same...

[RKHayden]

The one figure I pick out from that data is that the number of odt format unique files has significantly increased - from 600 to 1000 - which I would presume is big enough not to be an attribute of the difference in methodologies nor of the rounding. Except pdf, all the other popular formats have remained static.

[edent]

Big news! I've added a bunch of statistical data to https://github.com/alphagov/open-standards/tree/master/File%20Formats%20Statistics and some R scripts to process them.

Here's what I found. Note, these data have not been validated by anyone else, you should carefully check them. PDFs have been excluded from the graphs but are present in the data - see last graph.

We track every file which is published to GOV.UK - here is the cumulative total for each major file type.

CSV well in the lead. ODS rising steadily. Still some departments using RTF and other obscure file formats. ODT nowhere near doc/docx yet.

That also allows us to get the filetypes which each organisation publishes:

We also track downloads. It's a pain to get them at a daily level, so here are the quarterly download stats. Interesting to see that downloads a generally dropping, with the exception of CSV.

Very roughly, here are the rate at which open formats and closed formats are uploaded to the site (excluding PDFs): We allow dual publishing

Finally, I said we excluded PDFs from these graphs, here's the cumulative file uploads when we include PDF:

The huge number of PDF attachments and downloads dominates everything else. Something we're working on. See https://gds.blog.gov.uk/2018/07/16/why-gov-uk-content-should-be-published-in-html-and-not-pdf/

So, please play with the data and let us know anything interesting or useful you find.

If these data releases are helpful, I'll find a way to release them more regularly.

[RKHayden]

For each file in "Attachments to GOV.UK 2017-09-01 - 2018-09-13.csv" I have found the number of downloads for it listed in "Downloads from GOV.UK 2017-09-01 - 2018-09-13.csv".

From that I can calculate, for each department, the average downloads per unique file added to GOV.UK between 2017/09/01 and 2018/09/13. This should give some indication (no more than that) of whether making files available in formats that follow an open standard is serving a user need or not: if within one department (especially those bodies with a relatively narrow remit) download rates for open format are high, there's a clear user need; if they are very low, compared with the proprietary formats, then it suggests there isn't. Overall, it looks like there is a reasonable demand for the open formats: their download rates seem relatively high, considering the market dominance of one proprietary format.

In the table below I have only included the Open file formats and their proprietary equivalents. That excludes mainly image files and zip files. Figures are rounded to the nearest whole number. I've also excluded PDFs as the discussion about their use is a somewhat separate issue.

In doing this analysis I've noticed there is double counting for organisations with both English and Welsh names, for example Companies House / Tŷ'r Cwmnïau. I've not attempted to remove those duplicates in the analysis.

Organisation	doc	docx	rtf	txt	odt	xls	xlsm	xlsx	csv	ods	ppt	pptx	odp
Administration of Radioactive Substances Advisory Committee		526
Adran Gwaith a Phensiynau	145	177	78		473	20	229	179	50	136
Advisory Committee on Business Appointments	286
Advisory Committee on Clinical Excellence Awards	51	507	1080					1747		1597
Animal and Plant Health Agency	1012	4618			477	36		6	330	28
Armed Forces' Pay Review Body								359		109
Asiantaeth Iechyd Anifeiliaid a Phlanhigion	1012	4618			477	36		6	330	28
Asiantaeth Trwyddedu Gyrwyr a Cherbydau						0		41	42	143
Attorney General's Office	197	30		0		2		12	3
Bona Vacantia	1874	4410							12584
Border Force						4438				165
British Cattle Movement Service											130
British Hallmarking Council		9
Cabinet Office	67	410			119	70		153	100	50		81
Care Quality Commission	232	340						231
Central Arbitration Committee	6	16
Centre for Connected and Autonomous Vehicles								9
Centre for Environment, Fisheries and Aquaculture Science	974	44				29
Certification Officer	24	126
Chevening Scholarship Programme									1173
Civil Aviation Authority		137							0
Civil Nuclear Constabulary	0							6
Civil Nuclear Police Authority	0							6
Civil Procedure Rule Committee	13
Civil Service	94	423					128	45			715	358
Civil Service Fast Track Apprenticeship		403						1003
Civil Service Resourcing		870
Coal Authority		3			17			10	17
Commission on Human Medicines		77
Committee on Standards in Public Life	78	49						12
Companies House	9581							347	2231
Company Names Tribunal		37	14
Competition and Markets Authority		926			79	278		218		27	14
Copyright Tribunal						3
Criminal Injuries Compensation Authority								187
Crown Commercial Service	0	392			196	370		293	31	1020
Defence and Security Accelerator	0	55			120
Defence Equipment and Support	32
Defence Safety Authority	14	15
Defence Science and Technology Laboratory	8	156						47
Department for Business, Energy & Industrial Strategy	34	230	2574			323	410	150	16	63	2	56
Department for Business, Innovation & Skills	10	8					458	18	28
Department for Digital, Culture, Media & Sport	196	165	19		7	27	0	59	21	23		13
Department for Education	166	510		63	644	282	714	1036	77	405		743
Department for Environment, Food & Rural Affairs	145	1428			399	158	783	358	83	81		112
Department for Exiting the European Union		390						55	184			0
Department for International Development	11	72			332	1		83	6	12
Department for International Trade	14	463			408			17	7		0
Department for International Trade Defence & Security Organisation								92	5
Department for Transport	145	222	18	166	99	63	843	147	46	80	96
Department for Work and Pensions	145	177	78		473	20	229	179	50	136
Department of Health and Social Care	4437	1418	713		795	158	44	421	18	230
Disclosure and Barring Service	2344	1097			443	1064		774	114	102
Driver and Vehicle Licensing Agency						0		41	42	143
Driver and Vehicle Standards Agency	224	2330			1360	1042	0	7	263	354		4377
Driving Standards Agency						0			0
Education and Skills Funding Agency	45	464		30	1387	77	1819	648	166	1409		666
Education Funding Agency		3			1			22				0
Efficiency and Reform Group								69	97
Environment Agency	1	79			180	41	604	101	688	150	17
Family Procedure Rule Committee	4
FCO Services									4	2
Food Standards Agency	974					29		1401
Foreign & Commonwealth Office	149	247			97	18		11	78	195		241
Forensic Science Regulator										38
Forestry Commission	60	4				37	1354	326	3	591	0
Gangmasters Licensing Authority									4	2
Government Actuary's Department								0	3
Government Chemist		1
Government Digital Service		672			174				218	7			39
Government Economic Service	2827	587
Government Equalities Office		1005				19				143
Government Legal Department	101					2		5	4
Government Office for Science		47
Government Social Research Profession		552
Groceries Code Adjudicator		24			38						0	0
Gwasanaeth Llysoedd a Thribiwnlysoedd EM	215	331				750			53
Gweinyddiaeth Cyfiawnder	287	13	1060		58	213	19	127	4	320	12	119
Health and Safety Executive				0
Health and Social Care Information Centre		6						7
Her Majesty's Prison and Probation Service	63	82	1060		58	426		119	13	562		226
High Speed Two (HS2) Limited							84	394				6
Highways Agency								0
Highways England	26	45			40			46	28
HM Courts & Tribunals Service	215	331				750			53
HM Crown Prosecution Service Inspectorate				0		2		8	3
HM Inspectorate of Constabulary									2		0
HM Inspectorate of Constabulary and Fire & Rescue Services	389	255						231	2		0
HM Inspectorate of Prisons	75
HM Inspectorate of Probation	389	255						231
HM Land Registry	1750	1150		0				15	15
HM Passport Office		727							253	523
HM Prison Service		4	1060		58	437		120		604
HM Revenue & Customs	161	935		20	791	115	98	219	6762	1108
HM Treasury	125	247			2211	95	193	737	69	80
Home Office	714	423			250	16	181	327	129	163
Homes and Communities Agency	8	353				33	4228	578	27
Homes England	100	305					1110	2400	34
Immigration Enforcement		5			18	1				143	158
Independent Office for Police Conduct										65
Independent Police Complaints Commission									163	16
Independent Reconfiguration Panel	209
Industrial Injuries Advisory Council		0	2
Infrastructure and Projects Authority		163			504	18		225	24	554
Infrastructure UK						6		32
Innovate UK								77	13
Intellectual Property Office	58	88	59		3759	14		49	13			6
Legal Aid Agency	390	385			666	1875	693	614		115	19
Low Level Waste Repository Ltd	4	26				3	23	42
Low Pay Commission		36						1276
Marine Management Organisation	73	11			4	100	52	44	8	21
Maritime and Coastguard Agency	39	129			67	185		109	35	3
Medicines and Healthcare products Regulatory Agency	222	318			246	180		264	53		0	7729
Migration Advisory Committee	770							2
Military Aviation Authority						11
Ministry of Defence	259	629			307	45	24	109	74	34		289
Ministry of Housing, Communities & Local Government	290	2374			375	209	538	305	23	759
Ministry of Justice	287	13	1060		58	213	19	127	4	320	12	119
Monitor		10					10		11
National College for Teaching and Leadership	0	97			15	423		2940		463
National Counter Terrorism Security Office	650
National Offender Management Service		4	1127		58	354				13
National Probation Service	0										0
National security and intelligence	86				15	513				96
Natural England	118	381			54	51	1723	263		157
NHS Digital						20		350		215
NHS England		3					10
Northern Ireland Office	54	5					17	6		30
Northern Lighthouse Board									74
Nuclear Decommissioning Authority	21	18			2	21		14		29
Office for Civil Society		446							1
Office for Disability Issues					9
Office for Life Sciences								10
Office for Low Emission Vehicles		231			191			130	830
Office for Product Safety and Standards	128	151				30		98			10	83
Office of Financial Sanctions Implementation		532						135
Office of Manpower Economics		85						359		109
Office of Rail and Road										47
Office of Rail Regulation										96
Office of the Advocate General for Scotland	71	71						11	3
Office of the Chief Electoral Officer for Northern Ireland		6
Office of the Immigration Services Commissioner	104								67	25
Office of the Leader of the House of Commons	2	13
Office of the Regulator of Community Interest Companies	2503	2524							116
Office of the Schools Adjudicator	140	532			52
Office of the Secretary of State for Scotland		24					15	12	8
Office of the Secretary of State for Wales		0				4	9	5	3
Official Solicitor and Public Trustee	2415
Ofqual		1						123	12	194		386	40
Ofsted	497	207				93	367	364	47	204	986	1037
Parole Board	59	55
Planning Inspectorate	102	232			74	1		39	5	33
Police Remuneration Review Body		7
Prime Minister's Office, 10 Downing Street		2437							338
Prison Service Pay Review Body								359		109
Public Health England	144	147		0	196	39	447	189	6	70	9	33
Pubs Code Adjudicator		62										0
Queen Elizabeth II Conference Centre									9
Radioactive Waste Management						0		1
Rail Accident Investigation Branch	10
Regulator of Social Housing								4242
Royal Parks									5
Rural Development Programme for England Network		649				1101
Rural Payments Agency	195	129				592		2	10
Security Industry Authority									63	12
Sellafield Ltd	36	122						339
Senior Salaries Review Body		240
Single Source Regulations Office						5	171	6		34
Skills Funding Agency						53		1240	79
Social Mobility Commission								6574
Social Security Advisory Committee		0
Sport England		3478
Standards and Testing Agency		12704		0	10560			1571		3739
Student Loans Company								24		12
Surveillance Camera Commissioner		0
Swyddfa Ysgrifennydd Gwladol Cymru		0				4	9	5	3
Teaching Regulation Agency		1086			20	2013
The Charity Commission	1246	381			608	5592		356	21
The Insolvency Service	65	14			17	0		234	11	51
The National Archives									5
Traffic Commissioners for Great Britain	1		0						17
Transport Focus									74
Treasury Solicitor's Department				0				0	0
Tribunal Procedure Committee		175
Trinity House									74
Tŷ'r Cwmnïau	9581							347	2231
UK Anti-Doping	35	24
UK Atomic Energy Authority									4
UK Commission for Employment and Skills								24		2		4
UK Export Finance						3	271	12	11	6
UK Space Agency	36	90			85			49	39
UK Trade & Investment	7
UK Visas and Immigration	133	5142			662	4			44	266	158
Valuation Office Agency		46				487		861	52
Vehicle and Operator Services Agency						0			1
Vehicle Certification Agency								2	10	4
Veterinary Medicines Directorate	75	13			0	9					16	37
Welsh Government	0
Youth Justice Board for England and Wales	34					75				81