GrantNav JSON download isn't valid 360Giving data

[robredpath]

Describe the bug The GrantNav search results download button for JSON doesn't result in valid 360 data being returned.

To Reproduce Steps to reproduce the behavior:

• Do a search in GrantNav. I did a blank search, then filtered to CAD currency to get a small data set
• Click the JSON button at the top of the search results
• Upload the resulting file to the DQT

Expected behavior CoVE validation to pass

Screenshots https://www.loom.com/share/f98e42665c454620b7be728f5b37e5aa

Desktop (please complete the following information): Latest FF on macOS

[mariongalley]

Was discussing this with @KDuerden and Tania the other day. This is likely because of the explanatory note being too long, which causes issues in both CSV and, we think, JSON.

We're thinking of developing a 'simple download' which could potentially have more explanation within the sheet, in which case we might be comfortable with it not passing validation, but generally this is something we want to fix.

The main reason it was important previously was that you could download any dataset from GrantNav and upload it into Insights to visualise. This is no longer as important now that you can't upload a file to Insights and that you can open a GrantNav search of < 10,000 results in Insights, but it was useful for publishers to check their unpublished data looked right, so if we want that to be possible again we'd need to bring back the Insights upload.

[robredpath]

I'm not sure about the CSV, but in the JSON it's because the license has to be a uri but in the download we put a free-text license statement in:

❯ cat grantnav-20230214092102.json | jq
{
  "license": "See dataset/license within each grant. This file also contains OS data © Crown copyright and database right 2016, Royal Mail data © Royal Mail copyright and Database right 2016, National Statistics data © Crown copyright and database right 2016, see http://grantnav.org/datasets/ for more information.",
  "grants": [
...

[robredpath]

I think each grant object is valid against the grant schema, but the package isn't valid against the package schema.

[robredpath]

Interestingly, the CSV is valid because there isn't a License column but a License (see note) column, which gets detected in CoVE as an additional field

[robredpath]

The main reason it was important previously

Yeah, I don't know how important this is to fix, in reality. My interest is mostly on principle rather than pragma (if any tool in the world should create valid 360 data it's GrantNav!).

The one potential practical point is that if there was someone wanting to start developing software with 360Giving data, the GN JSON download may well be where they would start, to understand what the data looked like and what they might be able to do with it - even before getting in touch with the 360 team. I suspect that doesn't happen often!

[michaelwood]

We've decided this isn't something we're going to do as the fields which are included in the downloads are going to user selectable and we don't think we need to prevent people from making non-standard data with this feature - e.g. removing grant ID field or other required fields