As an advocate I want to be able to see details of the publishers of data and its provenance

robredpath commented 8 years ago

Satisfies #137 and #122

One set of users of GN are advocates of the standard. If they’re using their reputation, connections etc to promote it, they want to be assured that it’s good quality, or at least know what they can promise. They also want to be able to know who to contact if there's an issue with the data, or if they want to discuss it for any reason.

We should create publisher pages with:

details of the publisher,
a statement that says that issues with the data should be addressed directly to the publisher
the date of the publication of the data that we currently have loaded
which funders they publish data for
what manipulation (if any) we've carried out to the data since they published it

We should link to these publisher pages from each grant

robredpath commented 8 years ago

Issues with this: we don't currently store this data in GN!

robredpath commented 8 years ago

The source of truth for this data is Salesforce. Pulling this data in is part of the loading process. We should store it locally.

robredpath commented 8 years ago

We'd like to be able to load the data programmatically - pull the list from Salesforce, get all the data from publishers, load it into GN along with provenance info from Salesforce. However, some publishers are providing invalid/low-quality data that we're currently manually correcting, which precludes this. I believe @stevieflow is working on this, but we need a direction on it before we can start work.

robredpath commented 8 years ago

Implementation: pull provenance data from salesforce API, add to views etc as necessary

robredpath commented 8 years ago

We need to create publisher pages which are linked to from publisher pages. No need for these to have grants tables, though, just contact info and last published!

stevieflow commented 8 years ago

Re: date last published - I dont think this will also be collected (there's the dateModified in the data) - but a data package in Salesforce will have an update log (NB; the date we update the Salesforce record may not be the same date that a file was updated).

We know from work elsewhere that the "last updated" issue is tricky. We might want to avoid that in this iteration

robredpath commented 8 years ago

We should log how many records we drop due to non-compliance during loading, but other than that, we're good to go ahead and pull data straight from salesforce / publishers

Bjwebb commented 8 years ago

Pulling data from salesforce etc. isn't GrantNav specific, so I've been working on this in a separate repository https://github.com/ThreeSixtyGiving/datagetter

robredpath commented 8 years ago

We should talk to 360 to establish contact details for all the publishers that we currently have

robredpath commented 8 years ago

Happy for 'when was this data published?' to be 'when did we get this data?'

robredpath commented 8 years ago

@stevieflow I think this requires your attention :)

stevieflow commented 8 years ago

I've talked to 360 team about https://github.com/OpenDataServices/grantnav/issues/166#issuecomment-229664530

We will not have / publish specific names/contact details

We do have accessURL in Saleforce / JSON - which will be a link back to the publisher website to give more info (NB: this may not be recorded for all current publishers, but we could add a URL back to their site/contact page anyway)

Hence, I would suggest that for publisher contact details we construct a sentence / statement that looks something like.

This data was published by publisher/name. More info available on their [link]accessURL website [/link]

robredpath commented 8 years ago

@stevieflow this came up again on Weds. Now that Katherine's in post, the feeling was that it was worth revisiting the idea of maintaining in Salesforce a list of email addresses for data questions to be directed to.

The concern behind that was that whoever picks up the info@ email address might not know what to do with a comment about the accuracy of a grant record, and it takes away our opportunity to send a copy of the offending grant record with the request - it's a bit of a trek from a GrantNav page into your own FMS to find a grant, if the person even knows what the request is about, which is a pretty poor experience for both sides.

stevieflow commented 8 years ago

Hi

As in https://github.com/OpenDataServices/grantnav/issues/166#issuecomment-231390476 - i / 360 are not sure a named contact / email is the route to progress right now.

In tandem, there is a route to work with publishers to be clear about the feedback loop and process they have, or need to put in place - but I dont think that will be universally in place immediately.

So - with that in mind - have we anything we can do here to declare the source of the data and point the user in the right direction?

robredpath commented 8 years ago

Currently outstanding:

[x] collecting data that we don't currently, ie download date
wiring it all up
- [x] links to publisher pages from grant pages
- [x] adding publisher name, data source and retrieved date to downloads
[x] add publisher license and additional license fields to every row in downloads - subject to @Bjwebb 's discretion
Make sure that we have accessURL for all publishers - this should be a page. Publishers should be encouraged to put appropriate info on this page [ split out into https://github.com/OpenDataServices/grantnav/issues/254 ]

robredpath commented 8 years ago

Will resolve #191

Bjwebb commented 8 years ago

Make sure that we have accessURL for all publisher

Note that accessURL is actually a property of a dataset, rather than a publisher, which could cause problems if we want to use it as a "contact this publisher" link. At the moment the accessURL is generally the same for all of a publisher's datasets though (but I'm not sure if we can rely on this).

robredpath commented 8 years ago

That's worth noting, thanks @Bjwebb . If it doesn't affect us now though, I think we can safely ignore it :)

Bjwebb commented 8 years ago

On the branch, we now have a publisher page http://acceptable-license-valid.166-publisher-pages.grantnav-dev.default.opendataservices.uk0.bigv.io/publisher/360G-LBFEW linked to from a grant, and a datasets page http://acceptable-license-valid.166-publisher-pages.grantnav-dev.default.opendataservices.uk0.bigv.io/datasets/ which I've titled "Data used in grantnav" and added a preamble about non-360Giving data to the top.

Still remaining is work on downloads.

Bjwebb commented 8 years ago

Other outstanding tasks:

[x] ATM, every 360Giving dataset is listed on the datasets page. It would be more informative to list only those that are included in a particular deploy.
[x] Add information about the license of the non-360Giving datasets
Provenance information in the JSON download [ spilt out to https://github.com/OpenDataServices/grantnav/issues/253 ]
Create a script to check some of the assumptions we're making for GrantNav (e.g. that the accessURL is the same for all datasets by a publisher) [split out to https://github.com/OpenDataServices/grantnav/issues/252 ]

Bjwebb commented 8 years ago

Internal titanpad for working on some of this page comment.

Outstanding changes:

[x] Publisher pages should list funders they publish for
[x] Publisher pages should list prefix
[x] Copyright / Attribution headers
[x] How we present link to publisher on grant page - move to bottom, put in "box"
[x] How we present link to accessURL on publisher page; "GrantNav doesn't own any of the data... [publisher] has provided the following URL for information about this dataset:"/box titled contact information; flagging when we don't have it
[ ] Don't list publishers / datasets that aren't in the data due to conversion failing.

Bjwebb commented 8 years ago

The work so far has been merged.

Additional tasks now have their own issues, so closing this one.

ThreeSixtyGiving / grantnav

As an advocate I want to be able to see details of the publishers of data and its provenance #166