cew821
commented
11 years ago I am torn on this question, but it's a good one to discuss.
On the one hand, throwing every record into a flat JSON file creates a huge search and discovery problem. This is especially true given how many awesome, CKAN-friendly geospatial resources we have. For a simple example, take a look at the search results for "average income by state" on http://catalog.data.gov. Of the 1100+ results returned, 1038 (and most of the top results) are geospatial shape files, covering various geographies and years.
I'm guessing most folks making this query are looking for the state census survey data (the number one result on Google). I'm guessing very few users are looking for shape files.
The counter argument is that, yes, search and discovery is hard, but the only way to solve that problem is to put all our data out there, tagged with a solid common metadata format, and let the pubic/industry create portals to our data for various audiences / use cases.
I tend to think that for both machines and humans, it would be more useful to get a record called "US Geospatial Census Atlas" (illustrative title) that links to an awesome standards-based geospatial datastore full of great cuts of census data, instead of getting 1000 individual shape files from various geospatial repositories as a response to a query. For this reason, I think the hybrid approach proposed by @ddnebert could make sense as a way to find a good balance between quantity and quality.
ddnebert
MarinaNitze
mhogeweg
jpmckinney
@seanherron commented in #33 that "...we also have a deadline of November 9th for agencies to publish a ton of data..." The Implementation Guide says that we should...
"Conduct a zero-based review effort of all existing data. Give this effort a very short timeframe and the very specific goal of producing a simple list of all data assets within the agency. Stop at the due date rather than stopping at the 100 percent marker, which is very difficult to reach in a single pass. Repeat at regular intervals."
So, are we publishing a "ton of data," or are we working to produce a more usable resource of accessible and usable data assets?
@gbinal commented in #105 that "...we all know that NOAA and USGS are the two 900 lbs. gorillas when it comes to number of entries." Back when we did Geospatial One-Stop, in the earliest days of Data.gov, and now apparently in the current instantiation of what's much the same concept, we've had times where there are tons of records in whatever catalog is being aggregated from all this work. But how useful have those resources been to date? What kinds of use cases have we enabled from discovery to access to use leading to beneficial outcomes for society? How did the big aggregations of tons of records do something in and of themselves that no other pathway to the actual underlying data could accomplish?
Don't get me wrong - I fully believe in what we are doing. I do believe that we should end up with every data asset that the taxpayers (including me) have funded over all these years easily discoverable and accessible through aggregations like Data.gov and lots of other creative applications arising from this effort. We should put them out there and let smart technology and technologists come up with creative ways to narrow in on the useful from the stuff not appropriate for a particular inquiry.
However, on the data provider end, we have to make some choices about what "zero-based review effort" means. At least from our perspective in the USGS, we are trying to do something sustainable that will make "Repeat at regular intervals" mean, "as soon as data are released." We are committed to scientific integrity, validity, and excellence at every level of our organization. So, is it better for us to continue pushing every single metadata record that we can get to line up with the high-level and relatively scant POD implementation of DCAT JSON? Or should we put some amount of energy into trying to make sure that the records we do put into one or more (ref. #105) data.json files have viable and accessible distribution links, appropriate contact information, references to deeper metadata and associated scholarly publications, and the few other details such that when someone finds them they will have a pretty good shot at being able to put the data to use?
It's a balance between quality and quantity, for sure. My opinion is that balance could be characterized in a ratio like 3-1, quality over quantity. But that's my opinion. What's yours?
Just for fun, I posted a little survey (on my own time and not any official part of Project Open Data or in any way affiliated with the USGS) - http://www.surveymonkey.com/s/JGWV3NK (I'll post survey results somewhere and reference in a comment.) I'm interested in comments here and input to the survey. If you have anecdotes or artifacts that answer any of my use case questions on success (or failure) of big comprehensive cataloging efforts - past or current, I'd love to hear about them here.