Include support for FrictionlessData

[justinclift]

Also pointed out by @danfowler, Open Knowledge International seems fairly well aligned with what DBHub.io will provide.

They have an initiative called "Frictionless Data", whose purpose is seamless data exchange.

For it, they've developed a file/packaging format which would be useful for us to support (import and export):

• http://frictionlessdata.io
• http://specs.frictionlessdata.io
• http://specs.frictionlessdata.io/data-package/
• http://specs.frictionlessdata.io/table-schema/

It looks useful, as I've been wondering how to provide license information and other metadata with downloaded databases. This seems like an existing solution for that problem which we can make use of.

Update - OKI seems to have decided to develop their own competing platform instead of working with us. Not real sure adopting FrictionlessData is a good idea any more.

[justinclift]

DCAT may be worth investigating as well.

[chrisjlocke]

Northern Ireland elections??

[justinclift]

Oops, bad cut-n-paste when tired.

Thanks Chris, link now fixed. :smile:

The Northern Ireland elections link was just from when mucking around with new sample data sources. eg: https://dev1.dbhub.io/justinclift/Assembly%20Election%202017.sqlite

The source for that example uses DCAT. It also includes database structure info (table + meanings with individual field names + meanings too) in the fuller length description, which at the moment we can't fully replicate well.

On that note, our version of Markdown doesn't yet support tables, but that might (hopefully) just be a short term problem: https://github.com/rhinoman/go-commonmark/issues/8

[justinclift]

Related, and likely relevant:

• http://5stardata.info/en/
• https://certificates.theodi.org/en/

They seem to be recommended as best practise by the data.nicva.org staff, so we should probably incorporate that too when we're up to it.

[justinclift]

As a future thought, we might want to add an option for downloading in JSON format too.

Unlike CSV, JSON directly supports null values, which might be helpful for people with NULLs in their data set and whom (for some unknown reason) can't use SQLite.

[justinclift]

And there's GeoPackage too, which might be pretty easy as it's already based upon SQLite.

• https://en.wikipedia.org/wiki/GeoPackage
• http://www.geopackage.org
• https://github.com/opengeospatial
• https://github.com/opengeospatial/geopackage

[justinclift]

The HXL standard seems interesting too: http://hxlstandard.org

It's very simple, and we could definitely support capturing that info (it's hashtag based) when we add support for importing CSV's. When we add search filtering/capabilities to the website too, they could be searchable data points.

... kind of wondering if it'd make sense for DB4S to recognise them too when importing CSV's. Not sure what it should really do with them apart from "don't add them to the row data". Maybe send them along to the DBHub server when uploading though?

[captn3m0]

There seems to be a lot of (open-source, public) adoption for frictionlessdata standards, and it would be nice to see them supported here as well.

OKI seems to have decided to develop their own competing platform

I couldn't find this platform, only a few external ones that were funded at https://frictionlessdata.io/adoption/. The ones where I've used frictionless in the past were all unaffiliated (like Zenodo).

[justinclift]

Interesting. Looks like that's another +1 for FrictionlessData then.

Supporting it will just depend on someone implementing it.

@captn3m0 How's your Go coding skill, in case you'd consider it? :smile:

[captn3m0]

Want to get used to the platform first, and probably pick up something smaller - like OGP support perhaps.

[justinclift]

No worries at all. :smile:

[justinclift]

Hmmm, what's the "OGP" there stand for?

Searching for that online is turning up all kinds of stuff. :smile:

[captn3m0]

Open Graph protocol. #138

[justinclift]

Cool. Yeah, that sounds like a good first thing, to get familiar with stuff. :smile:

[justinclift]

You've seen this already yeah?

https://github.com/sqlitebrowser/dbhub.io/wiki/Local-development-setup

That's what we're using for our local development purposes, and it's working pretty well.

On, you might notice a lack of proper database migrations. That's what I'm currently putting time into (yesterday, today, maybe tomorrow too), so that'll be fixed soon.

[justinclift]

Note that implementing database migrations was completed a few days ago, not long after that above comment. :smile: