Export OpenML data to data package, Import data package to OpenML

[HeidiSeibold]

Would be nice if we could do that. Would make it much easier for people to upload data to and to work with data from OpenML.

Data packages are defined by:

• a csv file with the data
• a json file with thte meta data

Why?

To improve user friendliness. See also: https://docs.google.com/document/d/1c_RhDiXTK5bEsY5gGRuQwaF6fKilt4jKq2c_BRqyEDc/edit?usp=sharing

How is meta data specified in data packages?

https://specs.frictionlessdata.io/data-package/#metadata

This is related to #457

[HeidiSeibold]

To be able to import data packages into OpenML I think we need to first do the following steps:

• [ ] Find an interesting ML data set that is stored as a data package (e.g. on http://datahub.io, maybe https://datahub.io/core/population?)
• [ ] Read the data with a tool that can also upload data to OpenML or convert data to ARFF (e.g. R with farff or RWeka, note that R has problems with some data sets on datahub.io atm https://github.com/datahubio/qa/issues/31; python also has ARFF support)
• [ ] Upload the data to OpenML or export the data to ARFF
• [ ] Prepare a script that does all of the above and share it in this issue

Any help on this issue would be very appreciate :clap: :cake:

[pwalsh]

Copying my response from gitter.im at request of @HeidiSeibold

---

I note there are a few libs in Python for ARFF

• https://pypi.python.org/pypi/liac-arff
• https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.arff.loadarff.html
• https://pypi.python.org/pypi/arff

And we have a documented way to convert to/from data backends here:

https://github.com/frictionlessdata/tableschema-py#storage

And some example implementations of the storage API at:

• https://github.com/frictionlessdata/tableschema-sql-py
• https://github.com/frictionlessdata/tableschema-bigquery-py
• https://github.com/frictionlessdata/tableschema-elasticsearch-py

So writing an ARFF backend would be great!

[HeidiSeibold]

The issue https://github.com/datahubio/qa/issues/33 is in prinicple the same just the other way around. Both are equally helpful and important :smiley:

[joaquinvanschoren]

Interesting dataset: Maybe this one is a good place to start: https://datahub.io/anuveyatsu/farm-survey-simple

Nice thing is that they have all the attribute file types, offered as a JSON file. Should be easy to convert to ARFF. What is still missing is the task, i.e. what you want to predict. There is also no description of what the dataset is about.

I also couldn't figure out how to navigate DataHub. There are apparently 200+ datasets but I can only see a few of them on the website.

[HeidiSeibold]

I also couldn't figure out how to navigate DataHub. There are apparently 200+ datasets but I can only see a few of them on the website.

That's a bug https://github.com/datahubio/qa/issues/32

[joaquinvanschoren]

Feedback from the frictionlessdata gitter:

• The Table Schema spec is used for column types. https://frictionlessdata.io/specs/table-schema/ Table Schema is used to described Tabular Data Resources ( https://frictionlessdata.io/specs/tabular-data-resource/ ), and, Tabular Data Package is used to represent a collection of Tabular Data Resources ( https://frictionlessdata.io/specs/tabular-data-package/ )
• In the Python and Javascript implementations, we have an infer method on the Table class: - https://github.com/frictionlessdata/tableschema-py/blob/master/tableschema/table.py#L139 Which can also be called as a function on any supported data source: - https://github.com/frictionlessdata/tableschema-py/blob/master/tableschema/infer.py#L14 The infer algorithm is simple - I wouldn't rely on inferring types/formats only. The main use case is the get bootstrapped with data quickly, and possibly manually tweak an inferred schema for accuracy.

[HeidiSeibold]

There are now some machine learning data sets available as data packages: http://datahub.io/machine-learning

Example: http://datahub.io/machine-learning/seismic-bumps Also available on OpenML: https://www.openml.org/d/1500

I guess a first step now would be to check:

• [ ] which fields in the datapackage correspond to which fields in the ARFF file / XML file.
• [ ] if there is any information in the ARFF file / XML file that we cannot get from the data package.

See also discussion https://github.com/datahq/datahub-qa/issues/33#issuecomment-356368024

[HeidiSeibold]

We decided ot wait until https://github.com/frictionlessdata/datapackage-r/issues/13 is properly solved.