Parse Popular Dataset File Formats

[ct-clmsn]

Provide customized file readers for the following formats to assist with algorithm debugging. This is an arbitrary ordering.

• [x] Comma-Separated Values (CSVs) (done via #93)
• [ ] Matrix Market
• [ ] NumPy text storage format
• [ ] NumPy binary storage format
• [x] HDF5
• [ ] C++ Hypertable library

Review these formats and assess the level of effort and potential prioritization. Note, some might not be worth the effort implementing. May need to contact some projects to assess partnership opportunities.

• Review scikit-learn formats, select candidates
  • http://scikit-learn.org/stable/datasets/index.html
• Pandas dataframe formats
  • https://pandas.pydata.org/pandas-docs/stable/io.html
• Pandas dataframe frame type description
  • https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html
• Apache Arrow
  • https://arrow.apache.org
• Apache Parquet
  • https://parquet.apache.org
• SVM-Light
  • http://svmlight.joachims.org/
• Attribute-Relation File Format (ARFF)
  • https://weka.wikispaces.com/ARFF
  • https://www.cs.waikato.ac.nz/ml/weka/arff.html

[hkaiser]

Very nice, thanks! @hapoo this might be something for you!

[ct-clmsn]

@hkaiser @hapoo glad to hear it will work as a starting point for future conversations.

[parsa]

From Phylanx project mailing list (edited, condensed):

Permalink From: @ct-clmsn To: Phylanx Project Subject: Data Formats

There are a couple additional formats the team may want to consider or look into supporting.

• LMDB - Caffe
  • http://caffe.berkeleyvision.org/tutorial/data.html
  • https://github.com/intel/caffe/wiki/How-to-create-LMDB
• Tensorflow records
  • https://www.tensorflow.org/programmers_guide/datasets
• Dataframes
  • https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html

On the topic of dataframe... This would be some type or version of dataframe. Not all of the capabilities but, maybe some minimized set of features (column select, filter, join). Some of these capabilities could be abstracted and applied to LMDB and TF records. This could motivate work on HPX algorithms.

---

Permalink From: @hkaiser

Thanks @ct-clmsn! @hapoo, do you copy?

---

Permalink From: @ct-clmsn

Also this library might be useful for dataframe baseline functionality.

• https://github.com/gpuopenanalytics/libgdf

It is a GPU dataframe package.

---

Permalink From: @hapoo

Roger that, I looked at caffe's LMDB, it seems very interesting and has capabilities similar and better than HDF5 for deep learning. I will look at the rest as well.

---

Permalink From: @ct-clmsn

Thanks! Would a 'table' abstraction between these formats simplify processing?