Build feature extraction [binning] and feature data I/O [HDF5]

[hermands]

Proposal is to:

• FCS subroutines
  • Add FCS_subroutines/[FCS_feature_extraction].py that bins processed data and stores in FCS attribute
  • Add FCS_subroutines/[FCS_features_to_HDF5].py that pushes feature data into HDF5 structure
• Big Matrix routines
  • Add [Process_features].py that loads/accesses HDF5 structure, reads/?loads single table in DB with meta/label/annotation data, can convert between different data formats, and ?facilitates regression/random forest/SVM
• Subcommands
  • Add subcommands/[prep_ML_analysis].py:
  • This would query db and return file list, loop through FCS files and push data from each into FCS_subroutines/[FCS_feature_extraction].py and FCS_subroutines/[FCS_features_to_HDF5].py
  • This would also run same query on db and return a single pandas table of annotations (case-level mostly). This single table could be pushed into an analysis-specific sqlite db OR would it be straightforward to push this table into the same HDF5 file as the data (Could imagine the ability to modify this table in HDF5 without changing the feature data [to allow flexibility in annotations]??)
  • Add subcommands/[run_ML_analysis.py].py
  • This method would load in HDF5 file (features and annotations) and perform various analyses. This would likely be just a wrapper for many different methods

[davidpng]

HDF5 does not appear to support databases all that well. We can try packing the individual tables into the container but it will need to be reconstituted through a SQL declaration. Please see: http://www.pytables.org/moin/FAQ#IsPyTablesareplacementforarelationaldatabase.3F and this concerning the h5py I am using: http://docs.h5py.org/en/2.3/faq.html#what-s-the-difference-between-h5py-and-pytables

[davidpng]

In response to your proposal: FCS subroutines

1. Add FCS_subroutines/[FCS_feature_extraction].py that bins processed data and stores in FCS attribute _This has been implemented via calling FCS.feature_extraction() which generates a new class called ND_Feature_Extraction. The sparse array holding this information can be accessed by ND_FeatureExtraction.histogram. Addition functionality concerning decoding the index to a specific bin has been built in. Would you prefer this 'histogram' to be a variable within FCS instead?

[hermands]

Sounds good. Histogram can be not in FCS if it is always going to be pushed to HDF5 and it pushes itself. It sounds like we now have two histogram functions (1D and multi-D), but that is probably fine as long as that's all we need.

[davidpng]

HDF5 hierarchy /database_info_version /query /case_list /data/"case_tube_idx"/data_array /data/"case_tube_idx"/indices /data/"case_tube_idx"/indptr /data/"case_tube_idx"/shape

[davidpng]

HDF5_IO renamed to Feature_IO, new class MergedData_IO inherit similar things such as push and pull dataframes. This now handles IO to two separate HDF5 files with different schemas and properties.