activewarehouse / activewarehouse-etl

Extract-Transform-Load library from ActiveWarehouse
MIT License
240 stars 102 forks source link

Added XLSX support, optional row validation, added SQLite insert_update_db #131

Closed tylergannon closed 11 years ago

tylergannon commented 11 years ago
  1. Added xslx support through Roo gem.
    • Roo appears to be more fully featured than Spreadsheet, and it supports XLSX files whereas the latter does not. Other than that it seems to be a drop-in replacement.
  2. Made row validation optional on ExcelParser.
    • I was running into issues where it's no big deal for my script to be smart enough to skip invalid rows but the whole ETL job would barf because one row was missing a field. putting :validate => false in the parser options will skip row-level validation.
  3. Added SQLite3 to the insert_update_database destination. It works.
  4. The changes to engine.rb are for ActiveRecord 3.2.x, which will not allow mass-assignment unless expressly enabled on the model. This change seemed easier than modifying the model class ETL::Execution::Job since I didn't know what might break if I change that class around.
thbar commented 11 years ago

Thank you @tylergannon!