search

LBNL-ETA / Adapter

The Adapter software is developed at the Energy Efficiency Standards Department and it provides a convenient data table loader from various formats such as xlsx, csv, db (sqlite database), and sqlalchemy. Its main feature is the ability to convert data tables identified in one main and optionally one or more additional input files into database tables and Pandas DataFrames for downstream usage in any compatible software.
1 stars 1 forks source link

34 io pickle #35

Closed 0xd5dc closed 1 year ago

0xd5dc commented 1 year ago

These two IO methods will enable Python data read and write in pickle format.

lyralan commented 1 year ago

Thanks Zheng! I think it's generally a good idea to make adapter able to read/write more file formats, as db, csv, excel are not the most efficient ones as we play with very large data objects (especially with strings). I do agree with Evan that it'd be better if IO can incorporate pickle more consistently as before. But one step at a time is ok as well.

I also feel it's important to let people know what is being added to adapter, why pickle is selected (but not other formats), what the advantages are vs db/csv/excel, otherwise those who don't know much about pickle will continue using csv etc., those who know pickle will just use pd.to_pickle and pd.read_picke.