BritishGeologicalSurvey / etlhelper

ETL Helper is a Python ETL library to simplify data transfer into and out of databases.
https://britishgeologicalsurvey.github.io/etlhelper/
GNU Lesser General Public License v3.0
105 stars 25 forks source link

Add warning to documentation that `load` assumes that all items have the same columns #146

Closed volcan01010 closed 1 year ago

volcan01010 commented 1 year ago

Summary

As an ETLHelper user, I want it explicitly pointed out that load assumes that all items have the same columns, so that I don't miss any when I load heterogeneous data.

Description

The load function peeks at the first row of data to identify the column names used to autogenerate the insert query. If later items in the data contain extra columns, those columns will not be inserted. The process will continue without raising any error.

I don't think there is an easy way to check for each item, but we should explicitly make this clear in the documentation.

Acceptance criteria