BritishGeologicalSurvey / etlhelper

ETL Helper is a Python ETL library to simplify data transfer into and out of databases.
https://britishgeologicalsurvey.github.io/etlhelper/
GNU Lesser General Public License v3.0
104 stars 25 forks source link

Add benchmark test #148

Open volcan01010 opened 1 year ago

volcan01010 commented 1 year ago

tbc @rachelheaven

rachelheaven commented 1 year ago

Summary As a developer I want to run some performance benchmark tests so I can see what impact my changes have on speed of processing

Description

Several of the improvements to this codebase are intended to improve the speed of processing, or may unintentionally degrade the performance. Benchmark performance tests would help to monitor this. Add extraction / insert / transform of 10k, 100k and 1M rows to integration tests (which run against local database) and report timings.

Acceptance criteria