search

os-climate / DERA-ingest-pipeline

Apache License 2.0
0 stars 4 forks source link

Need a SQL test suite for SEC DERA data #7

Open MichaelTiemannOSC opened 2 years ago

MichaelTiemannOSC commented 2 years ago

We a test suite and automation process to ensure that when we change calculations for revenue, float, debt, cash, assets, income, etc., that we don't break things that are considered working.

@toki8 Feel free to assign as you see fit. @erikerlandson and @caldeirav Feel free to volunteer someone at Red Hat who could help build such an automated testing demonstration.

CC: @HeatherAck @caldeirav @erikerlandson

caldeirav commented 2 years ago

For this kind of test, my take is we should look at a test suite that runs well within our pythin-based pipelines environment. I think for this, a good candidate is: https://docs.greatexpectations.io/docs/

I understand it has been used successfully for OSDU and we may be able to get more information on best practices from John Archer on this.