finopsfoundation / focus_validator

Validator resource for checking datasets against the FOCUS specification
MIT License
15 stars 10 forks source link

New performance profiler and updates to test data generation #110

Closed johrenberger closed 10 months ago

johrenberger commented 10 months ago

test_performance_profiler.py - Constructed a new performance test leveraging cProfile to capture metrics, generates a polars friendly CSV and prints the first 10 lines in console. Shifted execution directly into the Validator class vs spinning up the app for more detailed output.

csv_random_data_generate_at_scale.py - experimented with polars and it was able to generate the test data much more efficiently. The code is a bit more compressed but I saw a nice performance boost in upfront test file generation.

Removed the following files as they are out of date: tests/fake_focuses.csv tests/focus_validator_performance_test.py tests/samples/csv_random_data_generator.py