Closed figueroa1395 closed 1 week ago
At this moment, the columnar based batch data is validated by converting the full batch to row based, which may have a big memory overhead. However, once #699 is finished, this will be upgraded to do the conversion per individual batches.
Issues
1 New issue
0 Accepted issues
Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code
Merging now and doing the incremental upgrade to per-batch scenario validation later. This way the functionality is already provided to the users.
Update validation functions to support columnar data:
validate_input_data
function.validate_batch_data
function.test_input_validation.py
test_batch_validation.py