Closed bhavaniravi closed 3 years ago
I think your current approach is a good one. I'm not sure how you could retrospectively validate data you've already yielded in any situation. Even in the simplest case where you are using a for
loop (and not using this library at all), you will have to either validate all the data first and then return it as one big list, or use a generator and risk returning data which you later find isn't valid:
from typing import Generator, List
def generator_validate(data: List[int]) -> Generator[int, None, None]:
for item in data:
if item % 3 == 0:
raise DataInvalidError()
else:
yield item
def list_validate(data: List[int]) -> List[int]:
ret = []
for item in data:
if item % 3 == 0:
raise DataInvalidError()
else:
ret.append(item)
return ret
Unless I am understanding incorrectly?
You're right @TMiguelT, Thankfully my only job after validation is to insert into a DB, so I am using one atomic transaction for all inserts. That should do good for now.
If you are using transactions, then could you not just rollback as soon as you hit a validation warning?
Yup that's the plan
When reading large files with pandas, we often set iterator true.
I have a use case where I have to discard the processing of the whole file even if one value is corrput. I am leaning towards implementing a validator as a generator like the following.
The problem is the above case is if there is an error in the 3rd chunk it won't stop the first 2 from processing.
How to handle this?