search

argonne-lcf / dlio_benchmark

An I/O benchmark for deep Learning applications
https://dlio-benchmark.readthedocs.io
Apache License 2.0
65 stars 30 forks source link

Change `max` to `abs` for preprocess time #240

Closed rayandrew closed 1 week ago

rayandrew commented 2 weeks ago

Hi @zhenghh04 and @hariharan-devarajan,

This PR addresses the problem where random.normal result can be negative. if we do max(t, 0) then we will reduce the variance of preprocess time when the generated value is negative.

It is better to use abs in this case.

At the same time, I add strict comparison to make sure user does not give negative value while specifying preprocess_time or preprocess_time_stdev in the config.