I suggest we implement a some kind of data checker that is performed during the data transfer step (is it the right place?).
Every chunk and block of data needs to be tested for presence of zero values, nan's and inf's. Then the result needs to be accumulated in the block's loop to come up with the statistics for a chunk. The result for every block can go into the debug file.
The user, however, can get a warning about the percentage of nan's/inf's/zeros with respect to all values in the chunk. This warning should go to the user's log. In case if there are no nan's/inf's the message does not appear in the user's log. And zeros alert should only if zeros hit a certain percentage (e.g. 25% of the data are zeros).
Let me know if it sounds sensible? It shouldn't really take a lot of time to calculate this, but could be of great use for users and developers.
Discussion started in #267 and continued in #369.
I suggest we implement a some kind of data checker that is performed during the data transfer step (is it the right place?).
Every chunk and block of data needs to be tested for presence of zero values, nan's and inf's. Then the result needs to be accumulated in the block's loop to come up with the statistics for a chunk. The result for every block can go into the debug file.
The user, however, can get a warning about the percentage of nan's/inf's/zeros with respect to all values in the chunk. This warning should go to the user's log. In case if there are no nan's/inf's the message does not appear in the user's log. And zeros alert should only if zeros hit a certain percentage (e.g. 25% of the data are zeros).
Let me know if it sounds sensible? It shouldn't really take a lot of time to calculate this, but could be of great use for users and developers.