This pull request updates PoolingGPUTest to allow for floating point round-off errors. With this update, all tests pass when using CUDA 11.1.1 and cuDNN 8.0.4.
cuDNN does not have sum pooling, per se, instead it has average pooling with a multiplication parameter. However, (sum/denominator)*denominator is not always exactly sum when using floating point arithmetic. This issue did not arise in PoolingGPUTest when using cuDNN version 7, but it does arise with cuDNN version 8.
Because the discrepancy is observed to arise only in the least-significant bits of the floating point representation, it makes sense to use relative error instead of absolute error in judging whether the test has passed. Accordingly, I added a PtwiseQuotientLayer after the Comparison layer, to compute (outputGPU - outputCPU) / outputCPU. Dividing by outputCPU scales all floating point errors to be on the order of magnitude of 2^-n, where n is the number of bits in the floating point representation's mantissa. For single-precision floats this is approximately 1.2e-7, which motivated the choice of nnzThreshold for the RequireAllZeroActivityProbe.
This pull request updates PoolingGPUTest to allow for floating point round-off errors. With this update, all tests pass when using CUDA 11.1.1 and cuDNN 8.0.4.
cuDNN does not have sum pooling, per se, instead it has average pooling with a multiplication parameter. However, (sum/denominator)*denominator is not always exactly sum when using floating point arithmetic. This issue did not arise in PoolingGPUTest when using cuDNN version 7, but it does arise with cuDNN version 8.
Because the discrepancy is observed to arise only in the least-significant bits of the floating point representation, it makes sense to use relative error instead of absolute error in judging whether the test has passed. Accordingly, I added a PtwiseQuotientLayer after the Comparison layer, to compute (outputGPU - outputCPU) / outputCPU. Dividing by outputCPU scales all floating point errors to be on the order of magnitude of 2^-n, where n is the number of bits in the floating point representation's mantissa. For single-precision floats this is approximately 1.2e-7, which motivated the choice of nnzThreshold for the RequireAllZeroActivityProbe.