search

overcat / fastcrc

A hyper-fast Python module for computing CRC(8, 16, 32, 64) checksum.
https://fastcrc.readthedocs.io
MIT License
19 stars 6 forks source link

Performance issue ... #4

Open kif opened 1 year ago

kif commented 1 year ago

Hi, I am looking for a fast and portable CRC implementation for Python ... but I think I found a performance bug:

import numpy,zlib,fastcrc
a = numpy.random.random((2048,2048))
%timeit zlib.crc32(a)
26.4 ms ± 29 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit fastcrc.crc32.aixm(a.tobytes())
83.9 ms ± 8.78 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit pyFAI.ext.fastcrc.crc32(a)
7.59 ms ± 2.07 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The implementation from pyFAI relies on the SSE4 instruction, thus not portable.

akhilles commented 1 year ago

FYI, crc-rs has a new (unreleased) implementation that's ~6x faster in micro-benchmarks: https://github.com/mrhooray/crc-rs#throughput-gibs. However, a portable implementation just won't be able to match the performance of a SIMD implementation.