search

IST-DASLab / marlin

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
Apache License 2.0
575 stars 45 forks source link

slight nondeterminism #27

Open MichoChan opened 3 months ago

MichoChan commented 3 months ago

Marlin internally uses locks to synchronize the threads. This canresult in very slight nondeterminism for Marlin.

why？

MichoChan commented 3 months ago

can provide a determinism version？