Avoid unwanted GCC optimizations in ARCv2/3 32/64bit atomics based on llock/scond
All the llock/scond based atomic operations read and write atomic counter field. However write operation performed by the scond instruction is not properly communicated to the compiler: inline assembly shows atomic argument as an input parameter and clobber argument is not 'memory'. As a result, compiler can optimize the usage of atomic argument. This issue can be observed with the following simple test functions:
static void test_atomic_simple(void)
{
int v0 = 0xfaceabab;
atomic_t v;
int r;
atomic_set(&v, v0);
r = v0;
atomic_inc(&v);
r += 1;
BUG_ON(v.counter != r);
}
static void test_atomic_simple64(void)
{
long long v0 = 0xfaceabadf00df001LL;
atomic64_t v;
long long r;
atomic64_set(&v, v0);
r = v0;
atomic64_inc(&v);
r += 1LL;
BUG_ON(v.counter != r);
}
Fix immediate operand usage in ARCv3 64bit atomics
ARCv3 ISA allows up to 32bit immediate operands. So passing 64bit argument as immediate operand effectively clears the upper 32bit word. Fix current 64bit atomic implementation by handle a 64bit argument as a register operand.
Implement kernel support for new 'fetch-and-operate' ARCv3 atomics
ARCv3 ISA provides ATLD family of instructions to support 'fetch-and-operate' atomic functions. Implement 'fetch-and-operate' kernel atomic functions using new instructions. Put new atomics under Kconfig option ARC_HAS_ATLD and disable it by default.
UPD:
Optimization issue is observed on HSDK with upstream 5.15 kernel. Appropriately adapted fix from this MR resolves the problem as well.
This PR provides the following major changes:
Avoid unwanted GCC optimizations in ARCv2/3 32/64bit atomics based on llock/scond All the llock/scond based atomic operations read and write atomic counter field. However write operation performed by the scond instruction is not properly communicated to the compiler: inline assembly shows atomic argument as an input parameter and clobber argument is not 'memory'. As a result, compiler can optimize the usage of atomic argument. This issue can be observed with the following simple test functions:
Fix immediate operand usage in ARCv3 64bit atomics ARCv3 ISA allows up to 32bit immediate operands. So passing 64bit argument as immediate operand effectively clears the upper 32bit word. Fix current 64bit atomic implementation by handle a 64bit argument as a register operand.
Implement kernel support for new 'fetch-and-operate' ARCv3 atomics ARCv3 ISA provides ATLD family of instructions to support 'fetch-and-operate' atomic functions. Implement 'fetch-and-operate' kernel atomic functions using new instructions. Put new atomics under Kconfig option ARC_HAS_ATLD and disable it by default.
UPD: Optimization issue is observed on HSDK with upstream 5.15 kernel. Appropriately adapted fix from this MR resolves the problem as well.