llvmbot
commented
2 months ago @llvm/issue-subscribers-backend-aarch64
Author: Niles Salter (Validark)
I can get LLVM to do a `bitcast i16 to <16 x i1>` + `sext <16 x i1> to <16 x i8>` like so:
```zig
export fn unmovemask16(x: u16) @Vector(16, u8) {
return @select(u8, @as(@Vector(16, u1), @bitCast(x)) == @as(@Vector(16, u1), @splat(1)),
@as(@Vector(16, u8), @splat(0xff)),
@as(@Vector(16, u8), @splat(0)));
}
export fn unmovemask64(x: u64) @Vector(64, u8) {
return @select(u8, @as(@Vector(64, u1), @bitCast(x)) == @as(@Vector(64, u1), @splat(1)),
@as(@Vector(64, u8), @splat(0xff)),
@as(@Vector(64, u8), @splat(0)));
}
```
Unfortunately, LLVM doesn't currently do anything smart for this. Here is the 16-bit version (compiled for Apple M2):
```asm
unmovemask_normal: // @unmovemask_normal
sub sp, sp, #16
sbfx w8, w0, #1, #1
sbfx w9, w0, #0, #1
fmov s0, w9
mov v0.b[1], w8
sbfx w8, w0, #2, #1
mov v0.b[2], w8
sbfx w8, w0, #3, #1
mov v0.b[3], w8
sbfx w8, w0, #4, #1
mov v0.b[4], w8
sbfx w8, w0, #5, #1
mov v0.b[5], w8
sbfx w8, w0, #6, #1
mov v0.b[6], w8
sbfx w8, w0, #7, #1
mov v0.b[7], w8
sbfx w8, w0, #8, #1
mov v0.b[8], w8
sbfx w8, w0, #9, #1
mov v0.b[9], w8
sbfx w8, w0, #10, #1
mov v0.b[10], w8
sbfx w8, w0, #11, #1
mov v0.b[11], w8
sbfx w8, w0, #12, #1
mov v0.b[12], w8
sbfx w8, w0, #13, #1
mov v0.b[13], w8
sbfx w8, w0, #14, #1
mov v0.b[14], w8
sbfx w8, w0, #15, #1
mov v0.b[15], w8
add sp, sp, #16
ret
```
Here is the emit for the 16-bit and 64-bit version compiled for x86-64 Westmere:
```asm
.LCPI0_0:
.byte 0
.byte 0
.byte 0
.byte 0
.byte 0
.byte 0
.byte 0
.byte 0
.byte 1
.byte 1
.byte 1
.byte 1
.byte 1
.byte 1
.byte 1
.byte 1
.LCPI0_1:
.byte 1
.byte 2
.byte 4
.byte 8
.byte 16
.byte 32
.byte 64
.byte 128
.byte 1
.byte 2
.byte 4
.byte 8
.byte 16
.byte 32
.byte 64
.byte 128
unmovemask16:
movd xmm0, edi
pshufb xmm0, xmmword ptr [rip + .LCPI0_0]
movdqa xmm1, xmmword ptr [rip + .LCPI0_1]
pand xmm0, xmm1
pcmpeqb xmm0, xmm1
ret
.LCPI1_0:
.byte 1
.byte 2
.byte 4
.byte 8
.byte 16
.byte 32
.byte 64
.byte 128
.byte 1
.byte 2
.byte 4
.byte 8
.byte 16
.byte 32
.byte 64
.byte 128
unmovemask64:
movq xmm3, rdi
punpcklbw xmm3, xmm3
pshuflw xmm0, xmm3, 80
pshufd xmm0, xmm0, 80
movdqa xmm4, xmmword ptr [rip + .LCPI1_0]
pand xmm0, xmm4
pcmpeqb xmm0, xmm4
pshuflw xmm1, xmm3, 250
pshufd xmm1, xmm1, 80
pand xmm1, xmm4
pcmpeqb xmm1, xmm4
pshufhw xmm2, xmm3, 80
pshufd xmm2, xmm2, 250
pand xmm2, xmm4
pcmpeqb xmm2, xmm4
pshufhw xmm3, xmm3, 250
pshufd xmm3, xmm3, 250
pand xmm3, xmm4
pcmpeqb xmm3, xmm4
ret
```
Here we have two strategies for accomplishing this task for 16 byte vectors. In the first, we use a `pshufb`, which in ARM-land could be replaced by `tbl`, to broadcast the first byte to the first 8 bytes, and the second byte to the second 8 bytes. Then we load up a mask, and `pand`+`pcmpeqb` to turn each bit into a byte. On ARM we have equivalents both of those instructions, but we also have `cmtst` which can do both of those steps in one.
The second strategy does `punpcklbw` on the 8 byte bitstring, which interleaves it with itself, which is equivalent to `zip1` on ARM. Then it uses `pshuflw xmm, xmm, 80`+`pshufd xmm0, xmm0, 80`. I don't think we have support for `tbl`-with-constant on ARM, so I think we have to just use the first strategy.
In open-code, to produce the second, third, and fourth vector, we might want to reuse the same vector (`0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1`) to pass into the `tbl` instruction, so we could take the scalar input and shl by 16, 32, and 48, and move them into vector registers separately. Alternatively, one could use `ext` once they are already in vector registers.
In a loop, however, it might make more sense to have a vector that holds each of the following:
```
0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3
4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5
6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7
```
That way, we can `tbl` with each of them in parallel, and run 4 `cmtst`s in parallel afterwards.
RKSimon
Validark
I can get LLVM to do a
bitcast i16 to <16 x i1>+sext <16 x i1> to <16 x i8>like so:Unfortunately, LLVM doesn't currently do anything smart for this. Here is the 16-bit version (compiled for Apple M2):
Here is the emit for the 16-bit and 64-bit version compiled for x86-64 Westmere:
Here we have two strategies for accomplishing this task for 16 byte vectors. In the first, we use a
pshufb, which in ARM-land could be replaced bytbl, to broadcast the first byte to the first 8 bytes, and the second byte to the second 8 bytes. Then we load up a mask, andpand+pcmpeqbto turn each bit into a byte. On ARM we have equivalents both of those instructions, but we also havecmtstwhich can do both of those steps in one.The second strategy does
punpcklbwon the 8 byte bitstring, which interleaves it with itself, which is equivalent tozip1on ARM. Then it usespshuflw xmm, xmm, 80+pshufd xmm0, xmm0, 80. I don't think we have support fortbl-with-constant on ARM, so I think we have to just use the first strategy.In size-optimized open-code, to produce the second, third, and fourth vector, we might want to reuse the same vector (
0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1) to pass into thetblinstruction, so we could take the scalar input and shl by 16, 32, and 48, and move them into vector registers separately. Alternatively, one could useextonce they are already in vector registers.However, it might make more sense for performance to have a vector that holds each of the following, especially in a loop:
That way, we can
tblwith each of them in parallel, and run 4cmtsts in parallel afterwards.