encodeULEB128() buffer overflow cause tblgen "stack smashing detected" crash

[Quuxplusone]

Bugzilla Link	PR37447
Status	NEW
Importance	P enhancement
Reported by	Kazushi (Jam) Marukawa (marukawa@nec.com)
Reported on	2018-05-14 00:28:22 -0700
Last modified on	2018-07-31 22:15:55 -0700
Version	trunk
Hardware	PC Windows NT
CC	llvm-bugs@lists.llvm.org
Fixed by commit(s)
Attachments	0001-Fixed-a-bug-that-may-cause-stack-smashing-detected-a.patch (2992 bytes, application/mbox)
Blocks
Blocked by
See also

Created attachment 20301
patch to fix described problem

When I tried to implement a backend for a CPU using 64 bit wide instructions,
tblgen crashes very often with "stack smashing detected" error.

I drilled down it and found tblgen calls encodeULEB128() with 64 bit value
for input and 8 bytes buffer for output sometimes.  This encodeULEB128()
encodes a given value into buffer using SDNV (Self-Delimiting Numeric Value)
format.  This format requires 10 bytes buffer at maximum (abs(64/7) = 10).
For example, 0x8000000000000000 input is converted into "80 80 80 80 80 80
80 80 80 01".

Once I applied attached patch, tblgen works fine with my modified backend.
Please review this patch and fix the bug.  Thanks.

In this patch, modification of TableGen/FixedLenDecoderEmitter.cpp is what
above crash causes.  Modification of ProfileData/InstrProf.cpp is what I
found by grep and this may not be needed.

[Quuxplusone]

Attached 0001-Fixed-a-bug-that-may-cause-stack-smashing-detected-a.patch (2992 bytes, application/mbox): patch to fix described problem

[Quuxplusone]

Modification to utils/TableGen/FixedLenDecoderEmitter.cpp is convert by c13d59697a3d2cbe26342c7c6982472768a6e92f commit. Thanks.