search

Iotic-Labs / py-ubjson

Universal Binary JSON draft-12 serializer for Python
Apache License 2.0
42 stars 12 forks source link

src/encoder.c:271: _encode_PyDecimal: Assertion `PyBytes_Check(str)' failed. #12

Closed mgorny closed 4 years ago

mgorny commented 4 years ago

The extension crashes while running the tests:

test_array (test.test.TestEncodeDecodeFpExt) ... python3.8: src/encoder.c:271: _encode_PyDecimal: Assertion `PyBytes_Check(str)' failed.
Aborted (core dumped)

To reproduce:

$ python setup.py build build_ext --inplace
$ python setup.py test
vtermanis commented 4 years ago

Thank you for the bug report. I'll try to have a look this evening.

Could you please add to the description:

mgorny commented 4 years ago 
$ python3.6 --version
Python 3.6.10
$ python3.7 --version
Python 3.7.6
$ python3.8 --version
Python 3.8.2

Gentoo Linux/amd64.

vtermanis commented 4 years ago

I could not reproduce the crash locally (Ubuntu 18.04 x64) against any of 3.6.9, 3.7.6 or 3.8.2.

However, the fact it fails on the line which attempts to PyBytes_GET_SIZE - I think the length retrieval should be against raw and not str - that looks like it might be the bug. I'll try that tomorrow.

In the mean time - @mgorny might you be able to paste in the compiler arguments which were used to build the extension your end, one for any of the objects and final linking, e.g.:

x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/vt/repos/iotic/py-ubjson/py3.8.2/include -I/usr/include/python3.8 -c src/python_funcs.c -o build/temp.linux-x86_64-3.8/src/python_funcs.o -std=c99
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/src/_ubjson.o build/temp.linux-x86_64-3.8/src/decoder.o build/temp.linux-x86_64-3.8/src/encoder.o build/temp.linux-x86_64-3.8/src/python_funcs.o -o /home/vt/repos/iotic/py-ubjson/_ubjson.cpython-38-x86_64-linux-gnu.so
mgorny commented 4 years ago

Sure:

x86_64-pc-linux-gnu-gcc-9.2.0 -march=x86-64 -mtune=k8 -mcx16 -msahf -msse3 --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=512 -O2 -pipe -frecord-gcc-switches -fPIC -I/usr/include/python3.6m -c src/_ubjson.c -o /tmp/portage/dev-python/py-ubjson-0.14.0/work/py-ubjson-0.14.0-python3_6/temp.linux-x86_64-3.6/src/_ubjson.o -std=c99
x86_64-pc-linux-gnu-gcc-9.2.0 -march=x86-64 -mtune=k8 -mcx16 -msahf -msse3 --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=512 -O2 -pipe -frecord-gcc-switches -fPIC -I/usr/include/python3.6m -c src/decoder.c -o /tmp/portage/dev-python/py-ubjson-0.14.0/work/py-ubjson-0.14.0-python3_6/temp.linux-x86_64-3.6/src/decoder.o -std=c99
x86_64-pc-linux-gnu-gcc-9.2.0 -march=x86-64 -mtune=k8 -mcx16 -msahf -msse3 --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=512 -O2 -pipe -frecord-gcc-switches -fPIC -I/usr/include/python3.6m -c src/encoder.c -o /tmp/portage/dev-python/py-ubjson-0.14.0/work/py-ubjson-0.14.0-python3_6/temp.linux-x86_64-3.6/src/encoder.o -std=c99
x86_64-pc-linux-gnu-gcc-9.2.0 -march=x86-64 -mtune=k8 -mcx16 -msahf -msse3 --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=512 -O2 -pipe -frecord-gcc-switches -fPIC -I/usr/include/python3.6m -c src/python_funcs.c -o /tmp/portage/dev-python/py-ubjson-0.14.0/work/py-ubjson-0.14.0-python3_6/temp.linux-x86_64-3.6/src/python_funcs.o -std=c99
x86_64-pc-linux-gnu-gcc-9.2.0 -shared -Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0 -march=x86-64 -mtune=k8 -mcx16 -msahf -msse3 --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=512 -O2 -pipe -frecord-gcc-switches /tmp/portage/dev-python/py-ubjson-0.14.0/work/py-ubjson-0.14.0-python3_6/temp.linux-x86_64-3.6/src/_ubjson.o /tmp/portage/dev-python/py-ubjson-0.14.0/work/py-ubjson-0.14.0-python3_6/temp.linux-x86_64-3.6/src/decoder.o /tmp/portage/dev-python/py-ubjson-0.14.0/work/py-ubjson-0.14.0-python3_6/temp.linux-x86_64-3.6/src/encoder.o /tmp/portage/dev-python/py-ubjson-0.14.0/work/py-ubjson-0.14.0-python3_6/temp.linux-x86_64-3.6/src/python_funcs.o -L/usr/lib64 -lpython3.6m -o /tmp/portage/dev-python/py-ubjson-0.14.0/work/py-ubjson-0.14.0-python3_6/lib/_ubjson.cpython-36m-x86_64-linux-gnu.so

If I were to guess, -DNDEBUG is stripping away runtime checks (on your system).

vtermanis commented 4 years ago

Reproduced by adding undef_macros=['NDEBUG'] to extension definition and then using python setup.py build_ext -ig

vtermanis commented 4 years ago

@mgorny , might you be able to test the proposed fix in #13 ? (It fixes it for me with asserts enabled and from code inspection I think this is the only place where PyBytes_GET_SIZE was mistakenly called against the wrong object.)

mgorny commented 4 years ago

I can confirm that this patch fixes the crash for me, and all tests pass after applying it.

vtermanis commented 4 years ago

Fixed released in v0.15.0: https://pypi.org/project/py-ubjson/0.15.0/