Passing an array of size zero doesn't work

[marktsuchida]

Calling a function with a pointer parameter with a Python object implementing the buffer protocol does not appear to work when the object has zero elements. Is this intended behavior?

>>> import cppyy, array
>>> cppyy.cppdef("void f(unsigned char const *buf) {}")
True
>>> cppyy.gbl.f(array.array('B', [1, 2, 3]))  # OK
>>> cppyy.gbl.f(array.array('B', []))         # Fails
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: void ::f(const unsigned char* buf) =>
    TypeError: could not convert argument 1 (could not convert argument to buffer or nullptr)

(My hope was that f() would get called, either with a non-null pointer that f should not dereference, or with nullptr.)

Furthermore, a memoryview of size 0 causes a crash:

>>> cppyy.gbl.f(memoryview(array.array('B', [1, 2, 3])))  # OK
>>> cppyy.gbl.f(memoryview(array.array('B', [])))         # Crashes, exit code 129
 *** Break *** segmentation violation

Full stack trace

(It looks like the same thing gets printed twice.) ```text >>> cppyy.gbl.f(memoryview(array.array('B', []))) *** Break *** segmentation violation [/Users/mark/tmp/venv/lib/python3.12/site-packages/cppyy_backend/lib/libcppyy_backend.so] (anonymous namespace)::TExceptionHandlerImp::HandleException(int) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/cppyy_backend/lib/libCoreLegacy.so] CppyyLegacy::TUnixSystem::DispatchSignals(CppyyLegacy::ESignals) (no debug info) [/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyUnicode_FromFormatV (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyUnicode_FromFormat (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::Utility::GetBuffer(_object*, char, int, void*&, bool) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::(anonymous namespace)::UCharArrayConverter::SetArg(_object*, CPyCppyy::Parameter&, CPyCppyy::CallContext*) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::CPPMethod::ConvertAndSetArgs(_object* const*, unsigned long, CPyCppyy::CallContext*) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::CPPFunction::Call(CPyCppyy::CPPInstance*&, _object* const*, unsigned long, _object*, CPyCppyy::CallContext*) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::(anonymous namespace)::mp_vectorcall(CPyCppyy::CPPOverload*, _object* const*, unsigned long, _object*) (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyEval_EvalCode (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] builtin_exec (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] gen_send_ex2 (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] gen_send_ex (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] method_vectorcall (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyVectorcall_Call (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyEval_EvalCode (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] run_eval_code_obj (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] run_mod (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pyrun_file (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyRun_SimpleFileObject (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyRun_AnyFileObject (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pymain_run_file_obj (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pymain_run_file (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] Py_RunMain (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] Py_BytesMain (no debug info) [/usr/lib/dyld] start (no debug info) *** Break *** segmentation violation [/Users/mark/tmp/venv/lib/python3.12/site-packages/cppyy_backend/lib/libcppyy_backend.so] (anonymous namespace)::TExceptionHandlerImp::HandleException(int) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/cppyy_backend/lib/libCoreLegacy.so] CppyyLegacy::TUnixSystem::DispatchSignals(CppyyLegacy::ESignals) (no debug info) [/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyUnicode_FromFormatV (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyUnicode_FromFormat (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::Utility::GetBuffer(_object*, char, int, void*&, bool) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::(anonymous namespace)::UCharArrayConverter::SetArg(_object*, CPyCppyy::Parameter&, CPyCppyy::CallContext*) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::CPPMethod::ConvertAndSetArgs(_object* const*, unsigned long, CPyCppyy::CallContext*) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::CPPFunction::Call(CPyCppyy::CPPInstance*&, _object* const*, unsigned long, _object*, CPyCppyy::CallContext*) (no debug info) [/Users/mark/tmp/venv/lib/python3.12/site-packages/libcppyy.cpython-312-darwin.so] CPyCppyy::(anonymous namespace)::mp_vectorcall(CPyCppyy::CPPOverload*, _object* const*, unsigned long, _object*) (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyEval_EvalCode (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] builtin_exec (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] gen_send_ex2 (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] gen_send_ex (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] method_vectorcall (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyVectorcall_Call (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyEval_EvalFrameDefault (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] PyEval_EvalCode (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] run_eval_code_obj (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] run_mod (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pyrun_file (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyRun_SimpleFileObject (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] _PyRun_AnyFileObject (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pymain_run_file_obj (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] pymain_run_file (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] Py_RunMain (no debug info) [/opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/Python] Py_BytesMain (no debug info) [/usr/lib/dyld] start (no debug info) ```

This was with Python 3.12.3 from Homebrew, macOS 14, arm64, cppyy 3.1.2, cppyy-backend 1.15.2, cppyy-cling 6.30.0, CPyCppyy 1.12.16.

I got the same behavior with Python 3.12.2, x86-64, from conda-forge (running on Rosetta 2), same cppyy versions, except that the error printed before crashing was *** Break *** floating point exception. The stack trace was essentially identical to the one above except that some frames were omitted. The exit code was 140 (SIGSYS?) instead of 129 (SIGHUP?), FWIW.

Passing bytes() raised TypeError similarly to the zero-length array.array, and passing np.frombuffer(bytes(), dtype=np.uint8) crashed similarly to the zero-length memoryview.

[wlav]

The segfault should obviously not happen (will fix that); the others are neither here nor there: an empty Python container is not the same as a null pointer. The code fails b/c it looks for a compatible buffer by asking the object for one through the Python buffer interface and doesn't get any. It doesn't know/check the type to figure if it's a compatible one that could have served up a buffer if only the object had a buffer: there's no end to those cases (bytearray, array, numpy.ndarray, bytes, memoryview, LowLevelView, ctypes.c_ubyte*0, ...).

To pass a null pointer here, use cppyy.nullptr:

>>> import cppyy, array
>>> cppyy.cppdef("void f(unsigned char const *buf) {}")
True
>>> cppyy.gbl.f(cppyy.nullptr)
>>> 

[wlav]

Interesting, the crash with the memoryview is in Python, not in cppyy:

    if (PyObject_CheckBuffer(pyobject)) {
        Py_buffer bufinfo;
        memset(&bufinfo, 0, sizeof(Py_buffer));
        if (PyObject_GetBuffer(pyobject, &bufinfo, PyBUF_FORMAT) == 0) {    // <- this crashes

and oddly, it only happens on Linux.

I've put in a protection against asking sequences of length 0 for their buffer info. However, this is once more a case where it's clear that an empty container-like object just isn't a nullptr.

[marktsuchida]

Thanks. I agree that an empty buffer is not the same thing as nullptr, but was approaching this from the example in the docs (void array_method(int* ad, int size)) in which a zero size would very much make sense (actually I'm trying to come up with a way to translate objects implementing the buffer protocol into std::span). The workaround of special-casing the zero-size case on the Python side will definitely work for me.

---

The code fails b/c it looks for a compatible buffer by asking the object for one through the Python buffer interface and doesn't get any.

I'm a little confused by this, because I'm pretty sure you do get a buffer interface from an empty buffer object:

In [3]: cppyy.include("Python.h")                                               
Out[3]: True                                                                                                            

In [20]: cppyy.cppdef("""
    ...: PyObject *i(PyObject *pyobj) {
    ...:     if (PyObject_CheckBuffer(pyobj) != 1) {
    ...:         PyErr_SetString(PyExc_TypeError, "Not a buffer");
    ...:         return NULL;
    ...:     }
    ...:     Py_buffer bufinfo;
    ...:     memset(&bufinfo, 0, sizeof(bufinfo));
    ...:     if (PyObject_GetBuffer(pyobj, &bufinfo, PyBUF_FORMAT) != 0)
    ...:         return NULL;
    ...:     PyObject *ret = PyUnicode_FromString(bufinfo.format);
    ...:     PyBuffer_Release(&bufinfo);
    ...:     return ret;
    ...: }
    ...: """)
Out[20]: True

In [21]: cppyy.gbl.i(42)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[21], line 1
----> 1 cppyy.gbl.i(42)

TypeError: _object* ::i(PyObject* pyobj) =>
    TypeError: Not a buffer

In [22]: cppyy.gbl.i(b'abc')
Out[22]: 'B'

In [23]: cppyy.gbl.i(bytes())
Out[23]: 'B'

The reason empty buffers are not treated as buffers appears to be because CPyCppyy::Utility::GetBuffer() returns 0 both for non-buffers and otherwise compatible buffers that happen to be empty. (But if you feel that maintaining this behavior (TypeError on empty buffer) is important for backward compatibility, I certainly understand.)

---

Interestingly, wrapping in memoryview throws an exception in this case (from the PyObject_GetBuffer() call):

In [24]: cppyy.gbl.i(memoryview(bytes()))
---------------------------------------------------------------------------
BufferError                               Traceback (most recent call last)
Cell In[24], line 1
----> 1 cppyy.gbl.i(memoryview(bytes()))

BufferError: _object* ::i(PyObject* pyobj) =>
    BufferError: memoryview: cannot cast to unsigned bytes if the format flag is present

And the exception is not limited to the empty case. So I guess memoryview just doesn't work when PyBUF_FORMAT is requested (alone). cppyy.gbl.j(memoryview(bytes())) does return 'B', where j() is identical to i() above except for PyBUF_FORMAT being replaced with PyBUF_ND | PyBUF_FORMAT.

The docs say "PyBUF_FORMAT can be |’d to any of the flags except PyBUF_SIMPLE. The latter already implies format B (unsigned bytes)." And PyBUF_SIMPLE equals 0, so PyBUF_FORMAT on its own may be problematic.

(I don't really understand why non-empty memoryview worked in my initial post. Perhaps it is handled by the fallback code after PyObject_GetBuffer() fails?)