61a746e0-56f6-4f74-bbde-a98c5612db23
commented
15 years ago The array.array type is an excellent type for storing a large amount of "native" elements, such as integers, chars, doubles, etc., without involving the heavy machinery of numpy. It's both blazingly fast and reasonably efficient with memory. The one thing missing from the array module is the ability to directly access array values from C.
This might seem superfluous, as it's perfectly possible to manipulate array contents from Python/C using PyObject_CallMethod and friends. The problem is that it requires the native values to be marshalled to Python objects, only to be immediately converted back to native values by the array code. This can be a problem when, for example, a numeric array needs to be filled with contents, such as in this hypothetical example:
/* error checking and refcounting subtleties omitted for brevity */
PyObject *load_data(Source *src)
{
PyObject *array_type = get_array_type();
PyObject *array = PyObject_CallFunction(array_type, "c", 'd');
PyObject *append = PyObect_GetAttrString(array, "append");
while (!source_done(src)) {
double num = source_next(src);
PyObject *f = PyFloat_FromDouble(num);
PyObject *ret = PyObject_CallFunctionObjArgs(append, f, NULL);
if (!ret)
return NULL;
Py_DECREF(ret);
Py_DECREF(f);
}
Py_DECREF(array_type);
return array;
}The inner loop must convert each C double to a Python Float, only for the array to immediately extract the double back from the Float and store it into the underlying array of C doubles. This may seem like a nitpick, but it turns out that more than half of the time of this function is spent creating and deleting those short-lived floating-point objects.
Float creation is already well-optimized, so opportunities for speedup lie elsewhere. The array object exposes a writable buffer, which can be used to store values directly. For test purposes I created a faster "append" specialized for doubles, defined like this:
int array_append(PyObject *array, PyObject *appendfun, double val)
{
PyObject *ret;
double *buf;
Py_ssize_t bufsize;
static PyObject *zero;
if (!zero)
zero = PyFloat_FromDouble(0);
// append dummy zero value, created only once
ret = PyObject_CallFunctionObjArgs(appendfun, zero, NULL);
if (!ret)
return -1;
Py_DECREF(ret);
// append the element directly at the end of the C buffer
PyObject_AsWriteBuffer(array, (void **) &buf, &bufsize));
buf[bufsize / sizeof(double) - 1] = val;
return 0;
}This hack actually speeds up array creation by a significant percentage (30-40% in my case, and that's for code that was producing the values by parsing a large text file).
It turns out that an even faster method of creating an array is by using the fromstring() method. fromstring() requires an actual string, not a buffer, so in C++ I created an std::vector\<double> with a contiguous array of doubles, passed that array to PyString_FromStringAndSize, and called array.fromstring with the resulting string. Despite all the unnecessary copying, the result was much faster than either of the previous versions.
Would it be possible for the array module to define a C interface for the most frequent operations on array objects, such as appending an item, and getting/setting an item? Failing that, could we at least make fromstring() accept an arbitrary read buffer, not just an actual string?
1fd7a44c-f7f2-43ed-9c9f-bafa512b8598
1409fbaa-f956-4d99-a567-589cf071a381
pitrou
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields: ```python assignee = None closed_at = None created_at =
labels = ['expert-C-API', 'type-feature', 'library']
title = 'C API for appending to arrays'
updated_at =
user = 'https://github.com/hniksic'
```
bugs.python.org fields:
```python
activity =
actor = 'Jeffrey.Kintscher'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'C API']
creation =
creator = 'hniksic'
dependencies = []
files = []
hgrepos = []
issue_num = 5141
keywords = []
message_count = 6.0
messages = ['81039', '81168', '81189', '87828', '87838', '87883']
nosy_count = 6.0
nosy_names = ['ggenellina', 'kxroberto', 'pitrou', 'hniksic', 'bfroehle', 'Jeffrey.Kintscher']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue5141'
versions = []
```