gladtosee
commented
5 years ago Open gladtosee opened 5 years ago
gladtosee
commented
5 years ago 
WojciechMula
commented
5 years ago @gladtosee To be honest I wasn't aware of this problem, you are the first one mentioning it. I need to learn a little bit about this issue. Thanks for these articles.
gladtosee
commented
5 years ago @WojciechMula After loading the data from the master process, the child process increments the ref count and causes a Copy-On-Write. Because Py_BuildValue() increase the reference count before returning from the automaton_build_output function.
#define Py_INCREF(op) ( \
_Py_INC_REFTOTAL _Py_REF_DEBUG_COMMA \
((PyObject *)(op))->ob_refcnt++)If i modify the code like this: copy on write does not happen.
//copy from cpython source - https://github.com/python/cpython/blob/v3.7.3/Objects/unicodeobject.c#L2380
PyObject*
_PyUnicode_Copy(PyObject *unicode)
{
Py_ssize_t length;
PyObject *copy;
if (!PyUnicode_Check(unicode)) {
PyErr_BadInternalCall();
return NULL;
}
if (PyUnicode_READY(unicode) == -1)
return NULL;
length = PyUnicode_GET_LENGTH(unicode);
copy = PyUnicode_New(length, PyUnicode_MAX_CHAR_VALUE(unicode));
if (!copy)
return NULL;
assert(PyUnicode_KIND(copy) == PyUnicode_KIND(unicode));
memcpy(PyUnicode_DATA(copy), PyUnicode_DATA(unicode),
length * PyUnicode_KIND(unicode));
// assert(_PyUnicode_CheckConsistency(copy, 1));
return copy;
}
static int automaton_build_output(PyObject* self, PyObject** result);
case STORE_ANY:
if(PyUnicode_Check(node->output.object)) {
//N: Same as O, except it doesn’t increment the reference count on the object.
*result = F(Py_BuildValue)("iN", idx, _PyUnicode_Copy(node->output.object));
}
else {
*result = F(Py_BuildValue)("iO", idx, node->output.object);
}
return OutputValue;@gladtosee Could you please provide a patch for this?
yuanchaofa
commented
4 years ago @WojciechMula After loading the data from the master process, the child process increments the ref count and causes a Copy-On-Write. Because Py_BuildValue() increase the reference count before returning from the automaton_build_output function.
#define Py_INCREF(op) ( \ _Py_INC_REFTOTAL _Py_REF_DEBUG_COMMA \ ((PyObject *)(op))->ob_refcnt++)If i modify the code like this: copy on write does not happen.
//copy from cpython source - https://github.com/python/cpython/blob/v3.7.3/Objects/unicodeobject.c#L2380 PyObject* _PyUnicode_Copy(PyObject *unicode) { Py_ssize_t length; PyObject *copy; if (!PyUnicode_Check(unicode)) { PyErr_BadInternalCall(); return NULL; } if (PyUnicode_READY(unicode) == -1) return NULL; length = PyUnicode_GET_LENGTH(unicode); copy = PyUnicode_New(length, PyUnicode_MAX_CHAR_VALUE(unicode)); if (!copy) return NULL; assert(PyUnicode_KIND(copy) == PyUnicode_KIND(unicode)); memcpy(PyUnicode_DATA(copy), PyUnicode_DATA(unicode), length * PyUnicode_KIND(unicode)); // assert(_PyUnicode_CheckConsistency(copy, 1)); return copy; } static int automaton_build_output(PyObject* self, PyObject** result); case STORE_ANY: if(PyUnicode_Check(node->output.object)) { //N: Same as O, except it doesn’t increment the reference count on the object. *result = F(Py_BuildValue)("iN", idx, _PyUnicode_Copy(node->output.object)); } else { *result = F(Py_BuildValue)("iO", idx, node->output.object); } return OutputValue;
I tried to use your code and reinstall, but there are some errors. Symbol not found: _PyUnicode_DATA Would you give me more details about how you solve your problem
pombredanne
commented
2 years ago @yuanchaofa do you mind to provide a PR or patch? it would be much easier to review. Thanks!
@WojciechMula hi? I am using the pyahocorasick well. But I have a problem.
A minor-page fault occurs, which increases the memory usage of the child process. (https://en.wikipedia.org/wiki/Copy-on-write) I forked after using gc.freeze(), but a page fault occurred. (https://docs.python.org/3/library/gc.html#gc.freeze)
What should I do??
I used perf to get the following results. perf record -e minor-faults -g -p PID