`pyhasattr` segfault

[IanButterworth]

I'm getting a segfault with PythonPlot.jl after a testsuite has finished, assuming that it's during finalizers https://github.com/stevengj/PythonPlot.jl/pull/24

signal (11): Segmentation fault
in expression starting at none:0
_PyInterpreterState_GET at /usr/local/src/conda/python-3.11.0/Include/internal/pycore_pystate.h:116 [inlined]
get_dict_state at /usr/local/src/conda/python-3.11.0/Objects/dictobject.c:251 [inlined]
new_dict at /usr/local/src/conda/python-3.11.0/Objects/dictobject.c:722 [inlined]
PyDict_New at /usr/local/src/conda/python-3.11.0/Objects/dictobject.c:841
type_ready_set_dict at /usr/local/src/conda/python-3.11.0/Objects/typeobject.c:6143 [inlined]
type_ready at /usr/local/src/conda/python-3.11.0/Objects/typeobject.c:6462 [inlined]
PyType_Ready at /usr/local/src/conda/python-3.11.0/Objects/typeobject.c:6513
_PyObject_GenericGetAttrWithDict at /usr/local/src/conda/python-3.11.0/Objects/object.c:1266
PyObject_GenericGetAttr at /usr/local/src/conda/python-3.11.0/Objects/object.c:1367 [inlined]
module_getattro at /usr/local/src/conda/python-3.11.0/Objects/moduleobject.c:761
PyObject_GetAttr at /usr/local/src/conda/python-3.11.0/Objects/object.c:916
PyObject_GetAttr at /home/ubuntu/.julia/packages/PythonCall/dsECZ/src/cpython/pointers.jl:299 [inlined]
macro expansion at /home/ubuntu/.julia/packages/PythonCall/dsECZ/src/Py.jl:131 [inlined]
pyhasattr at /home/ubuntu/.julia/packages/PythonCall/dsECZ/src/abstract/object.jl:37
#9 at /home/ubuntu/.julia/packages/PythonPlot/ei8cR/src/pygui.jl:194
macro expansion at ./asyncevent.jl:281 [inlined]
#666 at ./task.jl:134
unknown function (ip: 0x7fac1b57080f)
_jl_invoke at /cache/build/default-amdci4-2/julialang/julia-release-1-dot-8/src/gf.c:2377 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-2/julialang/julia-release-1-dot-8/src/gf.c:2559
jl_apply at /cache/build/default-amdci4-2/julialang/julia-release-1-dot-8/src/julia.h:1843 [inlined]
start_task at /cache/build/default-amdci4-2/julialang/julia-release-1-dot-8/src/task.c:931
Allocations: 860081766 (Pool: 859557939; Big: 523827); GC: 110

I noticed that pyhasattr is PyObject_GetAttr backed. Perhaps it needs some more protection against whatever is causing this?

[github-actions[bot]]

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

[github-actions[bot]]

This issue has been closed because it has been stale for 7 days. You can re-open it if it is still relevant.

[IanButterworth]

I don't see why this can be closed

[cjdoris]

Can you provide a MWE for this issue?

[schlichtanders]

@IanButterworth I am running into a very similar segmentationfault, but have no clue at why it is there and how to prevent it. If you have some ideas or MWE, any help is highly appreciated

I am currently trying to use PythonCall for extending Pluto towards Python. While when I load my routines on the shell directly, evrything works nicely, but if they are called somewhere in the Pluto webserver to do parsing, I get a similar segmentation fault. I cannot minify it yet...

[IanButterworth]

I wasn't able to reduce this. And the code it happened on was private. Sorry

[schlichtanders]

I was able to reproduce this - it is an interaction between PythonCall, HTTP and having set Julia Threads

spawn a container like this

FROM julia:1.9-bookworm AS usersetup
ENV JULIA_NUM_THREADS=auto
CMD julia

and run the following MWE

import Pkg
Pkg.add(["HTTP", "PythonCall", "Sockets"])
using HTTP
using PythonCall
import Sockets

function default_404(req = nothing)
    HTTP.Response(404, "Not found!")
end

router = HTTP.Router(default_404)

@warn "pid = ..."
os = pyimport("os")
pid = os.getpid()
@warn "pid = $pid"

function get_pid(request::HTTP.Request)
    @warn "pid = ..."
    os = pyimport("os")
    pid = os.getpid()
    @warn "pid = $pid"
    return HTTP.Response(200, string(pid))
end
HTTP.register!(router, "GET", "/pid", get_pid)

port = UInt16(32132)
serversocket = Sockets.listen(port)
server = HTTP.listen!(port; stream=true, server=serversocket, verbose=-1) do http::HTTP.Stream
    request::HTTP.Request = http.message
    request.body = read(http)
    response_body = router(request)        
    @show response_body
    request.response::HTTP.Response = response_body
    request.response.request = request

    try
        HTTP.setheader(http, "Content-Length" => string(length(request.response.body)))
        # https://github.com/fonsp/Pluto.jl/pull/722
        HTTP.setheader(http, "Referrer-Policy" => "same-origin")
        # https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#:~:text=is%202%20minutes.-,14.38%20Server
        HTTP.startwrite(http)
        write(http, request.response.body)
    catch e
        if isa(e, Base.IOError) || isa(e, ArgumentError)
            # @warn "Attempted to write to a closed stream at $(request.target)"
            @debug "that is fine error" exception=(e, catch_backtrace())
        else
            rethrow(e)
        end
    end
end

sleep(2)

HTTP.get("http://localhost:$port/pid")

you will see that while the outer call to os.getpid() works, the inner raises the segmentation fault.

[cjdoris]

PythonCall is not thread safe. Presumably HTTP is automatically threading your get_pid handler. See the linked article for advice.

[schlichtanders]

Thank you for linking this known problem to this one. It was quite tough to debug this.

I think it could be very helpful for others if a concrete example of the stacktrace is also included in the not-thread-safe-documentation. This helps identifying the problem even if threads are not obvious at first.

here my stacktrace version

[1] signal (11.1): Segmentation fault
in expression starting at REPL[18]:1
_PyInterpreterState_GET at /usr/local/src/conda/python-3.12.0/Include/internal/pycore_pystate.h:118 [inlined]
get_state at /usr/local/src/conda/python-3.12.0/Objects/obmalloc.c:866 [inlined]
_PyObject_Malloc at /usr/local/src/conda/python-3.12.0/Objects/obmalloc.c:1563 [inlined]
PyObject_Malloc at /usr/local/src/conda/python-3.12.0/Objects/obmalloc.c:801 [inlined]
PyUnicode_New at /usr/local/src/conda/python-3.12.0/Objects/unicodeobject.c:1208 [inlined]
unicode_decode_utf8 at /usr/local/src/conda/python-3.12.0/Objects/unicodeobject.c:4647
PyUnicode_DecodeUTF8 at /home/myhome/.julia/packages/PythonCall/qTEA1/src/cpython/pointers.jl:299 [inlined]
pystr_fromUTF8 at /home/myhome/.julia/packages/PythonCall/qTEA1/src/concrete/str.jl:1 [inlined]
pystr_fromUTF8 at /home/myhome/.julia/packages/PythonCall/qTEA1/src/concrete/str.jl:2
pystr at /home/myhome/.julia/packages/PythonCall/qTEA1/src/concrete/str.jl:10 [inlined]
Py at /home/myhome/.julia/packages/PythonCall/qTEA1/src/Py.jl:138 [inlined]
macro expansion at /home/myhome/.julia/packages/PythonCall/qTEA1/src/Py.jl:130 [inlined]
pyimport at /home/myhome/.julia/packages/PythonCall/qTEA1/src/concrete/import.jl:11
get_pid at ./REPL[12]:3
unknown function (ip: 0x7fbfa41ed942)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
Router at /home/myhome/.julia/packages/HTTP/SN7VW/src/Handlers.jl:439
unknown function (ip: 0x7fbfa41ecae2)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
#3 at ./REPL[16]:4
unknown function (ip: 0x7fbfa41e1cb2)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:819 [inlined]
invokelatest at ./essentials.jl:816 [inlined]
handle_connection at /home/myhome/.julia/packages/HTTP/SN7VW/src/Servers.jl:450
macro expansion at /home/myhome/.julia/packages/HTTP/SN7VW/src/Servers.jl:386 [inlined]
#16 at ./task.jl:514
unknown function (ip: 0x7fbfa41d7f6f)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
Allocations: 23329414 (Pool: 23311485; Big: 17929); GC: 35

[cjdoris]

Good idea.