python asgi stop response

[lovesharepc]

os: ubuntu server 22.04 unit version: 1.31.0 unit config

{
        "access_log": {
                "format": "{\"time_local\":\"$time_local\",\"status\":\"$status\",\"uri\":\"$uri\",\"request_uri\":\"$request_uri\",\"request_line\":\"$request_line\",\"remote_addr\":\"$remote_addr\",\"method\":\"$method\",\"host\":\"$host\",\"header_user_agent\":\"$header_user_agent\",\"header_referer\":\"$header_referer\",\"body_bytes_sent\":\"$body_bytes_sent\"}",
                "path": "/opt/nebula/log/nebula-statistics/nebula-statistics_unit_access.log"
        },

        "applications": {
                "nebula-statistics_api_1": {
                        "isolation": {
                                "namespaces": {
                                        "cgroup": true
                                }
                        },

                        "working_directory": "/opt/nebula/venv/venv_src/nebula-statistics_2023-11-22_1700617272",
                        "limits": {
                                "requests": 1700
                        },

                        "protocol": "asgi",
                        "path": "/opt/nebula/venv/venv_src/nebula-statistics_2023-11-22_1700617272",
                        "processes": 6,
                        "threads": 2,
                        "home": "/opt/nebula/venv/venv_src/nebula-statistics_2023-11-22_1700617272",
                        "callable": "app",
                        "type": "python 3.10",
                        "module": "nebula_statistics.asgi"
                }
        },

        "listeners": {
                "unix:/opt/nebula/socket/nebula-statistics_api_1.sock": {
                        "pass": "applications/nebula-statistics_api_1"
                }
        },

        "settings": {
                "http": {
                        "max_body_size": 104857600
                }
        }
}

my unit server will become stop response current I open debug log to check I find the requst is done (loadbalance log) but unit like missing this info here is debug log and there is no asgi_http_done and all unit log are stop so I think is grace quit is stock

2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] allocated_chunks 0
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] port{3304536,1} enqueue 28 notify 0
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: send mmap: (0,0,5648)
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] outgoing mmap allocation: (0,0,16384)
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] allocated_chunks 1
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] port{3304536,1} sendmsg 244
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] sendmsg(11, 244, 0): 244
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] port{3304536,1} enqueue 1 read_socket notify 1
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: send plain: 244
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: send: 4 fields, 228 bytes
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: write: 5648
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: asgi_http_response_body: 5648, 0
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: asgi_http_send type is 'http.response.body'
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: add_field #3: x-nebula-tracker-id: 88295d23-b820-462b-b27c-042548dd9781
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: add_field #2: x-nebula-session-expiry: 86400000
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: add_field #1: content-type: application/json
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: add_field #0: content-length: 5648
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] outgoing plain buffer allocation: (0x55f1a267e630, 228)
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: response_buf_alloc: 228
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: init: 200, max fields 4/132
2023-12-04 16:58:36.299 log=2023/12/04 16:58:36.299 [debug] 2367192#2367192 [unit] #508897: asgi_http_send type is 'http.response.start'
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] asgi_port_read(0x55f1a43cf248,0x55f1a2fda080): 2
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] port{2367192,0} recvmsg -1
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] recvmsg(18) failed: Resource temporarily unavailable (11)
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] asgi_port_read(0x55f1a43cf248,0x55f1a2fda080): 2
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] port{2367192,0} recv 16 read_queue
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] recvmsg(18): 16
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] asgi_port_read(0x55f1a43cf248,0x55f1a2fda080): 0
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] port{2367192,1} send 16 read_queue
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] sendmsg(33, 16, 0): 16
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] port{2367192,1} enqueue 17 notify 1
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] asgi_remove_reader 10 0x55f1a295c000
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] asgi_remove_port 10 0x55f1a295c000
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] quit: 1/1/1
2023-12-04 16:58:35.898 log=2023/12/04 16:58:35.898 [debug] 2367192#2367192 [unit] #0: graceful quit

please help me check what heppen here? thanks

[ac000]

Are you saying that at some point Unit stops serving requests?

Also, JFYI, this doesn't do what I'm guessing you think it does

                        "isolation": {
                                "namespaces": {
                                        "cgroup": true
                                }
                        },

I.e it doesn't actually create any cgroups. OTH if you're simply expecting it to change what appears in /proc/pid/cgroup and /proc/pid/mountinfo, then ignore me...

[lovesharepc]

yes, my unit stops serving requests

and thank let me know cgroup config , I will go to check

[ac000]

On Tue, 05 Dec 2023 01:17:58 -0800 owan @.***> wrote:

yes, my unit stops serving requests

If you're able to distil this down to a simple reproducer that would be helpful.

and thank let me know cgroup config , I will go to check

If you do actually want to have per-application cgroups, see here https://unit.nginx.org/configuration/#conf-app-cgroup

[lovesharepc]

I don't know how to do simple reproducer because this problem is not always happen does ther have any way to debug why unit stop serving requests ?

by the way, when unit stop serving requests the unit route process's FD will continuous increase

[ac000]

On Tue, 05 Dec 2023 17:43:36 -0800 owan @.***> wrote:

I don't know how to do simple reproducer because this problem is not always happen does ther have any way to debug why unit serving requests ?

Can you reproduce it without threads?

[lovesharepc]

thanks, I will go to test without threads