get_recourse and get_recourse_and_learn hang

step to reproduce: 1) build docker image

docker build -f .\Dockerfile --tag i3group/recourse-backend .

2) run container

docker run --rm -it -p 5000:5000 i3group/recourse-backend

2) make a http request (post)

http://localhost:5000/get_recourse_v2

with parameters

{
    "features": {
        "adult": [
            {
                "name": "age",
                "value": 27
            },
            {
                "name": "capital_gain",
                "value": 0
            },
            {
                "name": "capital_loss",
                "value": 0
            },
            {
                "name": "education",
                "value": "Bachelors"
            },
            {
                "name": "hours_per_week",
                "value": 40
            },
            {
                "name": "marital_status",
                "value": "Never-married"
            },
            {
                "name": "native_country",
                "value": "United-States"
            },
            {
                "name": "occupation",
                "value": "Adm-clerical"
            },
            {
                "name": "race",
                "value": "White"
            },
            {
                "name": "relationship",
                "value": "Not-in-family"
            },
            {
                "name": "sex",
                "value": "Female"
            },
            {
                "name": "workclass",
                "value": "Private"
            }
        ],
        "lendingclub": [
            {
                "name": "acc_now_delinq",
                "value": 0
            },
            {
                "name": "acc_open_past_24mths",
                "value": 6
            },
            {
                "name": "all_util",
                "value": 40
            },
            {
                "name": "annual_inc",
                "value": 13000
            },
            {
                "name": "application_type",
                "value": "Joint App"
            },
            {
                "name": "avg_cur_bal",
                "value": 1761
            },
            {
                "name": "bc_open_to_buy",
                "value": 1596
            },
            {
                "name": "bc_util",
                "value": 82.8
            },
            {
                "name": "chargeoff_within_12_mths",
                "value": 0
            },
            {
                "name": "collections_12_mths_ex_med",
                "value": 0
            },
            {
                "name": "debt_settlement_flag",
                "value": "N"
            },
            {
                "name": "delinq_2yrs",
                "value": 0
            },
            {
                "name": "delinq_amnt",
                "value": 0
            },
            {
                "name": "dti",
                "value": 82.36
            },
            {
                "name": "emp_length",
                "value": "1 year"
            },
            {
                "name": "fico_range_high",
                "value": 734
            },
            {
                "name": "fico_range_low",
                "value": 730
            },
            {
                "name": "grade",
                "value": "E"
            },
            {
                "name": "hardship_flag",
                "value": "N"
            },
            {
                "name": "home_ownership",
                "value": "RENT"
            },
            {
                "name": "il_util",
                "value": 57
            },
            {
                "name": "initial_list_status",
                "value": "f"
            },
            {
                "name": "inq_fi",
                "value": 1
            },
            {
                "name": "inq_last_12m",
                "value": 2
            },
            {
                "name": "inq_last_6mths",
                "value": 2
            },
            {
                "name": "installment",
                "value": 809.02
            },
            {
                "name": "int_rate",
                "value": 26.3
            },
            {
                "name": "last_fico_range_high",
                "value": 664
            },
            {
                "name": "last_fico_range_low",
                "value": 660
            },
            {
                "name": "loan_amnt",
                "value": 20000
            },
            {
                "name": "max_bal_bc",
                "value": 4617
            },
            {
                "name": "mo_sin_old_il_acct",
                "value": 25
            },
            {
                "name": "mo_sin_old_rev_tl_op",
                "value": 91
            },
            {
                "name": "mo_sin_rcnt_rev_tl_op",
                "value": 20
            },
            {
                "name": "mo_sin_rcnt_tl",
                "value": 20
            },
            {
                "name": "mort_acc",
                "value": 0
            },
            {
                "name": "mths_since_last_delinq",
                "value": 18
            },
            {
                "name": "mths_since_last_major_derog",
                "value": 30.5
            },
            {
                "name": "mths_since_rcnt_il",
                "value": 20
            },
            {
                "name": "mths_since_recent_bc",
                "value": 35
            },
            {
                "name": "mths_since_recent_inq",
                "value": 0
            },
            {
                "name": "mths_since_recent_revol_delinq",
                "value": 9
            },
            {
                "name": "num_accts_ever_120_pd",
                "value": 0
            },
            {
                "name": "num_actv_bc_tl",
                "value": 3
            },
            {
                "name": "num_actv_rev_tl",
                "value": 5
            },
            {
                "name": "num_bc_sats",
                "value": 4
            },
            {
                "name": "num_bc_tl",
                "value": 4
            },
            {
                "name": "num_il_tl",
                "value": 3
            },
            {
                "name": "num_op_rev_tl",
                "value": 11
            },
            {
                "name": "num_rev_accts",
                "value": 13
            },
            {
                "name": "num_rev_tl_bal_gt_0",
                "value": 5
            },
            {
                "name": "num_sats",
                "value": 13
            },
            {
                "name": "num_tl_120dpd_2m",
                "value": 0
            },
            {
                "name": "num_tl_30dpd",
                "value": 0
            },
            {
                "name": "num_tl_90g_dpd_24m",
                "value": 0
            },
            {
                "name": "num_tl_op_past_12m",
                "value": 0
            },
            {
                "name": "open_acc",
                "value": 13
            },
            {
                "name": "open_acc_6m",
                "value": 0
            },
            {
                "name": "open_act_il",
                "value": 2
            },
            {
                "name": "open_il_12m",
                "value": 0
            },
            {
                "name": "open_il_24m",
                "value": 2
            },
            {
                "name": "open_rv_12m",
                "value": 0
            },
            {
                "name": "open_rv_24m",
                "value": 4
            },
            {
                "name": "pct_tl_nvr_dlq",
                "value": 100
            },
            {
                "name": "percent_bc_gt_75",
                "value": 75
            },
            {
                "name": "policy_code",
                "value": 1
            },
            {
                "name": "pub_rec",
                "value": 0
            },
            {
                "name": "pub_rec_bankruptcies",
                "value": 0
            },
            {
                "name": "purpose",
                "value": "debt_consolidation"
            },
            {
                "name": "pymnt_plan",
                "value": "n"
            },
            {
                "name": "revol_bal",
                "value": 9173
            },
            {
                "name": "revol_util",
                "value": 28.8
            },
            {
                "name": "sub_grade",
                "value": "E5"
            },
            {
                "name": "tax_liens",
                "value": 0
            },
            {
                "name": "term",
                "value": "36 months"
            },
            {
                "name": "tot_coll_amt",
                "value": 0
            },
            {
                "name": "tot_cur_bal",
                "value": 21129
            },
            {
                "name": "tot_hi_cred_lim",
                "value": 52900
            },
            {
                "name": "total_acc",
                "value": 16
            },
            {
                "name": "total_bal_ex_mort",
                "value": 21129
            },
            {
                "name": "total_bal_il",
                "value": 11956
            },
            {
                "name": "total_bc_limit",
                "value": 9300
            },
            {
                "name": "total_cu_tl",
                "value": 3
            },
            {
                "name": "total_il_high_credit_limit",
                "value": 21000
            },
            {
                "name": "verification_status",
                "value": "Verified"
            }
        ]
    },
    "preferences": {}
}

Hi Giovanni, In the last iteration of dockerfile I removed uwsgi because it runs as admin but it is not a good idea. The problem still persists. On Mon, 21 Aug 2023 at 23:58, Giovanni De Toni @.***> wrote:

I had a look at this, and it seems related to how uwsgi deals with threads and python.

Basically, the app is loaded in a pre-forking mode, in which the workers are just "forked" from the main process. Such processes should share all the objects of the main process, but sometimes there are issues with it. In our case, the predict() method of our pytorch models hangs, thus blocking the entire app

If we run uwsgi --ini uwsgi.ini --lazy-apps the problem disappears, at the cost of using much more RAM (each process will have its own copy of the ML models).

I did not find any clever way to fix it nicely, since I myself have no idea where the issue could lie.

— Reply to this email directly, view it on GitHub https://github.com/geektoni/RecourseBackend/issues/2#issuecomment-1687100173, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALGBZO2SMEUI6US3LKCIGNDXWPKYBANCNFSM6AAAAAA3TRZW2U . You are receiving this because you authored the thread.Message ID: @.***>

--

Le informazioni contenute nella presente comunicazione sono di natura privata e come tali sono da considerarsi riservate ed indirizzate esclusivamente ai destinatari indicati e per le finalità strettamente legate al relativo contenuto. Se avete ricevuto questo messaggio per errore, vi preghiamo di eliminarlo e di inviare una comunicazione all’indirizzo e-mail del mittente.

-- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material.

geektoni / RecourseBackend

get_recourse and get_recourse_and_learn hang #2

--