statsmodels / statsmodels

Statsmodels: statistical modeling and econometrics in Python
http://www.statsmodels.org/devel/
BSD 3-Clause "New" or "Revised" License
10.15k stars 2.88k forks source link

Warnings cannot be ignored #9179

Open jaydu1 opened 7 months ago

jaydu1 commented 7 months ago

When running statsmodels.discrete.discrete_model.NegativeBinomial(...).fit_regularized(...), I encountered a warning from

https://github.com/statsmodels/statsmodels/blob/192670b31c9069b247b0901d66013853e2ca47b7/statsmodels/base/l1_solvers_common.py#L70-L71

Even though the results from model.summary look ok, the warning cannot be turned off and ignored with warnings.filterwarnings('ignore'). Below is the code that reproduces the problem:

import statsmodels as stats
import statsmodels.api as sm
import numpy as np
import warnings
with warnings.catch_warnings():
    warnings.filterwarnings("ignore")
    np.random.seed(0)
    Y = np.random.negative_binomial(1, 0.5, size=(100, ))

    X = np.random.normal( size=(100, 5))

    mod = stats.discrete.discrete_model.NegativeBinomial(Y, X).fit_regularized(
        method='l1', alpha=1e-12, trim_mode='off',
        method_kwargs={'warn_convergence': False}, qc_verbose=False, disp=False)

Thanks

josef-pkt commented 7 months ago

Thanks, confirmed, (it's not notebook specific) There is a hardcoded

        with warnings.catch_warnings():
                warnings.simplefilter("always")

but strange part is that it is when computing start_params, not in the main fit. When I provide start_params in the example, then the warning does not show up.

my guess is that this was supposed to set "ignore" instead of "always" to suppress problems in start_params computation.

The same pattern shows up in many/all fit_regualrized in discrete_models GenericZeroInflated fit_regularized does not do anything with warnings. _get_start_params for regular fit in ZI subclasses uses "ignore": warnings.simplefilter("ignore", category=ConvergenceWarning)