search

kearnz / autoimpute

Python package for Imputation Methods
MIT License
241 stars 19 forks source link

Imputation is stuck during the transform step #69

Closed sanchit97 closed 3 years ago

sanchit97 commented 3 years ago

I am using MICE imputer to perform imputations on a dataframe. However, during the imputerobj.transform step, the multiprocess sampling looks like it starts, but never progresses (even after 10 minutes of wait). Any solutions? @kearnz

No errors but stuck at following for >10 minutes. Dataframe size is about 4000*28:

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [σ, beta, alpha]
 |------------------------------------------------------| 0.00% [0/8000 00:00<00:00 Sampling 4 chains, 0 divergences
sanchit97 commented 3 years ago

Okay, looks like its because of PyMc3 sampling time. Never mind

kearnz commented 3 years ago

@sanchit97 closing this as it looks like you got what you need. let me know if any issues persist.

As a note, 4000*28 will result in a very long process, especially if you're trying to impute every column.

working with fewer columns is a better option... more rows shouldn't be a problem.