search

SauceCat / PDPbox

python partial dependence plot toolbox
http://pdpbox.readthedocs.io/en/latest/
MIT License
840 stars 129 forks source link

ValueError: No objects to concatenate error using pdp.pdp_isolate #51

Open chriseal opened 5 years ago

chriseal commented 5 years ago

I've got pdp_isolate to work on other features, but it throws an exception "ValueError: No objects to concatenate" when plotted for column "f66". Does this mean there isn't enough information to make a plot of this feature?

Code that draws the error is:

pdp_feature = pdp.pdp_isolate(
    model=model, 
    dataset=generic_features, 
    model_features=generic_features.columns.tolist(), 
    feature="f66",
    num_grid_points=10, grid_type='percentile', percentile_range=None, grid_range=None, cust_grid_points=None, memory_limit=0.5, n_jobs=1, predict_kwds={}, data_transformer=None
)

Error printout is:

ValueError                                Traceback (most recent call last)
<ipython-input-48-65f3cae83da8> in <module>
      7         model_features=self._xgb_col_names,
      8         feature=''.join(['f', str(self._featureName_to_featureIdx_map[feature])]),
----> 9         num_grid_points=10, grid_type='percentile', percentile_range=None, grid_range=None, cust_grid_points=None, memory_limit=0.5, n_jobs=1, predict_kwds={}, data_transformer=None
     10     )

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pdpbox\pdp.py in pdp_isolate(model, dataset, model_features, feature, num_grid_points, grid_type, percentile_range, grid_range, cust_grid_points, memory_limit, n_jobs, predict_kwds, data_transformer)
    165             ice_lines.append(ice_line_n_class)
    166     else:
--> 167         ice_lines = pd.concat(grid_results, axis=1)
    168
    169     # calculate the counts

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, sort, copy)
    226                        keys=keys, levels=levels, names=names,
    227                        verify_integrity=verify_integrity,
--> 228                        copy=copy, sort=sort)
    229     return op.get_result()
    230

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in __init__(self, objs, axis, join, join_axes, keys, levels, names, ignore_index, verify_integrity, copy, sort)
    260
    261         if len(objs) == 0:
--> 262             raise ValueError('No objects to concatenate')
    263
    264         if keys is None:

ValueError: No objects to concatenate

The array of data is:

array([ 1.42646187e-04,  2.20339505e-03, -3.71780779e-02, -3.07990126e-02,
       -1.16102087e-03, -1.56650202e-02, -2.06472276e-03, -2.08325083e-02,
        1.91286310e-02,  8.36141875e-03, -8.11077609e-04, -1.92386611e-02,
        2.00920603e-02, -3.85844310e-02, -3.05273896e-03,  1.50174930e-03,
        4.46882570e-03, -2.53156515e-02,  7.88625472e-03, -4.30667359e-03,
       -9.39565665e-03,  8.74410281e-04,  5.34033060e-02, -8.75319328e-03,
       -9.87543920e-03, -5.46208786e-03, -6.50628504e-03, -7.57054927e-03,
       -3.93052166e-03,  2.16708143e-03,  1.42646187e-04, -7.25448253e-03,
       -1.06866675e-02,  3.59743997e-02, -3.22111433e-03, -7.26964438e-03,
        2.44697544e-02,  1.66259945e-02, -3.95567627e-03,  1.93527167e-02,
        1.64243560e-03,  2.66140257e-02,  3.36265867e-02, -1.45173875e-03,
       -6.50628504e-03,  2.01777145e-02,  5.00185982e-03, -2.68591401e-02,
        5.35982088e-03,  7.15823115e-02, -4.28643707e-03,  1.62057894e-02,
       -6.83444130e-03,  5.02746656e-02,  2.21616214e-03, -9.68313760e-03,
       -3.70581025e-03,  8.12628710e-03, -3.70581025e-03, -8.75319328e-03,
        9.87255512e-03,  6.73500014e-03,  5.45383777e-02, -3.86630195e-03,
        2.85044068e-03,  3.06522130e-02,  3.29460041e-03,  9.06824300e-03,
       -3.16553335e-03,  6.66199922e-03,  5.24429083e-04, -3.71585490e-03,
        6.27464537e-03, -8.96973965e-03,  8.17747652e-03, -5.51070443e-04,
        5.00562964e-03, -2.66748822e-02,  1.38245766e-02,  6.50494563e-04,
       -9.44020712e-03,  1.96937834e-02, -2.77638032e-03, -1.82182974e-04,
        7.87300556e-03,  4.61156800e-03, -4.69126662e-02,  1.10852780e-02,
        9.89909316e-04,  3.60295491e-04,  1.57297735e-02,  4.62766075e-02,
        1.33140739e-02,  6.19039552e-03,  1.39541582e-02, -1.69532139e-03,
       -1.16102087e-03,  3.80017680e-03, -6.50628504e-03,  6.10508974e-03,
        1.15785507e-03,  1.68395233e-02,  3.28879539e-03,  1.30998048e-02,
       -7.75223512e-03,  1.76696777e-02, -2.06107510e-03,  2.55053545e-02,
       -4.71198937e-02,  6.10950021e-02,  4.04423462e-05, -3.27229081e-02,
       -1.28115640e-03, -8.02660978e-02,  4.14655803e-02, -2.31697945e-02,
        3.25156125e-02,  3.42166041e-02, -8.96781154e-02,  1.60982957e-03,
        4.81951811e-02,  1.33224275e-03,  1.14265095e-02, -2.24592418e-02,
       -2.36253054e-03,  2.93535839e-02,  1.33670513e-02,  6.95616360e-03,
       -4.05554484e-02, -1.17983934e-02, -4.75658884e-03, -1.99465422e-02,
        1.67505552e-02, -3.21242621e-03,  4.16127660e-03, -3.64498680e-02,
       -2.67821052e-03, -1.50303162e-02,  1.17175118e-02,  1.08760602e-02,
       -5.16998437e-03, -2.69782135e-04, -2.70069166e-03, -4.10286181e-02,
        1.24384512e-03,  1.07679725e-03, -2.89405080e-02, -1.72169721e-03,
        6.27027032e-02, -5.21718738e-04,  3.55731258e-02, -2.26476632e-02,
        1.39934603e-02,  1.19824431e-02, -1.38995866e-02,  4.92600824e-03,
       -7.93047226e-03,  4.36035646e-02, -1.69812569e-04, -6.50628504e-03,
       -7.92308831e-02,  1.37241203e-02,  1.26096808e-02,  1.72127291e-03,
       -2.10604483e-02, -3.86630195e-03,  1.76504359e-02, -6.50628504e-03,
       -4.36824084e-03,  3.47388067e-03,  2.25694330e-03, -1.09188288e-02,
       -2.17498918e-02, -2.32474048e-02,  2.99691884e-02, -2.52163580e-02,
        1.43217263e-02, -8.43296067e-04,  2.70521511e-02, -4.18762082e-02,
       -1.55742569e-02,  9.67830047e-02,  3.12716409e-02,  1.42646187e-04,
        3.46233434e-02, -4.70378050e-03, -1.00597171e-02, -3.10874944e-03,
        8.58906318e-04, -5.80679408e-04, -1.04566714e-02,  7.34680904e-03,
        1.05924652e-02, -2.06613277e-02, -3.16553335e-03,  1.77879256e-03,
       -4.37393282e-02, -5.83798230e-03, -6.18573615e-03, -1.14453858e-02,
       -7.59866074e-03, -2.21871359e-02, -3.16553335e-03, -1.00102993e-04,
        1.18235288e-03,  8.06930693e-03, -1.96816126e-04,  1.84993239e-02,
        2.13461349e-02,  7.07266741e-03,  1.42646187e-04, -5.47744218e-03,
       -1.03254946e-02, -1.73400374e-02, -2.58117361e-04,  1.24010993e-02,
        3.92948733e-03, -1.82873090e-02,  1.75378801e-04, -2.85078366e-03,
        5.82887766e-03, -9.36706224e-03,  2.93535839e-02,  1.69861258e-02,
        1.29833871e-02, -2.34977297e-03,  4.12969746e-04, -4.58429383e-03,
       -1.97912848e-02, -1.71597336e-02,  1.00754861e-02,  1.46873022e-02,
        1.29070842e-02, -8.75319328e-03, -2.44563897e-02, -2.40753008e-02,
       -2.29969049e-03,  5.32787323e-03, -9.60620144e-04,  5.82650104e-03,
        1.23422518e-03,  1.23086831e-02, -4.98560347e-03, -2.14837586e-03,
       -2.19718025e-03,  3.04158477e-03,  1.42646187e-04, -6.03845600e-03,
        2.68922183e-03,  4.90456739e-03, -2.64048818e-02, -4.25583690e-03,
        1.13904307e-03,  1.82139863e-02,  5.86379367e-03,  3.09502502e-02,
        1.04408664e-02,  3.30992549e-02,  3.26630269e-03, -8.84241306e-03,
        2.53524525e-03, -1.93375418e-02, -1.65741420e-03,  3.01436365e-02,
       -3.25712075e-04, -3.89183492e-02, -2.70689577e-03,  1.42646187e-04,
       -4.11961023e-02, -1.19694286e-02,  3.78084147e-03,  5.03621425e-03,
       -1.31770086e-03,  1.73333962e-03,  1.83711198e-02,  4.24728106e-03,
        4.81215520e-03, -2.05966443e-03,  1.55374284e-02,  2.09652199e-04,
       -5.55169662e-02,  1.13634914e-02,  2.09302581e-02,  1.42646187e-04,
        1.55923464e-02,  5.68731417e-03, -3.66920155e-03,  3.35823081e-02,
        2.59721886e-02,  1.55376680e-02,  1.98637057e-03, -5.16036812e-02,
        2.23336166e-02,  8.63353952e-03,  7.94881900e-03, -6.61724505e-04,
        1.48939556e-02,  1.56944799e-02, -1.52858182e-02,  1.28012873e-03,
       -1.90668483e-02, -6.50628504e-03,  7.59052706e-03, -3.16190751e-02,
       -2.24662311e-02,  1.34058745e-03, -2.34977297e-03,  6.91485104e-03,
        2.07674911e-02, -3.43720015e-03,  5.12697636e-03,  5.68418815e-03,
       -1.98302377e-02,  3.17619165e-03,  2.87206334e-02, -4.16068265e-04,
       -1.70174842e-02, -4.25886490e-03, -1.90537711e-02,  1.40145629e-02,
        1.91054495e-03, -1.76012040e-02,  5.62004485e-02, -7.56296167e-03,
        1.93897412e-02, -1.08269795e-03, -4.12494685e-02, -8.52674004e-02,
        2.93535839e-02, -2.34977297e-03, -4.35269843e-02,  8.49059462e-04,
       -3.10987210e-02, -2.11906108e-03,  2.75347206e-02, -1.16102087e-03,
       -2.14935108e-04, -9.37163282e-04,  2.08917108e-02, -1.49624887e-04,
       -1.20360853e-03, -1.84003701e-02,  5.69942226e-02, -6.51881530e-03,
        1.19581645e-02,  9.34846000e-03,  1.95967201e-02,  1.83098260e-02,
        1.50381701e-02, -1.01716106e-02,  1.19326200e-02, -8.45938115e-05,
       -5.39846427e-02,  4.12628460e-02,  5.94171647e-02, -3.16112209e-02,
       -1.04566714e-02,  2.13981361e-03,  2.17117295e-03,  5.24219656e-03,
       -7.49931840e-04, -7.69360893e-02,  1.04539920e-03, -1.25689487e-02,
       -1.11945172e-02, -1.06232041e-02,  7.95284125e-03, -1.19005699e-03,
       -7.01669455e-03,  1.35516141e-03, -1.18424677e-02,  7.34416499e-03,
        1.82665318e-02, -6.50628504e-03, -1.66917762e-04,  7.93482177e-03,
        7.15381390e-02,  7.11127527e-05, -1.48615269e-03,  2.64163194e-02,
       -4.44932982e-02, -2.34977297e-03,  4.55293195e-02, -8.35069547e-03,
        3.70642948e-03,  1.53445914e-02,  5.59201845e-03, -4.88137601e-02,
        1.52571208e-03,  2.52508119e-03, -1.16102087e-03,  1.21301764e-02,
        7.62865196e-03, -9.38930734e-03, -1.87311721e-03,  1.73434631e-02,
       -9.26327361e-03, -7.73302270e-03, -8.66310438e-03,  1.59627685e-02,
        2.15100444e-02, -2.01720393e-02, -6.26028935e-03, -1.04566714e-02,
        1.94997264e-02,  7.40022891e-03,  6.63133282e-02, -2.86199529e-02,
        1.88261116e-03,  1.98344906e-03,  2.16258155e-03,  1.30730564e-02,
       -1.16102087e-03, -2.62273463e-02, -6.53665300e-03, -2.45184235e-02,
       -4.12609788e-02, -2.54090496e-03, -2.54805998e-02,  6.54251064e-03,
       -6.50628504e-03,  1.90305898e-02,  3.77064587e-02,  7.18832359e-03,
       -3.14250026e-04, -3.75230486e-02, -3.92870243e-03,  1.00036256e-02,
        1.99581498e-02, -1.45951448e-02,  3.94904799e-04, -1.47333697e-02,
       -1.12384254e-02, -2.02326112e-02,  1.89755598e-02, -5.89852839e-03,
        7.44942716e-03, -1.07168755e-02, -1.68000304e-02,  1.22239083e-03])
imranahmed96 commented 5 years ago

Just had this bug, but was able to resolve by removing nan values.

I can't see any nans in the array you posted above, but maybe try a quick variable clean?

chriseal commented 5 years ago

I've checked the data, and as far as I can tell, as long as the feature doesn't include any null values in the array, it is not drawing this error. Since I used XGBoost, I had to mask the feature set for non-null values in the column of choice.

So it seems, I was unable to recreate the error. Though, I'm not entirely sure which dataset I was using when I posted this, so can't entirely attribute it to a variable clean. I'll repost if I find it again.

mattharrison commented 5 years ago

Ran into this issue as well. Any plans for supporting NaN? (since xgboost and other boosting libraries do)

wuyuyu1024 commented 3 years ago

My data has no NaN but still runs to this issue...btw I'm using XGBoost