json.decoder.JSONDecodeError when executing lgb.plot_tree

[AnchorBlues]

Hello. The JSONDecodeError occurred when I trying to visualize tree structure using the lgb.plot_tree function.

Environment info

Operating System: Ubuntu 14.04.5 LTS, Trusty Tahr

CPU/GPU model: cpu Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz

C++/Python/R version:

Python 3.7.0 (default, Jun 28 2018, 13:15:42) [GCC 7.2.0] :: Anaconda, Inc. on linux

lightgbm version : newest version (merged the commit until the commit id 087d30623aef4b505(Update README.md (#1790)))

source

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import lightgbm as lgb

# load data (binary classification)
test_size = 0.2
random_state = 42
data = load_breast_cancer()
x = data.data
y = data.target
x_train, x_test, y_train, y_test = train_test_split(x, y,
                                                    random_state=random_state,
                                                    test_size=test_size)
# create a new feature that takes a value -1 or 1.
np.random.seed(0)
dummy = 2 * np.random.randint(0, 2, size=len(x_train)) - 1

# add a small value to the dummy value of 0th instance
dummy[0] += 0.00001

x_train = np.c_[dummy, x_train]

# create json file that forces the tree to split by the dummy feature. threshold is 0.0.
s = """
{
    "feature": 0,
    "threshold": 0.0,
    "left": {
    },
    "right": {
    }
}
"""

with open("forced_splits-0.json", mode='w') as f:
    f.write(s)

# model training
model = lgb.LGBMClassifier(random_state=42, forced_splits="forced_splits-0.json", num_leaves=3)
model.fit(x_train, y_train)

# trying to visualize tree
ax = lgb.plot_tree(model, tree_index=0, figsize=(30, 8))
plt.savefig("./result.png")
plt.show()

error message

Traceback (most recent call last):
  File "<ipython-input-3-3f359887d4c5>", line 4, in <module>
    ax = lgb.plot_tree(model, tree_index=0, figsize=(30, 8))
  File "/home/anchorblues/anaconda3/envs/gbm/lib/python3.7/site-packages/lightgbm/plotting.py", line 447, in plot_tree
    show_info=show_info, precision=precision, **kwargs)
  File "/home/anchorblues/anaconda3/envs/gbm/lib/python3.7/site-packages/lightgbm/plotting.py", line 371, in create_tree_digraph
    model = booster.dump_model()
  File "/home/anchorblues/anaconda3/envs/gbm/lib/python3.7/site-packages/lightgbm/basic.py", line 2111, in dump_model
    ret = json.loads(string_buffer.value.decode())
  File "/home/anchorblues/anaconda3/envs/gbm/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/anchorblues/anaconda3/envs/gbm/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/anchorblues/anaconda3/envs/gbm/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 3958 column 14 (char 80286)

if the line

dummy[0] += 0.00001

is removed, the above code works and the error does not occur.

What is the cause of the error?

[AnchorBlues]

I found that line 3958 ( where json.decoder.JSONDecodeError occurred ) is as follows:

'"split_gain":nan,'

I think json can't manage 'nan' and raise json.decoder.JSONDecodeError. And the 99th tree structure of the above GBM model is as follows:

{'tree_index': 99,
 'num_leaves': 2,
 'num_cat': 0,
 'shrinkage': 0.1,
 'tree_structure': {'split_index': 0,
  'split_feature': 0,
  'split_gain': 'nan',
  'threshold': 1.0000000180025095e-35,
  'decision_type': '<=',
  'default_left': True,
  'missing_type': 'None',
  'internal_value': 0,
  'internal_count': 455,
  'left_child': {'leaf_index': 0,
   'leaf_value': 16899999938695358,
   'leaf_count': 454},
  'right_child': {'leaf_index': 1, 'leaf_value': 0, 'leaf_count': 1}}}

The leaf_value of the left_child is too large.

So I have two questions.

1. Why the split_gain has nan ?
2. Why the leaf_value has such a large value ?

[guolinke]

@AnchorBlues It is caused by your force-split. as these splits create very imbalance splits.

[AnchorBlues]

@guolinke I understand that is caused by imbalanced forced_split. Then, why the error does not occur when the line dummy[0] += 0.00001 is removed from the above code snippet ? I think this change doesn't change the degree of imbalance of splits.

[guolinke]

@AnchorBlues there is indeed a bug in this case. Thanks very much! fixed in #1809 . @jerryjliu any comments ?

[AnchorBlues]

@guolinke Thank you for fixing. After merging #1809 and building the sources, the error does not occur.

However, an another problem occurred. In the bellow case, memory error occurred.

source

import numpy as np
from sklearn.datasets import load_breast_cancer
import lightgbm as lgb

# load data (binary classification)
data = load_breast_cancer()
x_train = data.data
y_train = data.target

# create a new feature that takes a value 0 or 1.
np.random.seed(0)
dummy = np.random.randint(0, 2, size=len(x_train))
x_train = np.c_[dummy, x_train]

# create json file that forces the tree to split by the new feature. threshold is 0.5.
s = """
{
    "feature": 0,
    "threshold": 0.5,
    "left": {
    },
    "right": {
    }
}
"""

with open("forced_splits-0.json", mode='w') as f:
    f.write(s)

# model training
model = lgb.LGBMClassifier(random_state=42, forced_splits="forced_splits-0.json", num_leaves=3, n_estimators=10)
model.fit(x_train, y_train)

error message

[LightGBM] [Fatal] Check failed: tree->num_leaves() <= data_partition_->num_leaves() at /home/anchorbues/packages/LightGBM/src/treelearner/serial_tree_learner.h, line 60 .

This code didn't raise memory error at the 2.1.2 version of LigthGBM ( show https://github.com/Microsoft/LightGBM/issues/1783 ).

[guolinke]

@AnchorBlues I think you directly use the code in that branch? That branch is based on an out-of-date master branch. I just update it and everything should be fine now.

[AnchorBlues]

@guolinke Thanks! After building the newest sources including commit #1809, every code including forced-split work well.