Error after change lattice size

[LumingSun]

Hi, I'm using CannedRegressor with CalibratedLatticeConfig, everything is fine when the lattice_size in FeatureConfig is set to 2, but after I change lattice_size to 4, I got the following error `Traceback (most recent call last): File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1812, in _create_c_op c_op = pywrap_tf_session.TF_FinishOperation(op_desc) tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid value in tensor used for shape: -2147483648

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/data1/luming.slm/cardinality/lattice_regression/lattice_regression.py", line 457, in estimator.train(input_fn=train_input_fn) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1175, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1204, in _train_model_default self.config) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1163, in _call_model_fn model_fn_results = self._model_fn(features=features, kwargs) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/estimators.py", line 729, in calibrated_lattice_model_fn dtype=dtype) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/estimators.py", line 652, in _calibrated_lattice_model_fn model = premade.CalibratedLattice(model_config=model_config, dtype=dtype) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/premade.py", line 235, in init dtype=dtype) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/premade_lib.py", line 579, in build_lattice_layer lattice_input) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 757, in call self._maybe_build(inputs) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 2098, in _maybe_build self.build(input_shapes) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/lattice_layer.py", line 399, in build dtype=self.dtype) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 448, in add_weight caching_device=caching_device) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py", line 750, in _add_variable_with_custom_getter kwargs_for_getter) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 145, in make_variable shape=variable_shape if variable_shape else None) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 260, in call return cls._variable_v1_call(*args, kwargs) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 221, in _variable_v1_call shape=shape) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 199, in previous_getter = lambda kwargs: default_variable_creator(None, kwargs) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2597, in default_variable_creator shape=shape) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 264, in call return super(VariableMetaclass, cls).call(*args, *kwargs) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1518, in init distribute_strategy=distribute_strategy) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1651, in _init_from_args initial_value() if init_from_fn else initial_value, File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/lattice_layer.py", line 711, in call dtype=dtype) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/lattice_lib.py", line 492, in linear_initializer weights = batch_outer_operation(one_d_weights, operation=tf.add) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow_lattice/python/lattice_lib.py", line 333, in batch_outer_operation result = tf.reshape(result, shape=new_shape) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper return target(args, kwargs) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 195, in reshape result = gen_array_ops.reshape(tensor, shape, name) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 8234, in reshape "Reshape", tensor=tensor, shape=shape, name=name) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3485, in _create_op_internal op_def=op_def) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1975, in init control_input_ops, op_def) File "/data1/luming.slm/anaconda3/envs/tfl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1815, in _create_c_op raise ValueError(str(e)) ValueError: Invalid value in tensor used for shape: -2147483648`

It's really strange. And I got the same error message when I set the lattice size as 2, but add several more common features (that have no monotonic constraints). Any idea about this? Thanks.

[mmilanifard]

The error message is somewhat strange and hard to debug without more context. Can you please share a minimal code snippet that would reproduce this error? Say a Colab notebook. Which version of TF and the TFL library are you using?

ValueError: Invalid value in tensor used for shape: -2147483648 One clue is that -2147483648 is the minimum value of int32.

[LumingSun]

Thanks for replying. I tried and found that it might be caused by large number of features and lattice size. I wanted to solve a task with 20 features, and It's ok when there are 20 input features with lattice size 2 (2^20), but fail with lattice size 3 (3^20 3486784401). And it also fail with 14 feature lattice size 5, 5^14 (6103515625) but OK with 4^14 (268435456) and 8^10 (1073741824).