search

google-research / albert

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Apache License 2.0
3.23k stars 571 forks source link

Prediction Fails on default Colab #259

Closed Jadiker closed 4 months ago

Jadiker commented 2 years ago

I'm using the default Colab file, and setting the do_predict flag to True (the only thing that I don't leave as default) results in the following error:

INFO:tensorflow:Calling model_fn.
I0301 00:58:45.533245 139927177623424 estimator.py:1148] Calling model_fn.
WARNING:tensorflow:Entity <function file_based_input_fn_builder.<locals>.input_fn.<locals>.<lambda> at 0x7f42e704c830> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Str'
W0301 00:58:45.551807 139927177623424 ag_logging.py:146] Entity <function file_based_input_fn_builder.<locals>.input_fn.<locals>.<lambda> at 0x7f42e704c830> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Str'
WARNING:tensorflow:Entity <function _InputsWithStoppingSignals.insert_stopping_signal.<locals>._map_fn at 0x7f42e704ccb0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Index'
W0301 00:58:45.584016 139927177623424 ag_logging.py:146] Entity <function _InputsWithStoppingSignals.insert_stopping_signal.<locals>._map_fn at 0x7f42e704ccb0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Index'
ERROR:tensorflow:Error recorded from prediction_loop: in converted code:

    /tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:3970 _map_fn
        batch_size))
    /tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:4073 pad_features_and_labels
        batch_size)
    /tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:4131 _padding_mask
        array_ops.zeros((real_batch_size,), dtype=dtypes.int32),
    /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/array_ops.py:2338 zeros
        output = _constant_if_small(zero, shape, dtype, name)
    /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/array_ops.py:2295 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:6 prod

    /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:3052 prod
        keepdims=keepdims, initial=initial, where=where)
    /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:86 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    /tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py:736 __array__
        " array.".format(self.name))

    NotImplementedError: Cannot convert a symbolic Tensor (strided_slice:0) to a numpy array.

E0301 00:58:45.619720 139927177623424 error_handling.py:75] Error recorded from prediction_loop: in converted code:

    /tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:3970 _map_fn
        batch_size))
    /tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:4073 pad_features_and_labels
        batch_size)
    /tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:4131 _padding_mask
        array_ops.zeros((real_batch_size,), dtype=dtypes.int32),
    /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/array_ops.py:2338 zeros
        output = _constant_if_small(zero, shape, dtype, name)
    /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/array_ops.py:2295 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:6 prod

    /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:3052 prod
        keepdims=keepdims, initial=initial, where=where)
    /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:86 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    /tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py:736 __array__
        " array.".format(self.name))

    NotImplementedError: Cannot convert a symbolic Tensor (strided_slice:0) to a numpy array.

INFO:tensorflow:prediction_loop marked as finished
I0301 00:58:45.619978 139927177623424 error_handling.py:101] prediction_loop marked as finished
WARNING:tensorflow:Reraising captured error
W0301 00:58:45.620124 139927177623424 error_handling.py:135] Reraising captured error
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/content/albert/run_classifier.py", line 558, in <module>
    tf.app.run()
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/content/albert/run_classifier.py", line 525, in main
    enumerate(zip(predict_examples, result)):
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3078, in predict
    rendezvous.raise_errors()
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 136, in raise_errors
    six.reraise(typ, value, traceback)
  File "/usr/local/lib/python3.7/dist-packages/six.py", line 703, in reraise
    raise value
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3072, in predict
    yield_single_examples=yield_single_examples):
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/estimator.py", line 622, in predict
    features, None, ModeKeys.PREDICT, self.config)
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2857, in _call_model_fn
    config)
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3148, in _model_fn
    input_holders.generate_infeed_enqueue_ops_and_dequeue_fn())
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 1428, in generate_infeed_enqueue_ops_and_dequeue_fn
    self._invoke_input_fn_and_record_structure())
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 1519, in _invoke_input_fn_and_record_structure
    self._inputs_structure_recorder, host_device, host_id))
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 987, in generate_per_host_v2_enqueue_ops_fn_for_host
    num_invocations_per_step=max(1, ctx.num_of_replicas_per_host))
  File "/tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3879, in __init__
    stop=False, batch_size=batch_size, add_padding=add_padding))
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 1909, in map
    MapDataset(self, map_func, preserve_cardinality=False))
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 3434, in __init__
    use_legacy_function=use_legacy_function)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 2713, in __init__
    self._function = wrapper_fn._get_concrete_function_internal()
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/eager/function.py", line 1853, in _get_concrete_function_internal
    *args, **kwargs)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/eager/function.py", line 1847, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/eager/function.py", line 2147, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/eager/function.py", line 2038, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 2707, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/data/ops/dataset_ops.py", line 2652, in _wrapper_helper
    ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in converted code:

    /tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:3970 _map_fn
        batch_size))
    /tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:4073 pad_features_and_labels
        batch_size)
    /tensorflow-1.15.2/python3.7/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:4131 _padding_mask
        array_ops.zeros((real_batch_size,), dtype=dtypes.int32),
    /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/array_ops.py:2338 zeros
        output = _constant_if_small(zero, shape, dtype, name)
    /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/array_ops.py:2295 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:6 prod

    /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:3052 prod
        keepdims=keepdims, initial=initial, where=where)
    /usr/local/lib/python3.7/dist-packages/numpy/core/fromnumeric.py:86 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    /tensorflow-1.15.2/python3.7/tensorflow_core/python/framework/ops.py:736 __array__
        " array.".format(self.name))

    NotImplementedError: Cannot convert a symbolic Tensor (strided_slice:0) to a numpy array.
Jadiker commented 2 years ago

From this, adding %pip install numpy==1.19.5 just before the code block that does the prediction seems to have fixed the issue.

Jadiker commented 2 years ago

You may need to restart the notebook after running this line in order to have it take effect. (However, it should warn you of this automatically.)