bertsky
commented
2 years ago Note: I was trying to warmstart from calamari_models_experimental/deep3_fraktur19/0.ckpt.json.
Is there a known workaround?
Closed bertsky closed 2 years ago
bertsky
commented
2 years ago Note: I was trying to warmstart from calamari_models_experimental/deep3_fraktur19/0.ckpt.json.
Is there a known workaround?
andbue
commented
2 years ago Sorry, I've never encountered it myself. Since it disappeared in #314 after a reinstall, could you provide some more information on how you installed calamari?
Here is a dockerfile that I use on our hpc cluster, maybe that helps:
FROM nvidia/cuda:11.5.1-cudnn8-runtime-ubuntu18.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends software-properties-common && \
add-apt-repository -y ppa:ubuntu-toolchain-r/ppa && \
apt-get purge -y --auto-remove software-properties-common && \
apt-get install -y --no-install-recommends build-essential && \
apt-get install -y --no-install-recommends python3.7 python3.7-dev python3-pip python3-setuptools python3-wheel python3-distutils && \
apt clean && \
python3.7 -m pip --no-cache-dir install --upgrade pip && \
python3.7 -m pip --no-cache-dir install https://github.com/Calamari-OCR/calamari/tarball/master && \
apt-get purge -y --auto-remove build-essential && \
rm -rf /var/lib/apt/lists/*I installed from PyPI. Just tried --force-reinstall to no avail. Also tried updating, which gave me 2.2.2 intead of 2.2.1, but still no change (the error persists).
(Note: I cannot use Docker in my cluster, only Singularity – but that would not help with the different CPU arch.)
bertsky
commented
2 years ago Ah, found the culprit: It seems that TF has a soft dependency on certain Numpy versions. In my case, downgrading to numpy==1.19.5 for tensorflow==2.4.1 helped avoid the above error.
(I have now worked myself up to a problem with warmstarting... coming up.)
Thanks for your feedback!
bertsky
commented
2 years ago For the sake of completeness:
Here is the SO issue that brought me on the right track.
And this is what I had to do ultimately:
pip install --no-use-pep517 numpy==1.19.5Note: the issue re-appeared for me when I tried training models for Calamari 1 again (v1.0.5, everything else as above). This time numpy==1.19.5 did not help. Neither did installing OpenCV 4.5.5 (instead of 4.2.0).
Meanwhile, I found this TF issue, which seems to explain what we see, but offers no solution to date – only tf.compat.v1.disable_eager_execution() as a workaround.
bertsky
commented
2 years ago FYI, I just tried that in a brutish way, by adding that line to calamari_ocr.ocr.backends.tensorflow_backend right at the top at module level.
That made the error…
TypeError: Keras symbolic inputs/outputs do not implement `op`. You may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.…go away, but now I get…
NotImplementedError: Cannot convert a symbolic Tensor (ReadVariableOp_32:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supportedSomething's wrong in the state of Denmark here.
I have Calamari 2.2.1 / Python 3.7.4 / TF 2.4.1 / Numpy 1.21.6, and attempt calamari-train in a (Power9-based) HPC environment.
Unfortunately, I get the following exception:
stacktrace
``` INFO 2022-04-26 18:48:37,860 tfaip.trainer.trainer: Setting samples per epoch relative to dataset size with a factor of 1. INFO 2022-04-26 18:48:37,860 tfaip.trainer.trainer: Set steps per epoch to 18 INFO 2022-04-26 18:48:37,874 tfaip.scenario.scenariobase: Building training keras model INFO 2022-04-26 18:48:37,881 tfaip.scenario.scenariobase: Attempting to set no train scope INFO 2022-04-26 18:48:37,881 tfaip.scenario.scenariobase: Compiling training model including optimization INFO 2022-04-26 18:48:37,892 tfaip.scenario.scenariobase: Interference on the input shapes to build the graph. CRITICAL 2022-04-26 18:48:39,339 tfaip.util.logging: Uncaught exception Traceback (most recent call last): File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/scripts/train.py", line 29, in main return trainer.train() File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/training/trainer.py", line 190, in train last_logs = super().train(callbacks=callbacks) File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/device/device_config.py", line 134, in wrapper return func(*args, **kwargs) File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/trainer/trainer.py", line 209, in train training_graph_only=self._training_graph_only, File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/scenario/scenariobase.py", line 627, in setup_training self._keras_train_model((real_inputs, wrapped_targets, real_meta)) File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 952, in __call__ input_list) File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1091, in _functional_construction_call inputs, input_masks, args, kwargs) File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 822, in _keras_tensor_symbolic_call return self._infer_output_signature(inputs, args, kwargs, input_masks) File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 863, in _infer_output_signature outputs = call_fn(inputs, *args, **kwargs) File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper raise e.ag_error_metadata.to_exception(e) NotImplementedError: in user code: /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:72 call * outputs = self.graph.train(inputs, pre_proc_targets) /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:272 train * return self(wrap((inputs, targets))) /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:267 call * return self._graph(inputs_targets) /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:312 call * outputs = self.build_train_graph(inputs, targets, training=training) /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/model/graphbase.py:343 build_train_graph * return self.build_graph(inputs, training) /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/graph.py:83 build_graph * last_layer_output = layer(last_layer_output) /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/layer.py:77 call * return self._call(inputs, **kwargs) /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/model/layers/bilstm.py:50 _call * return self.lstm(inputs) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:539 __call__ ** return super(Bidirectional, self).__call__(inputs, **kwargs) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__ outputs = call_fn(inputs, *args, **kwargs) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/wrappers.py:653 call initial_state=forward_state, **kwargs) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:660 __call__ return super(RNN, self).__call__(inputs, **kwargs) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012 __call__ outputs = call_fn(inputs, *args, **kwargs) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent_v2.py:1157 call inputs, initial_state, _ = self._process_inputs(inputs, initial_state, None) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:859 _process_inputs initial_state = self.get_initial_state(inputs) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:643 get_initial_state inputs=None, batch_size=batch_size, dtype=dtype) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2507 get_initial_state self, inputs, batch_size, dtype)) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:2987 _generate_zero_filled_state_for_cell return _generate_zero_filled_state(batch_size, cell.state_size, dtype) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py:3003 _generate_zero_filled_state return nest.map_structure(create_zeros, state_size) /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659 map_structure structure[0], [func(*x) for x in entries], /sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/util/nest.py:659Probably the same as #314.