When using --warmstart with a (ckpt.json) model that uses non-default network architecture, I end up with incompatible shapes:
INFO 2022-04-27 18:46:19,061 tfaip.trainer.warmstart.warmst: Source model successfully loaded for warmstart.
ERROR 2022-04-27 18:46:19,074 tfaip.trainer.warmstart.warmst: Cannot assign to variable root/root/CalamariGraph/lstm_0/bidirectional/forward_lstm/lstm_cell_1/kernel:0 due to variable shape (720, 800) and valu
e shape (3, 3, 60, 120) are incompatible
Traceback (most recent call last):
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/tfaip/trainer/warmstart/warmstarter.py", line 144, in warmstart
target_model, [np.asarray(w) for w in src_model.weights if w.name not in names_to_ignore]
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/training/warmstart.py", line 33, in apply_weights
self.copy_weights_from_model(target_model, new_weights, *self.codec_changes)
File "/home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages/calamari_ocr/ocr/training/warmstart.py", line 38, in copy_weights_from_model
target_weight.assign(source_weight)
File "/sw/installed/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 891, in assign
(tensor_name, self._shape, value_tensor.shape))
ValueError: Cannot assign to variable root/root/CalamariGraph/lstm_0/bidirectional/forward_lstm/lstm_cell_1/kernel:0 due to variable shape (720, 800) and value shape (3, 3, 60, 120) are incompatible
Also passing the matching --network deep3 made the problem disappear.
This is somewhat unexpected, since I would assume that the architecture can be derived from the warmstart.
Maybe just a documentation issue:
When using
--warmstart
with a (ckpt.json) model that uses non-default network architecture, I end up with incompatible shapes:Also passing the matching
--network deep3
made the problem disappear.This is somewhat unexpected, since I would assume that the architecture can be derived from the warmstart.