Closed s0rin closed 2 months ago
osma
commented
3 months ago Do you have an existing data directory with models trained using a previous Annif version?
We recently upgraded many of our dependencies - see PR #771 . For example TensorFlow was upgraded. Sadly, this means that old models cannot be loaded anymore and projects need to be retrained.
Thanks for reporting this issue. I think this kind of problem should be handled better instead of throwing an ugly exception.
s0rin
commented
3 months ago Indeed, the annif docker container could run again after removing vocabs/ and projects/ from the annif-projects/data/ directory.
juhoinkinen
commented
3 months ago The TensorFlow/Keras model file has some metadata, which contains keras_version:
head -n1 data-fintoai/projects/yso-fi/nn-model.keras
metadata.json{"keras_version": "2.13.1", "date_saved": "2023-08-22@14:41:52"}PK!��33
config.json{"module": "keras.src.engine.functional", "class_name": "Functional", "config": {"name": "model", "trainable": true, "layers": [{"module": "keras.layers", "class_name": "InputLayer", "config": {"batch_input_shape": [null, 38586, 3], "dtype": "float32", "sparse": false, "ragged": false, "name": "input_1"}, "registered_name": null, "name": "input_1", "inbound_nodes": []}, {"module": "keras.layers", "class_name": "Flatten", "config": {"name": "flatten", "trainable": true, "dtype": "float32", "data_format": "channels_last"}, "registered_name": null, "build_config": {"input_shape": [null, 38586, 3]}, "name": "flatten", "inbound_nodes": [[["input_1", 0, 0, {}]]]}, {"module": "keras.layers", "class_name": "Dropout", "config": {"name": "dropout", "trainable": true, "dtype": "float32", "rate": 0.2, "noise_shape": null, "seed": null}, "registered_name": null, "build_config": {"input_shape": [null, 115758]}, "name": "dropout", "inbound_nodes": [[["flatten", 0, 0, {}]]]}, {"module": "keras.layers", "class_name": "Dense", "config": {"name": "dense", "trainable": true, "dtype": "float32", "units": 100, "activation": "relu", "use_bias": true, "kernel_initializer": {"module": "keras.initializers", "class_name": "GlorotUniform", "config": {"seed": null}, "registered_name": null}, "bias_initializer": {"module": "keras.initializers", "class_name": "Zeros", "config": {}, "registered_name": null}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "registered_name": null, "build_config": {"input_shape": [null, 115758]}, "name": "dense", "inbound_nodes": [[["dropout", 0, 0, {}]]]}, {"module": "keras.layers", "class_name": "Dropout", "config": {"name": "dropout_1", "trainable": true, "dtype": "float32", "rate": 0.2, "noise_shape": null, "seed": null}, "registered_name": null, "build_config": {"input_shape": [null, 100]}, "name": "dropout_1", "inbound_nodes": [[["dense", 0, 0, {}]]]}, {"module": "annif.backend.nn_ensemble", "class_name": "MeanLayer", "config": {"name": "mean_layer", "trainable": true, "dtype": "float32"}, "registered_name": "MeanLayer", "build_config": {"input_shape": [null, 38586, 3]}, "name": "mean_layer", "inbound_nodes": [[["input_1", 0, 0, {}]]]}, {"module": "keras.layers", "class_name": "Dense", "config": {"name": "dense_1", "trainable": true, "dtype": "float32", "units": 38586, "activation": "linear", "use_bias": true, "kernel_initializer": {"module": "keras.initializers", "class_name": "Zeros", "config": {}, "registered_name": null}, "bias_initializer": {"module": "keras.initializers", "class_name": "Zeros", "config": {}, "registered_name": null}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "registered_name": null, "build_config": {"input_shape": [null, 100]}, "name": "dense_1", "inbound_nodes": [[["dropout_1", 0, 0, {}]]]}, {"module": "keras.layers", "class_name": "Add", "config": {"name": "add", "trainable": true, "dtype": "float32"}, "registered_name": null, "build_config": {"input_shape": [[null, 38586], [null, 38586]]}, "name": "add", "inbound_nodes": [[["mean_layer", 0, 0, {}], ["dense_1", 0, 0, {}]]]}], "input_layers": [["input_1", 0, 0]], "output_layers": [["add", 0, 0]]}, "registered_name": "Functional", "build_config": {"input_shape": [null, 38586, 3]}, "compile_config": {"optimizer": "adam", "loss": "binary_crossentropy", "metrics": ["top_k_categorical_a�=W��cy"], "loss_weights": null, "weighted_metrics": null, "run_eagerly": null, "steps_per_execution": null, "jit_compile": null}}PK:uWThe file contains also the model weights in binary(?) , but somehow the metadata and the version number could surely be read, but I did not find any way to read it with TensorFlow.
It would be nice to show the version of TF that created the model on crashes like this.
juhoinkinen
commented
3 months ago Heh, googling did not give anything usable, CurreChat gave a directly working solution after few tries:
My apologies for the confusion. It looks like the information provided is from an actual .keras file which is a zip-like file format used by newer versions of Keras/TensorFlow for saving models.
from zipfile import ZipFile import json # Replace this with the path to your .keras file model_file_path = 'path_to_your_model.keras' # Opening the .keras file as a zip file with ZipFile(model_file_path, 'r') as zip: # Reading the metadata.json file inside the .keras zip with zip.open('metadata.json') as metadata_file: # Decoding the bytes to a string metadata_str = metadata_file.read().decode('utf-8') # Converting the string to a JSON (dictionary) object metadata = json.loads(metadata_str) # Now you can use or print the metadata print(metadata)
Using annif_app as in https://github.com/NatLibFi/Annif/blob/main/docker-compose.yml, the annif docker container was running a for while until pulling recently the image quay.io/natlibfi/annif:latest, since then it fails with the following error: