tnc-ca-geo / animl-ml

Machine Learning resources for camera trap data processing
Other
4 stars 1 forks source link

reexporting and repackaging FLP_Foxer_Model1.0-aug.82-0.3369-0.8745 causes mira deployment to error without logs #84

Closed rbavery closed 1 year ago

rbavery commented 2 years ago

this is the culprit cell I think. the keras version used to export the model in the past might differ from the keras version in the python 3.6 kernel we are using now, but I'm not sure.

from tensorflow.python.saved_model import builder
from tensorflow.python.saved_model.signature_def_utils import predict_signature_def
from tensorflow.python.saved_model import tag_constants

# Note: This directory structure will need to be followed - see notes for the next section
model_version = '1'
export_dir = 'export/Servo/' + model_version

# Build the Protocol Buffer SavedModel at 'export_dir'
try:
    builder = builder.SavedModelBuilder(export_dir)
    # Create prediction signature to be used by TensorFlow Serving Predict API
    print('loaded_model.input: {}'.format(loaded_model.input))
    print('loaded_model.output: {}'.format(loaded_model.output))
    signature = predict_signature_def(
        inputs={"inputs": loaded_model.input}, outputs={"score": loaded_model.output})

    from keras import backend as K

    with K.get_session() as sess:
        # Save the meta graph and varia?bles
        builder.add_meta_graph_and_variables(
            sess=sess, tags=[tag_constants.SERVING], signature_def_map={"serving_default": signature})
        builder.save()
except Exception as e: 
    print(e)

its hard to tell for sure since there are no other logs at https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Faws$252Fsagemaker$252FEndpoints$252Fmira-large-keras-endpoint-config-2022-07-06-21-39-38/log-events

to handle this I'm using the original model package that was saved back in 2021 that is still hosted on the notebook instance.

nathanielrindlaub commented 1 year ago

closing out as we're moving towards MIRAv2 which is PyTorch based.