search

Kevinouio / MSU-AI-Building-Classification

So this is the main submission for the AI project that classifies buildings around the MSU campus
Apache License 2.0
0 stars 0 forks source link

Issues while evaluating your submission #1

Open sushant097 opened 5 days ago

sushant097 commented 5 days ago

We got following issues while running your code:

python test.py 
2024-11-20 13:32:51.141934: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-11-20 13:32:51.142319: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-11-20 13:32:51.144143: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-11-20 13:32:51.149322: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1732131171.158225 3392193 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1732131171.160724 3392193 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-20 13:32:51.169653: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
W0000 00:00:1732131171.859781 3392193 gpu_device.cc:2344] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Traceback (most recent call last):
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/ops/operation.py", line 234, in from_config
    return cls(**config)
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/layers/normalization/batch_normalization.py", line 143, in __init__
    self.axis = int(axis)
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'list'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/saving/serialization_lib.py", line 718, in deserialize_keras_object
    instance = cls.from_config(inner_config)
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/ops/operation.py", line 236, in from_config
    raise TypeError(
TypeError: Error when deserializing class 'BatchNormalization' using config={'name': 'conv1_bn', 'trainable': True, 'dtype': 'float32', 'axis': [3], 'momentum': 0.99, 'epsilon': 1.001e-05, 'center': True, 'scale': True, 'beta_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'gamma_initializer': {'module': 'keras.initializers', 'class_name': 'Ones', 'config': {}, 'registered_name': None}, 'moving_mean_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'moving_variance_initializer': {'module': 'keras.initializers', 'class_name': 'Ones', 'config': {}, 'registered_name': None}, 'beta_regularizer': None, 'gamma_regularizer': None, 'beta_constraint': None, 'gamma_constraint': None}.

Exception encountered: int() argument must be a string, a bytes-like object or a real number, not 'list'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/test.py", line 48, in <module>
    model = tf.keras.models.load_model(weights_path)
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/saving/saving_api.py", line 196, in load_model
    return legacy_h5_format.load_model_from_hdf5(
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/legacy/saving/legacy_h5_format.py", line 133, in load_model_from_hdf5
    model = saving_utils.model_from_config(
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/legacy/saving/saving_utils.py", line 85, in model_from_config
    return serialization.deserialize_keras_object(
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/legacy/saving/serialization.py", line 495, in deserialize_keras_object
    deserialized_obj = cls.from_config(
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/models/sequential.py", line 350, in from_config
    layer = saving_utils.model_from_config(
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/legacy/saving/saving_utils.py", line 85, in model_from_config
    return serialization.deserialize_keras_object(
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/legacy/saving/serialization.py", line 495, in deserialize_keras_object
    deserialized_obj = cls.from_config(
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/models/model.py", line 526, in from_config
    return functional_from_config(
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/models/functional.py", line 499, in functional_from_config
    process_layer(layer_data)
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/models/functional.py", line 483, in process_layer
    layer = serialization_lib.deserialize_keras_object(
  File "/home/sg/AI-Club/Evaluation-Challenge/Evaluated-Submission/GavinJiangFanClub-MSU-AI-Building-Classification/venv/lib/python3.10/site-packages/keras/src/saving/serialization_lib.py", line 720, in deserialize_keras_object
    raise TypeError(
TypeError: <class 'keras.src.layers.normalization.batch_normalization.BatchNormalization'> could not be deserialized properly. Please ensure that components that are Python object instances (layers, models, etc.) returned by `get_config()` are explicitly deserialized in the model's `from_config()` method.

config={'module': 'keras.layers', 'class_name': 'BatchNormalization', 'config': {'name': 'conv1_bn', 'trainable': True, 'dtype': 'float32', 'axis': [3], 'momentum': 0.99, 'epsilon': 1.001e-05, 'center': True, 'scale': True, 'beta_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'gamma_initializer': {'module': 'keras.initializers', 'class_name': 'Ones', 'config': {}, 'registered_name': None}, 'moving_mean_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'moving_variance_initializer': {'module': 'keras.initializers', 'class_name': 'Ones', 'config': {}, 'registered_name': None}, 'beta_regularizer': None, 'gamma_regularizer': None, 'beta_constraint': None, 'gamma_constraint': None}, 'registered_name': None, 'build_config': {'input_shape': [None, 256, 256, 64]}, 'name': 'conv1_bn', 'inbound_nodes': [[['conv1_conv', 0, 0, {}]]]}.

Exception encountered: Error when deserializing class 'BatchNormalization' using config={'name': 'conv1_bn', 'trainable': True, 'dtype': 'float32', 'axis': [3], 'momentum': 0.99, 'epsilon': 1.001e-05, 'center': True, 'scale': True, 'beta_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'gamma_initializer': {'module': 'keras.initializers', 'class_name': 'Ones', 'config': {}, 'registered_name': None}, 'moving_mean_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'moving_variance_initializer': {'module': 'keras.initializers', 'class_name': 'Ones', 'config': {}, 'registered_name': None}, 'beta_regularizer': None, 'gamma_regularizer': None, 'beta_constraint': None, 'gamma_constraint': None}.

Exception encountered: int() argument must be a string, a bytes-like object or a real number, not 'list'

The code we used:

import os
import sys

import matplotlib.pyplot as plt
import seaborn as sns
import tensorflow as tf
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)
from tensorflow.keras.preprocessing.image import ImageDataGenerator

class Logger:
    """Log print statements to a file and the console."""

    def __init__(self, log_file):
        self.console = sys.stdout
        self.file = open(log_file, "w")

    def write(self, message):
        self.console.write(message)
        self.file.write(message)

    def flush(self):
        self.console.flush()
        self.file.flush()

# Define paths
test_dir = "../private-dataset"  # Test dataset directory
weights_path = "model_epoch_17.h5"  # Path to model weights
output_dir = "evaluation_results"  # Directory for saving results

# Create the output directory if it doesn't exist
os.makedirs(output_dir, exist_ok=True)

# Set up logging
log_file = "analysis.txt"
sys.stdout = Logger(log_file)

# Load the trained model
if not os.path.exists(weights_path):
    raise FileNotFoundError(f"Model weights file not found at: {weights_path}")
model = tf.keras.models.load_model(weights_path)

print(f"Model loaded from {weights_path}")

# Define image dimensions and batch size
img_height, img_width = 224, 224  # Adjust based on your model input
batch_size = 16

# Load the test dataset
if not os.path.exists(test_dir):
    raise FileNotFoundError(f"Test dataset directory not found at: {test_dir}")
test_datagen = ImageDataGenerator(rescale=1.0 / 255)
test_generator = test_datagen.flow_from_directory(
    test_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False,  # Important for confusion matrix
)

# Make predictions on the test set
predictions = model.predict(test_generator)
predicted_classes = tf.argmax(predictions, axis=1).numpy()

# Get true classes
true_classes = test_generator.classes
class_labels = list(test_generator.class_indices.keys())

# Calculate metrics
accuracy = accuracy_score(true_classes, predicted_classes)
precision = precision_score(
    true_classes, predicted_classes, average="weighted", zero_division=0
)
recall = recall_score(
    true_classes, predicted_classes, average="weighted", zero_division=0
)
f1 = f1_score(true_classes, predicted_classes, average="weighted", zero_division=0)

# Print metrics
print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1-Score: {f1:.4f}")

# Save metrics to a file
metrics_file = os.path.join(output_dir, "metrics.txt")
with open(metrics_file, "w") as f:
    f.write(f"Accuracy: {accuracy:.4f}\n")
    f.write(f"Precision: {precision:.4f}\n")
    f.write(f"Recall: {recall:.4f}\n")
    f.write(f"F1-Score: {f1:.4f}\n")
print(f"Metrics saved to {metrics_file}")

# Generate confusion matrix
cm = confusion_matrix(true_classes, predicted_classes)

# Plot confusion matrix
plt.figure(figsize=(10, 8))
sns.heatmap(
    cm,
    annot=True,
    fmt="d",
    cmap="Blues",
    xticklabels=class_labels,
    yticklabels=class_labels,
)
plt.xlabel("Predicted")
plt.ylabel("True")
plt.title("Confusion Matrix")

# Save confusion matrix to file
confusion_matrix_file = os.path.join(output_dir, "confusion_matrix.png")
plt.savefig(confusion_matrix_file, dpi=300, bbox_inches="tight")
plt.close()
print(f"Confusion matrix saved to {confusion_matrix_file}")

requirements.txt:

tensorflow
scikit-learn
seaborn
matplotlib
The complete project structure looks like this:

├── evaluation_results/          # Output folder for evaluation results
│   ├── analysis.txt             # Logs and detailed evaluation output
│   
├── model_epoch_17.h5  # model weights 
├── venv/                        # Python virtual environment
├── ../private_dataset/          # Private dataset with building images
│   ├── Butler Hall/             # Subfolder for Butler Hall images
│   ├── Carpenter Hall/          # Subfolder for Carpenter Hall images
│   ├── Lee Hall/                # Subfolder for Lee Hall images
│   ├── McCain Hall/             # Subfolder for McCain Hall images
│   ├── McCool Hall/             # Subfolder for McCool Hall images
│   ├── Old Main/                # Subfolder for Old Main images
│   ├── Simrall Hall/            # Subfolder for Simrall Hall images
│   ├── Student Union/           # Subfolder for Student Union images
│   ├── Swalm Hall/              # Subfolder for Swalm Hall images
│   ├── Walker Hall/             # Subfolder for Walker Hall images
├── test.py # evaluation script

We will not accept your submission if you don't create an evaluation script in Python and mention requirements.txt in your Repo.

Which we already did for you. Review the bug in your code and try to resolve it ASAP.

Kevinouio commented 5 days ago

Hey, I looked through our code and we had already done both for the evaluation metrics as we printed out the confusion matrix. Also, I'm not sure if you have read the repo carefully but we stated to make sure to run the file within Colab if running it locally does not work which from the error result that you had gotten, it is very likely that you ran the evaluation script on PyCharm. Also, we did also create our evaluation script, Test_Model.ipynb, so there was no need to make your evaluation script in Python. So I don't know if there was a misreading error but we did have our evaluation script called Test_Model.ipynb, stated in the repo. I will change the evaluation script to output the folder with the results instead of just a confusion matrix. That is all to be changed within the repo as we already have our evaluation script in Python called Test_model.ipynb.


├── evaluation_results/          # Output folder for evaluation results
│   ├── analysis.txt             # Logs and detailed evaluation output
│   
├── model_epoch_17.h5  # model weights 
├── ../dataset/          # Private dataset with building images
│   ├── Butler Hall/             # Subfolder for Butler Hall images
│   ├── Carpenter Hall/          # Subfolder for Carpenter Hall images
│   ├── Lee Hall/                # Subfolder for Lee Hall images
│   ├── McCain Hall/             # Subfolder for McCain Hall images
│   ├── McCool Hall/             # Subfolder for McCool Hall images
│   ├── Old Main/                # Subfolder for Old Main images
│   ├── Simrall Hall/            # Subfolder for Simrall Hall images
│   ├── Student Union/           # Subfolder for Student Union images
│   ├── Swalm Hall/              # Subfolder for Swalm Hall images
│   ├── Walker Hall/             # Subfolder for Walker Hall images
├── Test_Model.ipynb # evaluation script```