Submission: giving the trained model parameters (save and restore functions for a custom model)

[MaximeLee]

In the Notebook 6 'How to create a submission in codabench', it is said that we can provide the weights of the trained model in order to only to the evaluation. I want to do use this feature for a custom model (in Tensorflow).

• What is the script in the save() function?
• Should the custom model class implement the restore() function too?

[Mleyliabadi]

Hi,

We are working to provide an implementation example in Tensorflow for AirfRANS use case soon.

However, there exists an example of a model implementation for an another use case (independent from this challenge) and only for your inspiration. You can find an implementation of a Fully Connected architecture here. And its base class that manage the save and restore utilities in addition to training and evaluation of an architecture can be found here.

We inform you once the Tensorflow implementation for AirfRANS use case is ready.

Don't forget to join the discord channel for further information and updates concerning the competition.

[MaximeLee]

Ty for this clear answer.

I saw in the source code that the keras.Model.save_weights/keras.Model.load_weights function is used to save/load the weights of a model. Would it be better to use keras.Model.save/keras.Model.load functions to save/load custom models with a more complex architecture?

[mathscope29]

Hi,

Is there any news for this question? I have an independent model (in Tensorflow), and I would like to pass my pretrained in a folder trained_model. Could it be possible if this folder is the result of my following command:

model.save('trained_model', save_format='tf')

[Mleyliabadi]

Hi,

For general information, there is a new Notebook in starting kit, that shows how to train, evaluate, save and load an existing model based on Tensorflow library.

Now, you are completely free to use any save or load commands for a custom model which is implemented independently from LIPS. However, your predictions should be in a compatible shape with evaluation module of LIPS framework as it was introduced in third section of Notebook 4 and as it is provided below.

from lips.evaluation.airfrans_evaluation import AirfRANSEvaluation

# a predict function which outputs the predictions in a compatible shape and format readable by `AirfRANSEvaluation` class
predictions, observations = predict(model, benchmark._test_dataset)

# create an instance of AirfRANSEvaluation class giving right configs and paths

evaluator = AirfRANSEvaluation(config_path = BENCH_CONFIG_PATH,
                               scenario = BENCHMARK_NAME,
                               data_path = DIRECTORY_NAME,
                               log_path = LOG_PATH)

# get some extra data required for evaluation
observation_metadata = benchmark._test_dataset.extra_data

# evaluate your solution using `evaluate` function of `AirfRANSEvaluation` class
metrics = evaluator.evaluate(observations=observations,
                             predictions=predictions,
                             observation_metadata=observation_metadata)
print(metrics)

[mathscope29]

Hi,

Thank you for your reply :))

However, I still do not understand how to make the predictions in a compatible shape with evaluation module of LIPS framework in my pre-trained model and submit it to Codabench.

In Notebook 7, section III, the organizers gives an example file my_augmented_simulator.py in Pytorch, which implements the models showed in Notebook 4. However, in this my_augmented_simulator.py, there is no code to change predictions into a compatible shape with evaluation module of LIPS.

And in the new Notebook, as I understand, the example is only shown for the model which is instantiated with the TfFullyConnectedAirfoil class. (sorry if I misunderstood something @@)

Edit: Just a lillte precision, my goal here is to submit the pre-trained model and use Codabench evaluate the model.

[Mleyliabadi]

If you look at the third section of Notebook 4, you can find the predict function as below. Once the predictions are performed using the trained model, the dataset.reconstruct_output(predictions) function of dataset class allows to reconstruct the predictions to obtain the compatible shape required by LIPS framework.

def predict(model, dataset, device, dtype=torch.float32, pin_memory=True, num_workers=0):
    # set the model for the evaluation
    model.eval()
    predictions = []
    observations = []
    test_loader = process_dataset(dataset, training=False, dtype=dtype, pin_memory=pin_memory, num_workers=num_workers)
    # we dont require the computation of the gradient
    with torch.no_grad():
        for batch in test_loader:
            data, target = batch
            data = data.to(device)
            target = target.to(device)
            prediction = model(data)

            if device == torch.device("cpu"):
                predictions.append(prediction.numpy())
                observations.append(target.numpy())
            else:
                predictions.append(prediction.cpu().data.numpy())
                observations.append(target.cpu().data.numpy())
    # reconstruct the prediction in the proper required shape of target variables
    predictions = np.concatenate(predictions)
    predictions = dataset.reconstruct_output(predictions)
    # Do the same for the real observations
    observations = np.concatenate(observations)
    observations = dataset.reconstruct_output(observations)

    return predictions, observations

The reconstructed output is a dictionary including the target variables names ('x-velocity', 'y-velocity', 'pressure', 'turbulent_viscosity') as keys and the variable values (predictions) as dictionary values.

Here is an example :

predictions, observations = predict(model, benchmark._test_dataset, device="cpu")
print(predictions)

{'x-velocity': array([35.579407, 35.579407, 36.092613, ..., 70.30871 , 70.17227 ,
        69.79646 ], dtype=float32),
 'y-velocity': array([-0.19910145, -0.19910145, -0.2586422 , ..., -0.9752197 ,
        -0.7723541 , -1.1196766 ], dtype=float32),
 'pressure': array([ 63.878662,  63.878662,  45.01947 , ..., 107.7341  , 138.68344 ,
        182.95615 ], dtype=float32),
 'turbulent_viscosity': array([-0.01853248, -0.01853248, -0.01875828, ..., -0.01889038,
        -0.01877284, -0.0193758 ], dtype=float32)}

Once you have the observations and predictions in this shape, you can the evaluate function, as I mentioned in my previous comment to get the metrics and scores.

It is exactly the same for augmented simulators based on Tensorflow.

I confirm that in Notebook 7, there is a missing reconstruct_output function. But, if you have your pre-trained model, you should simply transform the predictions, and use the evaluate only flag. We are working on this notebook. Tomorrow, there will be a webinar on this subject (How to submit to codabench) with different examples. For sure this webinar will be recorded, in the case that you are not available.

[mathscope29]

Hi,

Thank you so much for your reply!!

I still have a question (I'm sorry). As I understand, for the submission on Codabench, we have to implement the prediction function as you mentioned above in the class of my_augmented_simulator.py, right? Look forward to see your new version of Notebook 7 for an example.

For the webinar, I am sorry that I will not available to participate. Could you please provide the recorded videos of these webinars (the third and fourth webinars) soon? Thank you again for these amazing notebooks and webinars!