Problem: assign weights to classes - solve via Genetic Algorithm/PSO?

[hypnopump]

If you solve this you'll be mentioned as a special contributor (it shouldn't be very hard but right now I don't have the time to do this if I want to keep developing the model).

A big problem is defining the ideal weights for the different classes. Since it can be framed as an optimization problem (find the ideal weights so the predictions are the best possible given a model structure, the different classes and some data), I think we could use a Genetic Algorithm/PSO/some derivative-free optimization algorithm to get the ideal weights. Basic formulation: Given:

• a model structure (depth, inputs, ...)
• The number of classes and the limits between them
• some data

Do:

1. Get a pretrained model and train it with the new class weights for a short time (1-2 epochs?)
2. Make predictions and evaluate them according to some metric (MSE wrt ground truth once padding is removed?)
3. If the metric is better than the previous model: replace old model by new model.
4. Else: Perform some mutation/perturbation on class weights
5. GO TO number 1 and repeat
6. STOP after k iterations without improvement / run the algorithm for k iterations (your choice)

Tips:

• The distance prediction notebook eats a lot of RAM (~4gb). My advice is to save the variables such as best model, best performance, model configuration, classes... on text files and load them on every iteration since each time a model is compiled with different weights for classes it consumes new RAM.
  • My preferred implementation would be:
    1. Create a genetic algorithm as a script on a separate file and execute the script containing the model and the evaluation from it as a terminal command.
    2. Store the results/metrics of a given model and its configuration on a text file. And acces them from the genetic algorithm script.

The genetic algorithm/PSO/other solution should be modular/adaptable to future changes in the model's part (that's why my preferred implementation would be as different scripts).

[hypnopump]

Done! Check the pipeline caller: https://github.com/EricAlcaide/MiniFold/blob/master/models/distance_pipeline/pipeline_caller.py