search

AthenaIak / AI-model-comparison

MIT License
0 stars 0 forks source link

Define a proper architecture for the model #25

Open AthenaIak opened 4 years ago

AthenaIak commented 4 years ago

We got started in this project without a very clear idea of the architecture. After developing a few features and a couple demos, the requirements are better understood.

  1. This project is about creating a library that can be used to compare two or more models.
  2. The library should work for any type of model (convolutional etc).
  3. Should work for models with the same structure.
  4. Should be expandable to compare models that have different structure. (While this is not in the scope of the first step, it should be easy to expand this).
  5. Should be able to compare the (trained) weights of the models.
  6. Should be expandable to compare datapoints, i.e. the output of neurons on a pre-defined set of input samples. (While this is not in the scope of the first step, it should be easy to expand this).
AthenaIak commented 4 years ago

Up until now, I mixed 2 things in the implementation: the experimentation/demos part, and the actual comparison part. After getting a feel of the requirements and challenges, I believe it's time to split these two. This repo should only contain the library that anyone can use to compare networks. Another repo can be setup for our experiments and demonstrations of usage.

A graph that outlines some considerations for the architecture:


 |-------------|     |-------| /--- creation
 | Experiments |-----| Model | ---- definition
 |-------------|     |-------| \--- training
       |                        \-- output/saving
       |
   |-------| /--- create/load model
   | Demos | ---- compare (using the library)
   |-------| \--- view comparison (using the library)
              \-- export report (using the library)

                keras.model
                    |              |---------|
                    V          /---| Weights |
 |---------|     |-------|    /    |---------|
 | Library |-----| Model | ---
 |---------|     |-------|    \    |------------|     |---------|
      |                        \---| Datapoints | --- | Samples |
      |                            |------------|     |---------|
 |------------| /--- view
 | Comparison | ---- report/output
 |------------| \--- return
      |
      |
 |------------|
 | Comparison | /--- tailor with parameters
 |  Metric    |
 |------------|

We expect models to be fed to the library ready to use (compiled/trained).

I've thought about the main object (used to compare/measure the distance between models), and it looks like it would be a good idea to create two variations: a) one for the metrics that measure distance between sets of weights, and b) one for those that measure distance between neuron outputs (by feeding the models with the sample data).

  1. list of items to be compared (weights for a / models for b) OR two attributes (assuming we only want to compare pairs of models) 1a. reference item 1b. target item

  2. comparisons (we should be able to add these like we add layers in a network, but the comparator+parameters should be unique)

Note1: If we go with the second option of comparing pairs of models, we can provide methods to cleanly clone the object and update just one or both of the models (e.g. clone(this,new_reference, new_target))