Data analysis from the bert iso preliminary experiments

[mmatena]

Note: A lot of this will be applicable to other experiments, but I'll focus on the BERT iso preliminary experiments for now.

Potential view of the data

• Merging vs original.
  • Show best weighting (and maybe 50-50 weighting)
• Impact of iso regularization on original fine-tune performance.
• Impact of iso regularization strength on relative merging performance.

  When I also have BERT large results

• Comparison of relative merging performance for equivalent iso regularization strength and tasks.

[mmatena]

Also putting idea for a few (relatively quick and simple) additional experiments here.

Additional experiments

• Try merging by simple average between checkpoints.
• Try some fine-tuning from the zero-shot merged checkpoint.
• Compare effect of the number of examples used to compute the Fisher.
• See the effect of merging from earlier checkpoints.

[mmatena]

Known errors

• I accidentally regularized the distance from the initialization for the heads as well as the body for the base sized experiments. I changed it to only regularize the body before running the large experiments.
  • Since merging only acts on the body, this might not a big issue when looking at relative performance of merging. It would probably have a bigger impact on absolute performance of the original and merged models.
  • Also the non-regularized model is unaffected by this bug.