mattochal
commented
3 years ago Hi,
Thank you for your interest in our work.
Variance It is true that random imbalance can produce higher levels of task variance. Dealing with this can be tricky. In the paper, we find that random oversampling (ROS/ROS+) helps reduce variance during training (see performance over epoch graphs in the appendix).
Query Set Imbalance We did not introduce query set imbalance in the main experiments to reduce the number of random variables and reduce the number of experiments. However, we will be updating the arXiv paper very soon. In the updated version, we do examine the effect of imbalance at the query set level during meta-training (these will be added to the appendix of the updated paper). In summary, having query imbalance has a slightly negative effect on accuracy on imbalanced tasks (mostly -1% to -2%, but occasionally -4% and -7%). Query set imbalance could be particularly disruptive for transductive FSL methods, however, we leave this investigation for future work.
Let me know if you have any further questions.
Dear author
Thank you for such interesting and solid work on imbalanced fsl. I found that experiments with such imbalanced data always had large variance. I wonder how do you deal with it. Also, I am wondering the reason you did not mention training query set imbalance too much.
Thank you