Regression

[pabhermoso]

Hello! Can we use this model for a global regression task? If so, what changes would need to be mande?

Thanks a lot in advance!

[liang3588]

I think the masked auto-encoder-based pre-training strategy is general, you can extend it to the global regression task. Can you explain what kind of global regression task?

[pabhermoso]

I have a set of Geometries (think brains for example) and would like to regress the age of the patient. I have something starting to work, but I noticed the embedding was changing drastically for the same geometry across epochs. I set the drop_path=0.0 and that helps. I changed the loss to MSE Loss too.

[liang3588]

I think maybe you can first verify whether the Transformer network is effective for your task. If it is effective, you can further extend the pre-training strategy.

[pabhermoso]

Agreed. Thanks!

[pabhermoso]

Interestingly I get a situation in which all the examples have an identical embedding after the max pooling. The head then learns the result for the training set but the model overfits terribly. Do you have any idea why this might be? Thanks a lot!