search

snap-stanford / SATURN

MIT License
108 stars 17 forks source link

Tunning Saturn #25

Closed Hrovatin closed 1 year ago

Hrovatin commented 1 year ago

I wanted to optimize Saturn integration, so I tried to tune pe_sim_penalty, which seemed one of the rare parameters affecting loss and thus potentially also integration strength. However, in a wide range (0.01, 0.1, 1, 10) this didn't affect batch or bio integration metrics in any clear way. Is this expected?

Yanay1 commented 1 year ago

We observed a small, 1-2% increase in accuracy on the logistic regression transfer accuracy task when setting the value to 1 vs 0 for the frog and zebrafish dataset.

How many seeds are you testing? (eg, for our results we ran 30 trials).

Hrovatin commented 1 year ago

I tested only 3 seeds and the variation between seeds visually seems to be the main driver of differences, not the weight.

Thank you for letting me know that you observed similar - I was concerned something was wrong with my code after observing so little differences.

Yanay1 commented 1 year ago

No problem! Please let us know if you have any other questions!