search

automl / TabPFN

Official implementation of the TabPFN paper (https://arxiv.org/abs/2207.01848) and the tabpfn package.
http://priorlabs.ai
Apache License 2.0
1.22k stars 109 forks source link

Several for-loops in evaluation don't seem to do anything #36

Open amueller opened 1 year ago

amueller commented 1 year ago

There's some for-loops in the evaluation that only work if they run exactly once, which seems pretty confusing. One is in https://github.com/automl/TabPFN/blob/main/tabpfn/RunFullDatasetAnalyses.ipynb, in "eval_methods". If there's more than one did, it only returns the results of the last iteration. The code always calls it with only one did, so that's fine, but then maybe remove the for-loop?

The other one is here: https://github.com/automl/TabPFN/blob/main/tabpfn/scripts/baseline_prediction_interface.py

That one has a TQDM attached, but it returns in the first iteration. Again, I think it's only called with lists of length one, but maybe then removing the loop would be better?

Btw, I rewrote eval_methods to take datasets, not "dids" since the dids rely on global state about how things are numbered and add some extra indirection. If you want a PR for the refactor, let me know.

SamuelGabriel commented 1 year ago

I know this is all very sub-optimally written for people working on the code now, as this is very much still the code in which we tried out a lot of inference tricks just before the deadline. Sorry for this.

Internally we moved on to a new implementation of this that is a bit cleaner. I think we will share that with the next publication.

amueller commented 1 year ago

That's fair, though from what Frank said, that's maybe still a bit out? By that time I might have to publish my own code for a submission, so then we'll have three different implementations. That doesn't seem ideal.