Open zsz00 opened 1 year ago
MNIST is a high-dimensional dataset, where pure symbolic regression is going to do quite poorly due to the combinatoric scaling. What you can try instead is something like described in https://arxiv.org/abs/2006.11287 (see interactive example of this at the end of https://colab.research.google.com/github/MilesCranmer/PySR/blob/master/examples/pysr_demo.ipynb).
Basically, write down a neural network like $$classification = MLP1(\sum{i} MLP_2(\text{patch}_i))$$,
where $\text{patch}$ is a patch of pixels (maybe give it 9 pixels?). Once you train this, then try to fit SR to $MLP_2$ and $MLP_1$ independently. Finally, arrange them in the same functional form.
For example, maybe you'll get something like: $$MLP_2 \approx (\text{pixel}_1 - \text{pixel}_2\ \ ,\ \ \text{pixel}_3 \times \text{pixel}_4)$$ and $$MLP_1 \approx y_1 \times y_2^2$$
Thus, your final equation would be: $$classification = \text{sigmoid}((\sum_{i} \text{pixel}_1 - \text{pixel}_2) \times (\sum_i \text{pixel}_3 \times \text{pixel}_4)^2 )$$
where the sum is over small patches of $3\times 3$ pixels.
Regarding applying SymbolicRegression to high-dimensional data sets in general, I imagine the recommendation would be to start with a feature-selection approach, and once a small number of highly-relevant features is selected apply SymbolicRegression?
I tried to use sr for mnist classification training, but the results were not good. I hope you can help me see where I need to improve MNIST is 28*28 images, 10 classes label.