PS-Tree

[hengzhe-zhang]

Hello, I have developed a new method named PS-Tree. Based on the experimental results, I found this method to achieve state-of-the-art results in terms of testing accuracy. Thus, I hope it can be added to SRBench and be viewed as a new baseline method. By the way, although it is a genetic programming based method, the special assumption of such a model may result in it hard to exactly recover those symbolic models in common symbolic regression benchmarks. Therefore, this package does not support the ground truth experiment at the current moment, and I am still working on that.

[athril]

Hi Hengzhe, Thank you for your contribution! Great job on the method development! If the performance of your method can be reproduced, it will surely become a new baseline model for symbolic regression. I am afraid we won't be able to include it in the coming release, as all of the tests have been run and the paper has already been submitted. We can certainly update our living review. Do you have a specific timeline when you plan to add ground truth support for PS-Tree? Thanks, Patryk

[hengzhe-zhang]

@athril Hi Patryk, Thank you for your response. I submitted my code to this repository in the hopes that my results will be independently verified by other organizations, which will help with the review process of my paper. Concerning ground truth support, it appears that there is still a lack of documentation that could explain how I should support that function, particularly for the definition of symbolically equivalent. Thanks, Hengzhe

[hengzhe-zhang]

@athril Hello! I realized that there was an error in my installation script. I have resolved the issue and verified it on my local server. Thus, I believe it can work now.

[lacava]

hi @zhenlingcn , thanks for your PR!

Therefore, this package does not support the ground truth experiment at the current moment, and I am still working on that.

We are requiring all SR methods to return a model string that is compatible with sympy, even if the method isn't geared toward exact solutions.

As you noted, we haven't updated the contribution guidelines yet to reflect this requirement, as we are still working on finalizing our initial experiment and working through requirements to make submissions easy to add to the benchmark. Sorry about that. I'll notify you when those guidelines are added.

[lacava]

FYI contribution guide has been updated: https://github.com/EpistasisLab/srbench/blob/master/CONTRIBUTING.md

[hengzhe-zhang]

FYI contribution guide has been updated: https://github.com/EpistasisLab/srbench/blob/master/CONTRIBUTING.md

Thanks. However, I am still unsure of what is the correct way to provide a model string for symbolic verification after reading that document.

[hengzhe-zhang]

@lacava I have updated my model to support outputting a string of the final model. It should be noted that my model is an ensemble model that contains many small symbolic models. However, it is clear that this benchmark needs a single symbolic model for verification. Thus, I simply extract the first symbolic model and convert it to a string. In the complexity calculation method, I sum the size of all symbolic models to ensure a fair comparison. I'm not sure whether this is enough to meet the requirements of SRBench.

[lacava]

@lacava I have updated my model to support outputting a string of the final model. It should be noted that my model is an ensemble model that contains many small symbolic models. However, it is clear that this benchmark needs a single symbolic model for verification. Thus, I simply extract the first symbolic model and convert it to a string.

No, you will need to return a string of the ensemble. Hopefully it is straightforward to do so. Note that we are planning to move the complexity calculation to a post-processing step, which will depend on the returned model. So it's important that the string that is returned is both sympy compatible (#58) and exactly the model that is being measured for performance.

[hengzhe-zhang]

@lacava Thanks. Indeed, it's not hard to output a string of the ensemble. However, it will be zero accuracy when checking the symbolic model because there are no mixture models in this benchmark. In the real world scenario, ensemble learning is necessary in some cases because it is possible that the data is generated from a mixture model, such as GMM. Maybe an acceptable idea is to add my algorithm into this benchmark as a black-box algorithm and only run black-box experiments.

[athril]

Hi @zhenlingcn https://github.com/zhenlingcn, Your method is expected to return a single solution for a given dataset. In case of an ensemble, this will be a single string per dataset that combines all the models of the ensemble. Please remember SRBench benchmark was/is intended to compare symbolic regression algorithms specifically. We have also covered some other popular regression methods (XGBoost, LGBM, some others from scikit learn), but only for reference. I am not sure if non-symbolic regression methods should be covered. I see benefits of that, but on the other hand this seems not to be the original goal of the benchmark.

We are extremely cautious on adding black-box methods to the benchmark. I personally don't see a legit way of doing this. Given that the collection of regression datasets is public, I can imagine an "evil" method that looks for specific rows or sets of features in the data and returns the results of the best performing method among those included in the benchmark. This would surely put the method on top of the competitors, which would be totally inappropriate.

On Fri, Jan 7, 2022 at 10:37 PM Hengzhe Zhang @.***> wrote:

@lacava https://github.com/lacava Thanks. Indeed, it's not hard to output a string of the ensemble. However, it will be zero accuracy when checking the symbolic model because there are no mixture models in this benchmark. In the real world scenario, ensemble learning is necessary in some cases because it is possible that the data is generated from a mixture model, such as GMM. Maybe an acceptable idea is to add my algorithm into this benchmark as a black-box algorithm and only run black-box experiments.

— Reply to this email directly, view it on GitHub https://github.com/cavalab/srbench/pull/45#issuecomment-1007878168, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC26CM63J6R75IGDR7SY6SDUU6WQDANCNFSM47RKFL3Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[hengzhe-zhang]

@athril Thanks. I can understand that this is a benchmark targeting symbolic regression algorithms. However, it seems that all black-box problems are drawn from PMLB. Thus, I believe it is also a good venue for evaluating other kinds of machine learning algorithms. As for your concern, I agree that it is possible to use the meta-learning technique to overfit this benchmark. Maybe one possible solution is to only accept formally published algorithms.

As for my current issue, another possible solution is to allow my ensemble learning method to evolve a single model when running on symbol discovery tasks and evolving an ensemble on black-box tasks. This is possible in the real world scenario, but I'm not sure whether it is suitable for this benchmark.

[lacava]

@zhenlingcn let me know what you'd like to do. In the current setup the methods are not aware of whether they are training on synthetic or real-world data.

[hengzhe-zhang]

@lacava To be specific, I have used a decision tree in my algorithm. However, the decision tree is only suitable for real-world tasks. Thus, it would be better if the evaluation function could tell me about whether it is a formula discovery task or a real-world regression task. Then, I would be able to decide whether or not to use the decision tree in my algorithm.

By the way, I discovered that you have developed competition in GECCO 2022. Thanks for providing such a venue to evaluating symbolic regression algorithms.

[hengzhe-zhang]

@lacava By the way, I can fully understand that injecting more prior knowledge might be unfair for existing algorithms. Thus, I agree to change my algorithm to accommodate the current requirement of SRBench. The above proposal is for reference only.

[MohammadSoltani100]

Hi Hengzhe, Thank you for your contribution I have some questions: 1- how can I get a complete user guide (Tutorial) of PS-Tree package? I want to know model's hyperparameters. 2- has this model explicit expression output such as Gplearn package? how? Thanks

[hengzhe-zhang]

@MohammadSoltani100 Thanks for your interest. This default hyperparameter is robust to many cases. There still lacks a guideline for this package. I'm trying to improve the documentation and even algorithm performance, but the progress is slow.

In fact, if you are dealing with a machine learning task, I suggest you use my other package "Evolutionary Forest" (https://github.com/hengzhe-zhang/EvolutionaryForest). This package is more flexible and robust because it is built on the idea of ensemble learning.

PS-Tree can output an explicit expression, the simplest way is "print(x.model())", where x represents the trained model.

[MohammadSoltani100]

Thank you for your valuable response. best regards.

On Wed, Apr 6, 2022 at 6:30 PM Hengzhe Zhang @.***> wrote:

@MohammadSoltani100 https://github.com/MohammadSoltani100 Thanks for your interest. This default hyperparameter is robust to many cases. There still lacks a guideline for this package. I'm trying to improve the documentation and even algorithm performance, but the progress is slow.

In fact, if you are dealing with a machine learning task, I suggest you use my other package "Evolutionary Forest" ( https://github.com/hengzhe-zhang/EvolutionaryForest). This package is more flexible and robust because it is built on the idea of ensemble learning.

PS-Tree can output an explicit expression, the simplest way is "print(x.model())", where x represents the trained model.

— Reply to this email directly, view it on GitHub https://github.com/cavalab/srbench/pull/45#issuecomment-1090308299, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASPWCXFETHQXSZCBHKE6ZQLVDWKIXANCNFSM47RKFL3Q . You are receiving this because you were mentioned.Message ID: @.***>