zeinebchelly
commented
7 years ago I think it makes more sense to use the 10-cross validation instead; specifically if we think in terms of complexity. Concerning the metric, maybe we can focus on FPR? What do you think? And I totally agree with adding a section showing the performance of the developed method in comparison to other state-of-the-art methods.
2016-11-23 14:24 GMT+01:00 Pablo García Sánchez notifications@github.com:
After discussing with @unintendedbear https://github.com/unintendedbear and looking to the current results we have thought a new experimental section that could improve the paper a lot and ease future steps.
First, we think that the fitness we are using is complicated to understand and requires a lot of computational time. Besides we do not find any reference to justify its use. So, we are going to the go back to the classical 10-cross validation, as it has been used widely: Divide the dataset in A and B (10 times) and use A for the training (whole evolution) and then B at the end of the evolution to validate (not DURING the evolution as we are doing now). So, we need to decide the fitness using some of the classical error metrics (see this wikipedia link to decide which error to use as fitness https://en.wikipedia.org/wiki/Positive_and_negative_predictive_values ), or the coverage (as we are doing now), or the fitness used by [1] (positives, negatives and depth).
Also, it would be a great improvement to the paper to compare with other method in the literature. Guys in [1] deal with each individual as a rule, and the model is the whole population. We could do some experiments with that structure and decide which method is better.
So, the steps would be:
- Chose the most appropriate metric for fitness and validation
- Execute with classic 10-cross validation and the chosen error metric as fitness and validation. We will have 10 final validation metrics, and therefore the average of our method.
- Execute with every individual as a rule, instead as a tree (I will modify the code for this), and compare with the previous method.
What @zeinebchelly https://github.com/zeinebchelly and @JJ https://github.com/JJ think?
[1] Ivan De Falco, Antonio Della Cioppa, Ernesto Tarantino: Discovering interesting classification rules with genetic programming. Appl. Soft Comput. 1(4): 257-269 (2002)
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/geneura-papers/2016-PPSN/issues/21, or mute the thread https://github.com/notifications/unsubscribe-auth/AJgODv-hEi4vR--qXr_EUirgxMcfSaXBks5rBD6RgaJpZM4K6koM .
After discussing with @unintendedbear and looking to the current results we have thought a new experimental section that could improve the paper a lot and ease future steps.
First, we think that the fitness we are using is complicated to understand and requires a lot of computational time. Besides we do not find any reference to justify its use. So, we are going to the go back to the classical 10-cross validation, as it has been used widely: Divide the dataset in A and B (10 times) and use A for the training (whole evolution) and then B at the end of the evolution to validate (not DURING the evolution as we are doing now). So, we need to decide the fitness using some of the classical error metrics (see this wikipedia link to decide which error to use as fitness ), or the coverage (as we are doing now), or the fitness used by [1] (positives, negatives and depth).
Also, it would be a great improvement to the paper to compare with other method in the literature. Guys in [1] deal with each individual as a rule, and the model is the whole population. We could do some experiments with that structure and decide which method is better.
So, the steps would be:
1) Chose the most appropriate metric for fitness and validation 2) Execute with classic 10-cross validation and the chosen error metric as fitness and validation. We will have 10 final validation metrics, and therefore the average of our method. 3) Execute with every individual as a rule, instead as a tree (I will modify the code for this), and compare with the previous method.
What @zeinebchelly and @JJ think?
[1] Ivan De Falco, Antonio Della Cioppa, Ernesto Tarantino: Discovering interesting classification rules with genetic programming. Appl. Soft Comput. 1(4): 257-269 (2002)