egraphs-good / extraction-gym

benchmarking e-graph extraction
MIT License
24 stars 15 forks source link

Improve the results from the faster-greedy-dag extractor #34

Closed TrevorHansen closed 6 months ago

TrevorHansen commented 6 months ago

Compared to the current/old faster-greedy-dag extractor, this is a bit slower but produces better results. I think it's mostly luck though, it just visits nodes in a way that for the benchmarks we have produces better results. I don't think there's any theoretical reason it's better.

`` ################################################### faster-greedy-dag-new vs faster-greedy-dag-old

extractors: ['faster-greedy-dag-new', 'faster-greedy-dag-old'] data/babble/physics_scientific_unsolved_4h_ellisk_2019-07-20T18.09.34--bench000_it0.json differs in dag cost: 89 91 data/babble/list_list_hard_test_ellisk_2019-02-15T11.26.41--bench007_it10.json differs in dag cost: 419 418 data/babble/list_list_hard_test_ellisk_2019-02-15T11.43.28--bench007_it7.json differs in dag cost: 298 299 data/babble/list_list_hard_test_ellisk_2019-02-15T11.43.28--bench009_it13.json differs in dag cost: 395 392 data/babble/list_list_hard_test_ellisk_2019-02-15T11.43.28--bench008_it8.json differs in dag cost: 330 329 data/babble/list_list_hard_test_ellisk_2019-02-15T11.43.28--bench011_it19.json differs in dag cost: 364 363 data/babble/list_list_hard_test_ellisk_2019-02-15T11.31.39--bench010_it16.json differs in dag cost: 347 346 data/babble/physics_scientific_unsolved_4h_ellisk_2019-07-20T18.09.34--bench001_it1.json differs in dag cost: 144 143 data/babble/list_list_hard_test_ellisk_2019-02-15T11.31.39--bench009_it11.json differs in dag cost: 367 366 data/babble/list_list_hard_test_ellisk_2019-02-15T11.31.39--bench006_it6.json differs in dag cost: 246 247 data/babble/list_list_hard_test_ellisk_2019-02-15T11.43.28--bench010_it17.json differs in dag cost: 381 380 data/babble/physics_scientific_unsolved_4h_ellisk_2019-07-20T18.05.46--bench004_it4.json differs in dag cost: 183 182 data/babble/physics_scientific_unsolved_4h_ellisk_2019-07-20T18.09.34--bench006_it7.json differs in dag cost: 153 152 data/babble/list_list_hard_test_ellisk_2019-02-15T11.35.48--bench010_it12.json differs in dag cost: 371 369 data/babble/physics_scientific_unsolved_4h_ellisk_2019-07-20T18.16.45--bench001_it1.json differs in dag cost: 126 125 data/babble/list_list_hard_test_ellisk_2019-02-15T11.26.41--bench010_it15.json differs in dag cost: 427 426 data/babble/list_list_hard_test_ellisk_2019-02-15T11.35.48--bench005_it5.json differs in dag cost: 245 244 data/babble/list_list_hard_test_ellisk_2019-02-15T11.43.28--bench005_it5.json differs in dag cost: 231 232 data/babble/list_list_hard_test_ellisk_2019-02-15T11.39.19--bench004_it5.json differs in dag cost: 235 234 data/babble/physics_scientific_unsolved_4h_ellisk_2019-07-20T18.09.34--bench004_it4.json differs in dag cost: 122 120 data/babble/physics_scientific_unsolved_4h_ellisk_2019-07-20T18.09.34--bench005_it5.json differs in dag cost: 141 140 data/flexc/67addae9-f7db-4ebb-a02e-e1e045f87fc4.json differs in dag cost: 37 36 data/flexc/4a29e9e7-a232-4608-b7d2-37796d0ecd97.json differs in dag cost: 37 36 data/flexc/03d369d0-bb23-477a-8ea1-a452efc850fe.json differs in dag cost: 37 36 data/flexc/db344dbd-fb20-4e9e-badb-076d7a3809bc.json differs in dag cost: 37 36 data/flexc/86d12271-435d-463d-a7b4-e671423f0ced.json differs in dag cost: 37 36 data/tensat/nasneta.json differs in dag cost: 16.374893040569077 16.322293040515433 data/tensat/nasneta_acyclic.json differs in dag cost: 16.35845509585488 16.301627096203447 data/egg/math_simplify_factor.json differs in dag cost: 6 5 data/fuzz/14.json differs in dag cost: 114.32003810419093 124.07667387412016 data/fuzz/15.json differs in dag cost: 134.39646038194644 160.99453847121393 data/fuzz/1.json differs in dag cost: 108.96027900550405 143.4529619974897 data/fuzz/6.json differs in dag cost: 97.14818527661234 137.76199410064035 data/fuzz/5.json differs in dag cost: 46.30896318297725 54.138384368771 data/fuzz/12.json differs in dag cost: 115.532456438086 178.68563937989165 data/rover/box_filter_3iteration_egraph.json differs in dag cost: 1701 1819 cumulative time for faster-greedy-dag-new: 3585ms cumulative time for faster-greedy-dag-old: 3180ms cumulative tree cost for faster-greedy-dag-new: 32017750419805 cumulative tree cost for faster-greedy-dag-old: 32017750420420 cumulative dag cost for faster-greedy-dag-new: 79973 cumulative dag cost for faster-greedy-dag-old: 80251 Cumulative time for faster-greedy-dag-new: 3585ms Cumulative time for faster-greedy-dag-old: 3180ms faster-greedy-dag-new / faster-greedy-dag-old geo mean tree: 0.9907 dag: 0.9960 micros: 1.2354 quantiles tree: 0.3535, 1.0000, 1.0000, 1.0000, 1.1419 dag: 0.6466, 1.0000, 1.0000, 1.0000, 1.2000 micros: 0.4228, 1.1455, 1.1981, 1.3196, 3.3375

oflatt commented 6 months ago

This is an interesting experiment- not sure if we should merge it since we don't have a reason why it's better Maybe you could merge it as faster-greedy-dag-alt? Or just keep this a draft.

mwillsey commented 6 months ago

Yeah I think we close this for now, unless something changes and makes it better. Fewer more distinct extractors seems like the way to go for now. Happy to re-open if other feel differently.