soap117 / DeepRule

BSD 3-Clause "New" or "Revised" License
128 stars 44 forks source link

Recreate numbers in table 2 #9

Open crazysal opened 3 years ago

crazysal commented 3 years ago

Hi, is there an evaluation script to re-create numbers of table 2 from paper.

The test_pipe_type_cloud.py only gives json as output. Where are the eval metrics ?

crazysal commented 3 years ago

This would be really helpful. Used pre-trained models to benchmark on https://chartinfo.github.io/
the performance score is 0.34 (task 6b) for line charts specifically where their and your metrics described in the paper match exactly.

xuewenyuan commented 1 year ago

@crazysal how about the score of task 6a

crazysal commented 1 year ago

out of box is 0.73