search

liangrj2014 / ISPD24_contest

20 stars 0 forks source link

Cost function -- just to be sure... #29

Open profmadden opened 9 months ago

profmadden commented 9 months ago

I'd like to confirm that I've got the scoring metrics right. We're using evaluator_v2.cpp. The posted beta submission results for some of the benchmarks are these:

design WL cost via cost overflow cost raw score runtime /s median runtime /s scaled score
Ariane133_51 9335109 3060400 10369862 22765372 4 11 22100882
Ariane133_68 9443754 2981836 7825647 20251238 8 12 20014313
BlackParrot 58347098 19740536 35450029 113537664 27 70.5 110393434
Nvdla 21345766 4630872 23933347 49909986 4 20 47592238
MemPool-Tile 8407884 3425828 3569040 15402753 3 10 14867672
MemPool-Group 262817992 76146200 70345580 409309772 81 375 391210938

The reference example outputs have these results:

design WL cost via cost overflow cost total cost
Ariane133_51 9296578 3358316 18676 12673570
Ariane133_68 9447807 3301284 22630 12771722
BlackParrot(AKA BSG_chip?) 58096126 20876700 41432310799597248 41432310878570080
Nvdla 21278324 5049112 184071082179836 184071108507272
MemPool-Tile 8410248 3805264 36906810 49122323
MemPool-Group 262678902 88644880 453736853003 454088176786

If I'm understanding things correctly -- the beta submissions for the Ariane benchmarks have high overflow (compared to the reference examples), resulting in overflow cost dominating the total cost. The example outputs for the Ariane benchmarks have better scores. For the remaining benchmarks, the beta submissions have lower overflow, and better total cost than the reference examples. Wire lengths and via costs are similar in both. For the final evaluation, the run times of the various tools will be used to scale the scores.

Does this all look correct?

liangrj2014 commented 9 months ago

Hi,

I'd like to double check that whether you are using the latest version of .cap files? We've increased the overflow weights defined in the .cap files. With the latest .cap file, I would expect the reference example outputs have higher overflow costs.

Also, I would suggest to use evaluator.cpp, which is the latest version, though in most cases evaluator_v2.cpp and evaluator.cpp should give similar scores.

profmadden commented 8 months ago

Ahh -- I didn't realize that the cap files had changed as well. Here's what I'm getting for the reference designs, using what I think is the latest evaluator. Let me know if these look horribly wrong!

design WL cost via cost overflow cost total cost
Ariane133_51 9296578 3358316 10826548 23481442
Ariane133_68 9447807 3301284 17345476 30094568
BlackParrot(AKA BSG_chip?) 58096126 20876700 47862659041789100032 47862659041868070912
Nvdla 21278324 5049112 212639152264839296 212639152291166752
MemPool-Tile 8410248 3805264 42624780325 42636995837
MemPool-Group 262678902 88644880 524157262855231 524157614179013
liangrj2014 commented 8 months ago

Hi,

It looks correct to me. Thanks!