search

automl / NASLib

NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.
Apache License 2.0
528 stars 117 forks source link

Val_accuracy is negative for "room_layout " in "zc_transbench101_micro.json" and "zc_transbench101_macro.json" #188

Closed SliverySky closed 1 month ago

SliverySky commented 2 months ago

I download "zc_transbench101_micro.json" from the given link (https://drive.google.com/file/d/1SBOVAyhLCBTAJiU_fo7hLRknNrGNqFk7/view). Then, I found the val_accuracy is negative for the room_layout dataset. Similar situation happens in the room_layout dataset for" zc_transbench101_macro.json".

Neonkraft commented 2 months ago

Hi @SliverySky ,

Thank you for your interest in our work!

The room-layout task of Trans-NAS-Bench-101 is to estimate a 3d bounding box, and as such, it does not have accuracy as a metric. This task uses MSE Loss as its metric. The zc_transbench101_{micro/macro}.json benchmarks used the keyword "val_accuracy" for uniformity when consuming the API (as the metric to maximize).

I hope this helps!

Screenshot from 2024-09-16 14-38-54

SliverySky commented 1 month ago

Thank you very much for your answer!