search

google-research-datasets / tpu_graphs

Apache License 2.0
120 stars 43 forks source link

How to get the score on the test dataset for the tiles task? #10

Closed baziotis closed 9 months ago

baziotis commented 9 months ago

Running the following (note the --test_mode argument):

python tiles_train.py ... --test_mode=metrics

gives NaNs for the test dataset (in the .jsonz file)

baziotis commented 9 months ago

If it helps, this line:

runtimes = graph.node_sets['config']['runtimes']

in top_error_performance() which is called here, gives a tensor of zeros all the time, and this seems to be why we get the NaNs.

samihaija commented 9 months ago

This is expected. We decided to remove the labels from the test files (everything has runtime of zero). You may either:

  1. Submit your CSV to Kaggle, to get your test scores: kaggle.com/competitions/predict-ai-model-runtime
  2. Use the validation set as-if it was the testing set. This way, the scores (= runtimes) are present and your evaluation metrics will not be NaN