[Question] How do I use the NASbench problem?

[HideakiImamura]

Now, I try to use the NASbench problem with kurobako. I downloaded the dataset from https://storage.googleapis.com/nasbench/nasbench_full.tfrecord. I executed the following commands to generate result.json, but got the following error. How do I use the NASbench problem with kurobako?

(venv) hideakibookpuro:kurobako-native mamu$ kurobako solver random | tee solver.json
{"random":{}}
(venv) hideakibookpuro:kurobako-native mamu$ kurobako problem nasbench nasbench_only108.tfrecord | tee problem.json{"nasbench":{"dataset":"nasbench_only108.tfrecord","encoding":"A"}}
(venv) hideakibookpuro:kurobako-native mamu$ kurobako studies --solvers $(cat solver.json) --problems $(cat problem.json) | kurobako run > result.json
(ALL) [00:00:00] [STUDIES     10/10 100%] [ETA  0s] canceled

Error: Other (cause; assertion failed: `left == right`; assertion failed: `(left == right)` (left: `None`, right: `Some("nasbench-rs")`))
HISTORY:
  [0] at /Users/mamu/.cargo/registry/src/github.com-1ecc6299db9ec823/nasbench-0.1.2/src/dataset.rs:54
  [1] at /Users/mamu/.cargo/registry/src/github.com-1ecc6299db9ec823/nasbench-0.1.2/src/dataset.rs:30
  [2] at /Users/mamu/.cargo/registry/src/github.com-1ecc6299db9ec823/kurobako_problems-0.1.7/src/nasbench.rs:56
  [3] at /Users/mamu/.cargo/registry/src/github.com-1ecc6299db9ec823/kurobako-0.1.9/src/problem.rs:77
  [4] at /Users/mamu/.cargo/registry/src/github.com-1ecc6299db9ec823/kurobako-0.1.9/src/problem.rs:33
  [5] at /Users/mamu/.cargo/registry/src/github.com-1ecc6299db9ec823/kurobako-0.1.9/src/runner.rs:214
  [6] at /Users/mamu/.cargo/registry/src/github.com-1ecc6299db9ec823/kurobako-0.1.9/src/runner.rs:131
  [7] at /Users/mamu/.cargo/registry/src/github.com-1ecc6299db9ec823/kurobako-0.1.9/src/runner.rs:141
  [8] at /Users/mamu/.cargo/registry/src/github.com-1ecc6299db9ec823/kurobako-0.1.9/src/main.rs:82

[sile]

Sorry for the lack of documentation. You need to convert nasbench_full.tfrecord to a binary format (that can be loaded by kurobako) using https://github.com/sile/nasbench-rs as follows:

// Convert to binary format.
$ wget https://storage.googleapis.com/nasbench/nasbench_full.tfrecord
$ nasbench convert nasbench_full.tfrecord nasbench_full.bin
$ ls -lh
-rw-rw-rw- 1 foo foo 328M May 12 16:47 nasbench_full.bin
-rw-rw-rw- 1 foo foo 2.0G May 12 16:45 nasbench_full.tfrecord

// Specify the binary file when generating a problem.
$ kurobako problem nasbench nasbench_full.bin

I'll add documentation relating to this topic soon.

[HideakiImamura]

Thank you for your reply. I can convert to binary format as you suggested! Thank you!

[sile]

FYI

Since kurobako-v0.1.10, it provides a built-in command to convert the TFRecord file. You can execute the conversion by executing $ kurobako dataset nasbench convert $FROM $TO. And, I added a wiki page about the NASBench dataset: https://github.com/sile/kurobako/wiki/NASBench