Reproduce Figure 8 in the NASBench paper

[wenwei202]

Hi, I am trying to duplicate the curves in Figure 8. I have some questions:

1. what is the mutation rate p? How could I map it to the mutation probability of edges and node operations? I check the code here, only one edge or one node is mutated at each time, is it mapped to the case that p=1? How could I get p=0.5 and p=2?
2. I assume the tournament size t is the number of samples from population, which are compared to get the best model as a parent. Is it correct?
3. Were the curves obtained by searching validation accuracy or test accuracy? I tried by searching validation accuracy and plotting test accuracy, I observed small up-and-down fluctuations along the test curves even I averaged over 100 runs. This is normal and expected, because a higher validation accuracy can map to a lower test accuracy. How did you avoid it in Figure 8? My results of median of 100 runs:

My results of mean of 100 runs: (The results don't match the results in Figure 8)

4. Is the x-axis median of 100 runs or means of them or something others? Is the x-axis in a linear scale?

Thank you so much.