NN Results Discussion

[abrievanaardt]

I will attend to this section tomorrow:)

[annabosman]

Hi Abrie, will you be able to get to the paper this week? The deadline is creeping up on us!

[abrievanaardt]

Hi Anna. Sorry for keeping you with this. I have opened another branch, nnDiscussion, where I'm working on this. However, I need to run more experiments through the night so I will finish that section tomorrow when I get home. I think I have good news: I'm seeing neutrality around the origin using tanh (I think previous results were wrong since I had a small bug in the code. I fixed the bug before running all my other experiments but never repeated it for small domains). It goes to 0 as you move away and then it increases as you go further and further away. The latter is already visible from the graphs in the paper, but below is an idea of what happens very close to the origin.

I'm repeating the experiment with finer grained steps and doing the same for measure 2. Please let me know if there is something else that I should include (or experiment with).

[annabosman]

Hi Abrie,

I don't think we need more experiments for this paper - it is already overflowing with content. It's great that you found the bug before we sent it in! I will read the current version and see how it all fits together.

[annabosman]

For the sake of completeness, could you add the exact formulas of calculating M1 and M2? I.e., express the metrics mathematically. I think it will be useful for anyone who wants to use them in future.

[annabosman]

I have a question: In algorithm 3, what do you mean by "Normalise the sample range in walk to [0, 1];"? Do we normalise the resulting fitness values? As that doesn't seem right.

[annabosman]

To save page space, you could plot degree of neutrality for really small areas around the origin as well as for the larger ones on a single plot. For the graph to be readable, apply logarithmic scaling to x-axis. The title of the x-axis can then be log(x^max).

[annabosman]

I have added slightly more formal definitions of the metrics. Let me know if you need any input from my side!

[abrievanaardt]

Thanks Anna. I made some edits and added the updated graphs. If you want to follow the progress, I'm on the nnDiscussion branch. After merging and scaling the graphs like you proposed I realised that I used two different activation functions. Tanh and sigmoid. Tanh, being symmetrical, was used for the region around the origin (to confirm LeCun's statements). So I'm afraid it might be inappropriate to leave the graphs like that. Although I have to say that I got the same plots in larger domains with tanh (unfortunately don't have the data for them), but I don't know if there is a statistically significant difference. So I don't want to 'pretend' only having used tanh. So I'm not sure what to do? The graphs are on the nnDiscussion branch if you want to have a look. I think I'll put in a pull request and we can discuss it there.

About the normalisation. After our last discussion at Cafe27 we decided that instead of using a static epsilon, it should be scaled with the range of the fitness values in the sample. This is how I implemented it in the code. However, if I'm correct, having epsilon proportionate to the range would have the same effect as squashing the range between 0 and 1 and leaving epsilon constant across problems?

[abrievanaardt]

Hi Anna. So I merged the nnDiscussion with the base branch. I will update the graphs with the real tanh results as soon as I have them, probably very early tomorrow morning.

Please add, change, or remove what you see fit in the NN results section. I'm not very comfortable with the observations I'm making and I might be missing important discussion and/or conclusions (I might even be talking complete rubbish:D).

I also added a bit at the conclusion, please scrutinise all my changes since you last read the paper:)

[annabosman]

Have you used architectures with both activation functions? I think we should rather stick to one, otherwise observations become all muddled. Whatever data we have by tomorrow, we'll have to go with - Monday is the last day to submit. I haven't checked the paper yet, but we'll have to be specific in the description, too.

I will work on the paper tomorrow morning. If NN data is not too convincing, we can always leave it out of he paper altogether.

About the scaling: I see, it does make sense this way. Perhaps I'll rewrite the description to clarify the reason for scaling.

[annabosman]

Hi Abrie, The results look good! I would like to re-draw the NN graphs, though - is the latest data on github? I think the axis labels should be more interpretable. I see that the gnuplot scripts are there, I'll see what I can do.

[abrievanaardt]

Hi Anna, let me put the latest data there, I didn't get to that this morning.

[annabosman]

Alright! We should also fix that "16th" dimension on the benchmark functions graphs.

I have checked just now - CEC'17 submission website is still open. But I would like to submit asap.

[abrievanaardt]

I'm at work but I'm quickly logging into my pc at home so that I can get you the latest data. I can fix the 16th dimension as well (if you'd prefer) I thought I had done that. Let me know what exactly I should do and what you want to work on so that we don't both work on the same things:D

[annabosman]

Let me re-do the NN graphs and expand the discussion a little. You can fix the 16th dimension. That should be all :)

[abrievanaardt]

So you fix the scripts of the NN graphs but before you run them I'll update the data. And I'll fix the dimensions. Deal.

I think we should both work on the same branch, say directly on master? Updates would be quicker that way.

[annabosman]

Agreed. I don't think there's a point in branching out at this stage.

[abrievanaardt]

Yes! I added the new data. I don't have gnuplot installed here so please just verify that I didn't make a funny mistake.

I'll quickly do the 16th dimension in a separate commit.

[abrievanaardt]

It seems like the data for that graph is already correct, the script probably just has to be rerun and the graph replaced. Please could you check if running the gnuplot script for that graph fixes the problem?

[annabosman]

Will do.

[annabosman]

On M1, re-running the script still has 16 dimension in - apparently because it is given for some of the data? I think restricting the xrange to [2:15] should do the trick.

[abrievanaardt]

I managed to run it, it seems there is another issue. The data is correct (as far as I can tell now) but the plot still goes to 16 for some of the data items.

[abrievanaardt]

Haha yes. I think that would be the hot fix right now.

[annabosman]

M2 also prints 16 dimensions.

[annabosman]

Ok, adding xrange fixes it! I got it from here :)

[abrievanaardt]

Awesome. Just restrict the M2 graph to [2:15] as well!

Let me know if you need help, I'll be online.

[annabosman]

Alright, I am done with the paper, will push everything to github now. If you find time, please take a quick look. Otherwise, I'll just have to trust my own judgement and send it in. Thanks for all the hard work!

[abrievanaardt]

Anna, at the NN Results, I think there is a mixup between x^max and ln(x^max) in the discussion.

[abrievanaardt]

I see the paper has been submitted already, I hope we can still make changes?

I also see that I have contracted my second name while the others are written in full. My second name is Abraham, if it's at all possible to make edits.

[abrievanaardt]

Nevermind the x^max mixup, I only realise now that the X axis ticks were changed. I'm with you now:D

[annabosman]

Hi Abrie,

Yes, we can still edit - I have just decided to rather be safe than sorry. If the paper is accepted, we'll have yet another round to make changes.

With x^max, I did not really like the notation and changed it to x_max (kinda looks like a power otherwise). I thought I was consistent throughout, but I'll double-check.

I'll add your second name now.

[annabosman]

...And now we wait till 6 March :)

[kmalan]

Hi Anna & Abrie, Thanks for all the work. Paper is looking really good! I am working through the final version (in between a session on Quantum Projective Simulation .... don't ask). I will post comments on postgrad before 4pm. Katherine

[abrievanaardt]

Ooh I didn't even see the notation change, there's no problem with that as far as I can see, but I thought we both referred to different things in the discussion, Xmax vs scaled Xmax, but that wasn't the case.

I made a few grammar changes on master

[annabosman]

I'll wait for Katherine's feedback and re-submit one last time.

[annabosman]

Alright, Katherine's comments have been incorporated into the paper - I think I'll re-submit.