Full batch sensitivity analysis (adjacency reconstruction, feature reconstruction, and both)

[wehlutyk]

Test the following parameters:

• node features: exact network communities, noisy network communities, noisy network communities with redundancy, random
• dim_ξ: [2, ..., 10]
• dim_l1: [dim_ξ, ..., 16]
• n_ξ_samples: [1, 5, 10]
• bias in all layers: True/False

For each run, save:

• the original and final-predicted adjacency and/or features
• the evolution of the training losses
• ξ distribution plots before and after training (for dim_ξ ≥ 2, use a dimension reduction technique such as t-SNE)

[jaklevab]

Finished parameter exploration for adjacency reconstruction without feature reconstruction: See gcn-ae-explore.ipynb in branch issue-6-sensitivity-analysis

[wehlutyk]

18 adds the following parameters of interest: mini-batch size, length of random walks in the mini-batch, and both those quantities w.r.t. the network size and average community sizes. (I might take over for that if you don't to explore them @jaklevab, as I keep adding parameters to this :p )

[jaklevab]

Ok so the minibatch sensitivity analysis gives the same result as the full batch when taking the full batch as minibatch size. Remains to be seen how the reconstruction loss behaves with the different parameters affecting the RW and the minibatch. @wehlutyk are you done with the minibatch parametrization?

[wehlutyk]

Ok so the minibatch sensitivity analysis gives the same result as the full batch when taking the full batch as minibatch size.

Great!

Remains to be seen how the reconstruction loss behaves with the different parameters affecting the RW and the minibatch. @wehlutyk are you done with the minibatch parametrization?

Well, I started, then realised that what I wanted to test would have taken 6 months to run, and decided I wasn't sure which were the right parameter ranges to choose. So I moved on to real data sets (#8) in order to know when it would be necessary (memory-wise) to use a mini-batch size that's not the full batch, which then led me to #21 because it's currently so slow on a ~10,000 nodes network.

So once I'm done with #21 (today or next week), doing #8 should show us the relevant parameter ranges we need to test for the mini-batch and RW (and should give more material for NetSci).

[wehlutyk]

(I started the minibatch parametrisation in #19)

[wehlutyk]

Ok, so we have a bit of a mess with the validation of minibatching and the sensitivity analyses. I'm reorganising:

• this issue is for full-batch sensitivity analysis, and progress is tracked in @jaklevab 's PR #12 (renamed both this issue and the PR accordingly)
• minibatch validation (i.e., checking it works as expected when we don't have convolutions) and sensitivity analysis (i.e. seeing how it behaves when we change the RW and seed sizes) were mixed together in #19. I'm closing that in favour of #28 and #29.