Benchmark the main code and set a default config

[DesmondYuan]

For the initial benchmarking, use

• Network size m=20
• Adam with weight decay 1e-4
• STEER batching
• LHS sampling #3 implemented at aa1d808145008655834a152440088861e9873b46
• Moving avg early stopping #5 implemented at 81d3577382cc9e115d23050234ebc335b6fd54c9

Adding an inference loss in addition to the prediction loss #4 implemented at 09cc46afffc704b96a08ffa1a315517a37b9f499 0c00a66b74e5b6620aa2c9d61d2ef3bf65c09bc3

[DesmondYuan]

A few notes:

• c99a16d1092bdf0783906c703eaac79b1cd5a213 Train/Val/Test splits is also implemented as default.
• 1920db3f2c9f6b51e3ba3807f7d6e927e0618d82 Added a expr_name to name subfolders under figs and checkpoints

[DesmondYuan]

Seems like we have all the features we need #3 #4 #5. Shall we now do some quick test of the default config?

[jiweiqi]

I think we are good to go, to get some sense of

• typical code running time
• how many nodes we can handle well
• how many experimental datasets we need
• how good the learned w/alpha is

[DesmondYuan]

Trying with network size: 100, train/dev/tes=100/20/20

Conclusion1: use a smaller lr, e.g. 1f-3

test01: Using lr=5f-3 (default, 36s/epoch on my local machine, 1.2hr in total)

test02: Using lr=1f-3. (1.3hr in total)

test03: Using ntotal=1 (43 min)

issue: #6

[jiweiqi]

Are those results from endpoint data instead of dynamic data? We may start from dynamic data which should be much easier

[DesmondYuan]

Are those results from endpoint data instead of dynamic data? We may start from dynamic data which should be much easier

Dynamic for test01 and test02. ntotal=1 only for test03