Merge upstream v2018.09 release

[takikawa]

Merge upstream release as part of the lwAFTR release process.

[takikawa]

@dpino: Tagged you as a reviewer on this because I had to modify some code from your PR #1159 to make new tests that were merged into lib.yang.data pass. Would appreciate a quick look. :)

[takikawa]

Whoops, hadn't realized @dpino is on holiday at the moment. @xray7224 do you think you could take a look at the merge-related fixes instead?

[takikawa]

CI is still failing due to flakiness with the snabbnfv test ERROR testlog/program.snabbnfv.selftest.sh. I think this problem occurs mainly with the docker image used to run CI but not on our CI servers outside of docker.

I'm going to debug it and fix it, but in a separate PR since it's not tied to any of the changes in this release.

I think otherwise this PR should be ready to merge. Commit 9703dd0 fixes an issue that came up with code introduced for this lwAFTR release (though not related to the upstream merge itself).

[tsyesika]

I've been testing this branch and your script on snabb1 produces:

[jtallon@snabb1:~/snabb/src/program/lwaftr/tests/release-benchmarks]$ sudo SNABB_LWAFTR_CPU0=1 SNABB_LWAFTR_CPU1=10 SNABB_LOADTEST_CPU0=11 SNABB_LOADTEST_CPU1=10 SNABB_PCI0=02:00.0 SNABB_PCI1=82:00.0 SNABB_PCI2=02:00.1 SNABB_PCI3=82:00.1 SNABB_PCI4=03:00.0 SNABB_PCI5=83:00.0 SNABB_PCI6=03:00.1 SNABB_PCI7=83:00.1 /home/jtallon/snabb/src/program/lwaftr/tests/release-benchmarks/release-benchmarks.sh 2>/dev/null 
>> Compiling configurations (may take a while)
>> Running loadtest: 1 instance, 2 NIC interface (log: /tmp/tmp.vFQqA5JHro/tmp.PSCJHEokD1)
>> Success: 7.425
>> Running loadtest: 1 instance, 2 NIC interfaces (from config) (log: /tmp/tmp.vFQqA5JHro/tmp.KKoGtGRs7P)
>> Success: 9.998
>> Running loadtest: 1 instance, 1 NIC (on a stick) (log: /tmp/tmp.vFQqA5JHro/tmp.RsZNcXiYFy)
>> Success: 9.96
>> Running loadtest: 1 instance, 1 NIC (on-a-stick, from config file) (log: /tmp/tmp.vFQqA5JHro/tmp.BskOs2N3T8)
>> Success: 9.999
>> Running loadtest: 2 instances, 2 NICs (from config) (log: /tmp/tmp.vFQqA5JHro/tmp.zmQtFnJg1c)
>> Running loadtest 2: 2 instances, 2 NICs (from config) (log: /tmp/tmp.vFQqA5JHro/tmp.xlkhsoSFOA)
>> Success: 9.999
>> Success: 5.615
>> Running loadtest: 2 instances, 1 NIC (on a stick, from config) (log: /tmp/tmp.vFQqA5JHro/tmp.DJ0GM2IHmW)
>> Running loadtest 2: 2 instances, 1 NIC (on a stick, from config) (log: /tmp/tmp.vFQqA5JHro/tmp.1PABQj2BKa)
>> Success: 9.999
>> Success: 9.999
>> Running loadtest: 1 instance, 1 NIC, 2 queues (log: /tmp/tmp.vFQqA5JHro/tmp.5ydXNXZpCv)
>> Success: 9.999

I think this is an accurate reflection of the performance we're getting. It took me a while to figure out what all the environment variables should be. The lwaftr branch is currently performing at:

[jtallon@snabb1:~/snabb/src/program/lwaftr/tests/release-benchmarks]$ sudo SNABB_LWAFTR_CPU0=1 SNABB_LWAFTR_CPU1=10 SNABB_LOADTEST_CPU0=11 SNABB_LOADTEST_CPU1=10 SNABB_PCI0=02:00.0 SNABB_PCI1=82:00.0 SNABB_PCI2=02:00.1 SNABB_PCI3=82:00.1 SNABB_PCI4=03:00.0 SNABB_PCI5=83:00.0 SNABB_PCI6=03:00.1 SNABB_PCI7=83:00.1 /home/jtallon/snabb/src/program/lwaftr/tests/release-benchmarks/release-benchmarks.sh 2>/dev/null 
>> Compiling configurations (may take a while)
>> Running loadtest: 1 instance, 2 NIC interface (log: /tmp/tmp.0QNEtXNmaD/tmp.NWGSCaKuEr)
>> Success: 7.849
>> Running loadtest: 1 instance, 2 NIC interfaces (from config) (log: /tmp/tmp.0QNEtXNmaD/tmp.M1Wr3zKt9S)
>> Success: 9.999
>> Running loadtest: 1 instance, 1 NIC (on a stick) (log: /tmp/tmp.0QNEtXNmaD/tmp.mEtHmfk8DM)
>> Success: 9.999
>> Running loadtest: 1 instance, 1 NIC (on-a-stick, from config file) (log: /tmp/tmp.0QNEtXNmaD/tmp.uojHvZFfsY)
>> Success: 9.999
>> Running loadtest: 2 instances, 2 NICs (from config) (log: /tmp/tmp.0QNEtXNmaD/tmp.5qx2jW6qBx)
>> Running loadtest 2: 2 instances, 2 NICs (from config) (log: /tmp/tmp.0QNEtXNmaD/tmp.QGstDoRuWX)
>> Success: 1.114
>> Success: 5.525
>> Running loadtest: 2 instances, 1 NIC (on a stick, from config) (log: /tmp/tmp.0QNEtXNmaD/tmp.GLbm50ZfdB)
>> Running loadtest 2: 2 instances, 1 NIC (on a stick, from config) (log: /tmp/tmp.0QNEtXNmaD/tmp.ut4LHyXxJi)
>> Success: 9.999
>> Success: 9.999
>> Running loadtest: 1 instance, 1 NIC, 2 queues (log: /tmp/tmp.0QNEtXNmaD/tmp.w5cV5ddnrC)
>> Success: 9.999

So it seems to perform about as well as it did prior to this update (i.e. no performance regressions).