Open mcallaghan-sandvine opened 5 years ago
I suggest to run this pcap file throw our ASTF offline tool. The offline tool will simulate MSS and RTT using TCP stack and Ipg,RTT will be as you should expect from TCP. We will send instructions how to run this. There is more ways to do the same.
Thanks, Hanoh
On Thu, 4 Oct 2018 at 23:36 Matt Callaghan notifications@github.com wrote:
As per deep investigations into #143 https://github.com/cisco-system-traffic-generator/trex-core/issues/143 and #146 https://github.com/cisco-system-traffic-generator/trex-core/issues/146, we were deeply assessing frame.time_delta values for a long netflix flow.
Consider
- due to #143 https://github.com/cisco-system-traffic-generator/trex-core/issues/143, we are forced into the following configuration:
long_flow.yaml
- duration : 9999 generator : distribution : "seq" clients_start : "4.0.0.1" clients_end : "4.0.0.1" servers_start : "5.0.0.1" servers_end : "5.0.20.255" cap_ipg : false <-- do NOT use pcap, it is not sanitized cap_info :
- name: LAB-5928_longflow/netflix_480p-wide_avc_Thor_noGRO_B_45secs_21k_pkts.pcap w : 1 cps : 0.001 <-- for testing purposes, only send 1 flow and halt ipg : 10 <-- shoot for a "small as possible" inter-packet-delay for non-TCP-control-pkts rtt : 10000 <-- real-world 10ms RTT is fine
Invoked with:
sudo ./t-rex-64 -f long_flow.yaml -c 1 -m 1 -d 9999
Version : v2.43 DPDK version : DPDK 17.11.0 User : hhaim Date : Jul 11 2018 , 09:40:05 Uuid : 3cb87d62-84d5-11e8-8eba-0006f62b3e88 Git SHA : e74fc281e57dcd60cac05c2fc43df65967a63671
- the real world netflix capture can be baselined/viewed in #146 https://github.com/cisco-system-traffic-generator/trex-core/issues/146 (it is 45secs long, RTT is ~10ms, but HTTPS data streaming payloads have very low time deltas (frame.time_delta) because Netflix uses HTTP PIPELINING https://en.wikipedia.org/wiki/HTTP_pipelining
$ tshark -Q -z io,stat,500 -r netflix_480p-wide_avc_Thor_noGRO_B_45secs_21k_pkts.pcap
==================================== IO Statistics Duration: 44.0 secs Interval: 44.0 secs Col 1: Frames and bytes ---------------------------------- 1 Interval Frames Bytes ---------------------------------- 0.0 <> 44.0 20911 20936860 ====================================
- the resulting TRex output here is MUCH longer (as noted in #146 https://github.com/cisco-system-traffic-generator/trex-core/issues/146), but incorrectly blamed #143 https://github.com/cisco-system-traffic-generator/trex-core/issues/143 -- upon further review, we suspect that TRex us treating the data ACKs and subsequent client data payload pkts with RTT delay, but in fact in reality, there should be only using ipg delay
146 (comment)
https://github.com/cisco-system-traffic-generator/trex-core/issues/146#issuecomment-425560148
$ tshark -Q -z io,stat,500 -r trex_ipg_10us_netflix_long_flow.pcap
====================================== IO Statistics Duration: 143.8 secs Interval: 143.8 secs Col 1: Frames and bytes ------------------------------------ 1 Interval Frames Bytes ------------------------------------ 0.0 <> 143.8 20911 20936860 ======================================
Capture was taken client-side, without GRO, (shouldn't be a problem/concern)
As one may expect, the incoming flux of packets for a Netflix flow are quite frequent (<10us inter-arrival), but some are slightly delayed (a few hundred microseconds)
Here is an example of these streaming pipelined HTTP packets (from raw source pcap) [image: netflix_raw_pcap_frame_3000] https://user-images.githubusercontent.com/34753045/46500996-34ffea00-c7f2-11e8-82c3-6e17d876f09e.png
Compare with the output from TRex [image: trex_netflix_output_frame_3000] https://user-images.githubusercontent.com/34753045/46501105-87410b00-c7f2-11e8-9869-321524bf7ee8.png
- the time deltas here are significantly higher than the source, 10ms :( - it clearly interprets these as TCP-control-flow-pkts, and imposes the RTT delay on those sequences
- the "pktB" sequences (i.e. frame 3003) have an inter-packet-delay of 0us (pulling from the ipg=10us value, but not accurate due to #143 https://github.com/cisco-system-traffic-generator/trex-core/issues/143 )
We recognize that this is probably a costly thing for TRex to process... somehow in pre-processing, it needs to recognize that this flow sequence is HTTP PIPELINING, and that it should apply ipg delay to these pkt sequences and NOT rtt delays
Perhaps this handling could be added including a flag to enable/disable HTTP pipeline inspection or something to avoid default performance hit?
Thoughts welcome!
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cisco-system-traffic-generator/trex-core/issues/156, or mute the thread https://github.com/notifications/unsubscribe-auth/AMbjvUbDVtabIHA19c6ntszA92Y_6V1Iks5uhnE-gaJpZM4XI3bQ .
-- Hanoh Sent from my iPhone
using v2.46 version you can run this on netflix pcap and it will be replay with rtt of 5msec network ./astf-sim -f [netflix.pcap] --rtt 5 -o netflix_5msec.pcap
the output netflix_5msec.pcap will be with the right IPG/RTT
I would tend to agree that ASTF would likely not have this issue as it would track packet-by-packet and hopefully know to interpret HTTP pipelining as inter-packet-delay timings rather than RTT.
However, we would need this to be accommodated in STF. (For now we're going to have to proceed w/out this, and sometime in the future we'll come back to it and hopefully address/fix in the code base.)
Not sure though if pcap re-capture (processing) by ASTF would fix it though... (needs testing as Hanoh suggests)
I think we are not on the same page. The idea is to run the offline tool on the pcap for running them in STF mode. for ASTF there is no need for offline tool. The tool will solve the problems you are describing here.
On Wed, Oct 10, 2018 at 10:04 PM Matt Callaghan notifications@github.com wrote:
I would tend to agree that ASTF would likely not have this issue as it would track packet-by-packet and hopefully know to interpret HTTP pipelining as inter-packet-delay timings rather than RTT.
However, we would need this to be accommodated in STF. (For now we're going to have to proceed w/out this, and sometime in the future we'll come back to it and hopefully address/fix in the code base.)
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cisco-system-traffic-generator/trex-core/issues/156#issuecomment-428693549, or mute the thread https://github.com/notifications/unsubscribe-auth/AMbjvXTITObU2VF1krZ82p4WSdxf_VODks5ujkTCgaJpZM4XI3bQ .
-- Hanoh Sent from my iPhone
Hm. Well currently our use case and solution is to use STF. (we chose this mode to accomplish our goals) - and this is one of the limitations/issues uncovered w/ that mode.
Switching modes to ASTF is not an option for us at this time.
I’m not suggesting to move to ASTF
Hanoh
On Thu, 11 Oct 2018 at 16:31 Matt Callaghan notifications@github.com wrote:
Hm. Well currently our use case and solution is to use STF. (we chose this mode to accomplish our goals) - and this is one of the limitations/issues uncovered w/ that mode.
Switching modes to ASTF is not an option for us at this time.
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/cisco-system-traffic-generator/trex-core/issues/156#issuecomment-428955916, or mute the thread https://github.com/notifications/unsubscribe-auth/AMbjvUILAaeb9XSYeCw1XpdJuWTqR4vpks5uj0gYgaJpZM4XI3bQ .
-- Hanoh Sent from my iPhone
As per deep investigations into #143 and #146, we were deeply assessing
frame.time_delta
values for a long netflix flow.Consider
Invoked with:
frame.time_delta
) because Netflix uses HTTP PIPELINING https://en.wikipedia.org/wiki/HTTP_pipelining====================================
$ tshark -Q -z io,stat,500 -r trex_ipg_10us_netflix_long_flow.pcap
======================================