Trex gui is consuming tons of cpu and memory after some time

[phschoen]

I usually leave the gui running and after some time (5h+) the gui gets unusable and slows down the whole system. here is a ps dump after 2 days letting it running without any traffic generation:

USER      PID    %CPU %MEM    VSZ      RSS     TTY        STAT START   TIME    COMMAND
phschoen  18763  93.3 3.6     12493080 1139896 pts/42 SI+ Nov 16       3758:10 java -jar trex-stateless-gui.jar

[phschoen]

was also reproduceable in 3.4

[EgorBlagov]

Hi,

is the GUI connected to trex during all this period? Or there is only GUI window open and nothing else?

[phschoen]

gui is opened and connected to a remote server which is not sending something. however some ports are reserved (2 out of 8). I will try if its also happening if no server is connected.

[phschoen]

actually its even growing just by opening the trex gui and leave it running. Also each click in a drop down menu is increasing the memory e.g. opening and closing the help dropdown is increasing the memory by 40k each time

[EgorBlagov]

@phschoen, thanks for the update I'll take a look into it.

[phschoen]

any updates?

[EgorBlagov]

Hi,

regarding menubar memory leak, I've checked on very lightweight jfx app on Java 8 (which we are using), and the same leak is reproduced, I've just selected dropdown and hold right arrow, and memory started to grow. So this issue is related to jfx itself, maybe in future if we will migrate to newer java and javafx it will be resolved (if newer javafx doesn't have this issue).

regrading main memory leak seems that I've found an error (looks like it related to logs, because once I've disabled logging for some classes who spam to logs and connected to trex memory is stalled around 257 MiB.

Please let me know your platforms (linux/windows/mac) and I'll prepare a build for your so you can check on your scenario.

Thanks, Egor.

[EgorBlagov]

HI, @phschoen

Please check latest v4.5.5. Once I've validated it was around 305-307 maximal after 30-40 minutes.

[phschoen]

hi, i tested the v4.5.5 on Ubuntu Linux and used java 8.211:

$java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

memory seems fine but cpu usage still goes over the roof and slows the pc down after some time (1 core with ~95% use no mater what).

[EgorBlagov]

Please, let me know latest scenario, how did you test it, and how long?

ср, 15 мая 2019 г., 23:50 Philipp Schönberger notifications@github.com:

hi, i tested the v4.5.5 on Ubuntu Linux and used java 8.211:

$java -version java version "1.8.0_211" Java(TM) SE Runtime Environment (build 1.8.0_211-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

memory seems fine but cpu usage still goes over the roof and slows the pc down after some time (1 core with ~95% use no mater what).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cisco-system-traffic-generator/trex-stateless-gui/issues/101?email_source=notifications&email_token=AINDXDI57TTXJWP7LKX3KITPVQ5MZA5CNFSM4GFDRAR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVPIQCQ#issuecomment-492734474, or mute the thread https://github.com/notifications/unsubscribe-auth/AINDXDMO2ESB44IC464NQQDPVQ5MZANCNFSM4GFDRARQ .

[phschoen]

i created a small script to plot it over time and gather some infos i started trex and created two simple ip/udp traffic with minimal pps and started it and let this running for some time.

trafic.yml

--- - name: "ip/udp-1" stream: action_count: 0 enabled: true flags: 2 flow_stats: enabled: true rule_type: "latency" stream_id: 2 isg: 0.0 mode: rate: type: "pps" value: 1.0 type: "continuous" total_pkts: 2 pkts_per_burst: 0 ibg: 0.0 count: 1 packet: binary: "AAAAAAAAAAAAAAAAgQAAZAgARQAATgTSAAB/Ece3CgEtCgoBLgoAZABkADIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" meta: "eyJwcm90b2NvbF9zZWxlY3Rpb24iOnsiaXNfdGFnZ2VkX3ZsYW5fc2VsZWN0ZWQiOnRydWUsImlzX2lwdjRfc2VsZWN0ZWQiOnRydWUsImlzX3RjcF9zZWxlY3RlZCI6ZmFsc2UsImlzX3VkcF9zZWxlY3RlZCI6dHJ1ZSwiaXNfcGF0dGVybl9zZWxlY3RlZCI6ZmFsc2UsImZyYW1lX2xlbmd0aF90eXBlIjoiRml4ZWQiLCJmcmFtZV9sZW5ndGgiOiIxMDAiLCJtYXhfbGVuZ3RoIjoiMTUxOCIsIm1pbl9sZW5ndGgiOiI2NCJ9LCJldGhlcm5ldCI6eyJ0eXBlIjoiMDgwMCIsImlzX292ZXJyaWRlIjpmYWxzZX0sIm1hYyI6eyJzb3VyY2UiOnsiYWRkcmVzcyI6IjAwOjAwOjAwOjAwOjAwOjAwIiwiY291bnQiOiIxNiIsIm1vZGUiOiJUUmV4IENvbmZpZyIsInN0ZXAiOiIxIn0sImRlc3RpbmF0aW9uIjp7ImFkZHJlc3MiOiIwMDowMDowMDowMDowMDowMCIsImNvdW50IjoiMTYiLCJtb2RlIjoiRml4ZWQiLCJzdGVwIjoiMSJ9fSwiaXB2NCI6eyJzb3VyY2UiOnsiYWRkcmVzcyI6IjEwLjEuNDUuMTAiLCJjb3VudCI6IjE2IiwibW9kZSI6IkZpeGVkIiwic3RlcCI6IjEifSwiZGVzdGluYXRpb24iOnsiYWRkcmVzcyI6IjEwLjEuNDYuMTAiLCJjb3VudCI6IjE2IiwibW9kZSI6IkZpeGVkIiwic3RlcCI6IjEifX0sInRjcCI6eyJpc19vdmVycmlkZV9kc3RfcG9ydCI6ZmFsc2UsImNoZWNrc3VtIjoiQjNFMyIsImRzdF9wb3J0IjoiMTAyNSIsInNyY19wb3J0IjoiMTAyNSIsImlzX292ZXJyaWRlX3NyY19wb3J0IjpmYWxzZSwiYWNrX251bWJlciI6IjAiLCJpc19vdmVycmlkZV9jaGVja3N1bSI6ZmFsc2UsInVyZ2VudF9wb2ludGVyIjoiMCIsImlzX3VyZyI6ZmFsc2UsImlzX2FjayI6ZmFsc2UsImlzX3BzaCI6ZmFsc2UsImlzX3JzdCI6ZmFsc2UsImlzX3N5bmMiOmZhbHNlLCJpc19maW4iOmZhbHNlLCJzZXF1ZW5jZV9udW1iZXIiOiIxMjkwMTgiLCJ3aW5kb3ciOiIxMDI0In0sInVkcCI6eyJpc19vdmVycmlkZV9kc3RfcG9ydCI6dHJ1ZSwiY2hlY2tzdW0iOiJGRkJBIiwiZHN0X3BvcnQiOiIxMDAiLCJzcmNfcG9ydCI6IjEwMCIsImlzX292ZXJyaWRlX2xlbmd0aCI6ZmFsc2UsImlzX292ZXJyaWRlX3NyY19wb3J0Ijp0cnVlLCJpc19vdmVycmlkZV9jaGVja3N1bSI6ZmFsc2UsImxlbmd0aCI6IjI2In0sInBheWxvYWQiOnsidHlwZSI6IkZpeGVkIFdvcmQiLCJwYXR0ZXJuIjoiMDAifSwidmxhbiI6eyJjZmkiOiIwIiwiaXNfb3ZlcnJpZGVfdHBfaWQiOmZhbHNlLCJ0cF9pZCI6IkZGRkYiLCJ2X2lkIjoiMTAwIiwicHJpb3JpdHkiOiIwIn0sImFkdmFuc2VkX3Byb3BlcnRpZXMiOnsiY2FjaGVfdmFsdWUiOiI1MDAwIiwiY2FjaGVfc2l6ZV90eXBlIjoiRW5hYmxlIn0sIm1vZGVsX3ZlcnNpb24iOiIxLjAifQ==" model: "" self_start: true advanced_mode: false vm: split_by_var: "" instructions: [] cache_size: 5000 stream_id: 0 - name: "ip/udp-2" stream: action_count: 0 enabled: true flags: 2 flow_stats: enabled: true rule_type: "latency" stream_id: 3 isg: 0.0 mode: rate: type: "pps" value: 1.0 type: "continuous" total_pkts: 2 pkts_per_burst: 0 ibg: 0.0 count: 1 packet: binary: "AAAAAAAAAAAAAAAAgQAAZAgARQAATgTSAAB/Ece3CgEtCgoBLgoAyADIADIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" meta: "eyJwcm90b2NvbF9zZWxlY3Rpb24iOnsiaXNfdGFnZ2VkX3ZsYW5fc2VsZWN0ZWQiOnRydWUsImlzX2lwdjRfc2VsZWN0ZWQiOnRydWUsImlzX3RjcF9zZWxlY3RlZCI6ZmFsc2UsImlzX3VkcF9zZWxlY3RlZCI6dHJ1ZSwiaXNfcGF0dGVybl9zZWxlY3RlZCI6ZmFsc2UsImZyYW1lX2xlbmd0aF90eXBlIjoiRml4ZWQiLCJmcmFtZV9sZW5ndGgiOiIxMDAiLCJtYXhfbGVuZ3RoIjoiMTUxOCIsIm1pbl9sZW5ndGgiOiI2NCJ9LCJldGhlcm5ldCI6eyJ0eXBlIjoiMDgwMCIsImlzX292ZXJyaWRlIjpmYWxzZX0sIm1hYyI6eyJzb3VyY2UiOnsiYWRkcmVzcyI6IjAwOjAwOjAwOjAwOjAwOjAwIiwiY291bnQiOiIxNiIsIm1vZGUiOiJUUmV4IENvbmZpZyIsInN0ZXAiOiIxIn0sImRlc3RpbmF0aW9uIjp7ImFkZHJlc3MiOiIwMDowMDowMDowMDowMDowMCIsImNvdW50IjoiMTYiLCJtb2RlIjoiRml4ZWQiLCJzdGVwIjoiMSJ9fSwiaXB2NCI6eyJzb3VyY2UiOnsiYWRkcmVzcyI6IjEwLjEuNDUuMTAiLCJjb3VudCI6IjE2IiwibW9kZSI6IkZpeGVkIiwic3RlcCI6IjEifSwiZGVzdGluYXRpb24iOnsiYWRkcmVzcyI6IjEwLjEuNDYuMTAiLCJjb3VudCI6IjE2IiwibW9kZSI6IkZpeGVkIiwic3RlcCI6IjEifX0sInRjcCI6eyJpc19vdmVycmlkZV9kc3RfcG9ydCI6ZmFsc2UsImNoZWNrc3VtIjoiQjNFMyIsImRzdF9wb3J0IjoiMTAyNSIsInNyY19wb3J0IjoiMTAyNSIsImlzX292ZXJyaWRlX3NyY19wb3J0IjpmYWxzZSwiYWNrX251bWJlciI6IjAiLCJpc19vdmVycmlkZV9jaGVja3N1bSI6ZmFsc2UsInVyZ2VudF9wb2ludGVyIjoiMCIsImlzX3VyZyI6ZmFsc2UsImlzX2FjayI6ZmFsc2UsImlzX3BzaCI6ZmFsc2UsImlzX3JzdCI6ZmFsc2UsImlzX3N5bmMiOmZhbHNlLCJpc19maW4iOmZhbHNlLCJzZXF1ZW5jZV9udW1iZXIiOiIxMjkwMTgiLCJ3aW5kb3ciOiIxMDI0In0sInVkcCI6eyJpc19vdmVycmlkZV9kc3RfcG9ydCI6dHJ1ZSwiY2hlY2tzdW0iOiJGRkJBIiwiZHN0X3BvcnQiOiIyMDAiLCJzcmNfcG9ydCI6IjIwMCIsImlzX292ZXJyaWRlX2xlbmd0aCI6ZmFsc2UsImlzX292ZXJyaWRlX3NyY19wb3J0Ijp0cnVlLCJpc19vdmVycmlkZV9jaGVja3N1bSI6ZmFsc2UsImxlbmd0aCI6IjI2In0sInBheWxvYWQiOnsidHlwZSI6IkZpeGVkIFdvcmQiLCJwYXR0ZXJuIjoiMDAifSwidmxhbiI6eyJjZmkiOiIwIiwiaXNfb3ZlcnJpZGVfdHBfaWQiOmZhbHNlLCJ0cF9pZCI6IkZGRkYiLCJ2X2lkIjoiMTAwIiwicHJpb3JpdHkiOiIwIn0sImFkdmFuc2VkX3Byb3BlcnRpZXMiOnsiY2FjaGVfdmFsdWUiOiI1MDAwIiwiY2FjaGVfc2l6ZV90eXBlIjoiRW5hYmxlIn0sIm1vZGVsX3ZlcnNpb24iOiIxLjAifQ==" model: "" self_start: true advanced_mode: false vm: split_by_var: "" instructions: [] cache_size: 5000 stream_id: 2

start.sh output

############### CPU: model name : Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz ############### Mem: total used free shared buffers cached Mem: 30918 11050 19868 289 106 2243 -/+ buffers/cache: 8699 22218 Swap: 32518 658 31860 ############### java: java version "1.8.0_211" Java(TM) SE Runtime Environment (build 1.8.0_211-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode) ############### trex version version=v4.5.5-0-gb80ac59 ############### start trex: ############### pid is 18080

start.sh to get cpu and mem

#!/bin/bash echo "###############" echo "CPU:" cat /proc/cpuinfo | grep "model name"| uniq echo "" echo "###############" echo "Mem:" free -m echo "" echo "###############" echo "java:" java -version echo "" echo "###############" echo "trex version" unzip -p trex-stateless-gui.jar version.properties echo "" echo "" echo "###############" echo "start trex:" java -jar trex-stateless-gui.jar > trex.log & pid=$! trap "echo '' kill $pid ! ; kill $pid ; exit -1; ./plot.sh" SIGINT SIGTERM START_TIME=$SECONDS echo "" echo "###############" echo "pid is $pid" rm trex_stats.dat touch trex_stats.dat while true do ELAPSED_TIME=$(($SECONDS - $START_TIME)) echo -n "$ELAPSED_TIME ">>trex_stats.dat ps -o pid,pcpu,rss,pmem -p $pid | tail -n1 >> trex_stats.dat sleep 10 if [[ $? -ne 0 ]] then break fi done

plot.sh to get graph

#!/bin/bash gnuplot << eor set title 'trex cpu/mem usage' set ylabel 'cpu %' set y2label 'mem kilobytes' set xlabel 'time in sec' set grid set terminal pngcairo size 2000,1000 set output 'cpu.png' set datafile separator whitespace set style line 1 \ linecolor rgb '#0060ad' \ linetype 1 linewidth 2 \ pointtype 7 pointsize 0.5 set style line 2 \ linecolor rgb '#ff60ad' \ linetype 2 linewidth 2 \ pointtype 7 pointsize 0.5 set ytics 10 nomirror set y2tics set key outside below plot 'trex_stats.dat' using 1:3 title "CPU" with linespoints linestyle 1 , \ 'trex_stats.dat' using 1:4 title "Mem" with linespoints linestyle 2 axes x1y2 eor