Investigate why filesystem I/O impacts live stream

[njam]

Problem: During high I/O on the janus-gateway machine we experience stuttering of the video stream. The clients experience "packet loss" (probably those packets are never sent?).

---

To reproduce: Create a big file:

truncate -s 100M 100M

Then drop FS caches and copy that file to another file in a loop:

while (true); do sync && echo 3 > /proc/sys/vm/drop_caches && rm -f 100M-copy && rsync -a 100M 100M-copy && echo -n .; done

To observe "iowait" run this command in another terminal:

sudo iostat -x 1

During that operation the WebRTC stream will stutter substantially.

---

Unfortunately this can't be solved with "ionice". This command (with "ionice") has the exact same effect:

while (true); do sync && echo 3 > /proc/sys/vm/drop_caches && rm -f 100M-copy && ionice -c3 rsync -a 100M 100M-copy && echo -n .; done

[kris-lab]

@njam do we still experience this problem on the production? If yes, what should we do?

[njam]

I haven't checked, but I guess it's still the case as we didn't fix it.

I guess we would need to investigate in rtpbroadcast how we can prevent it from happening. Might be a lengthy investigation.

[kris-lab]

Yep, but let's verify this once we have Debian-8 in place or maybe with current janus5 instance?

[njam]

Agree, let's test it with Debian 8. But I think we can postpone it until after the scalability project.

[kris-lab]

closing for now.