search

tariromukute / CoreNetworkTrafficGenerator

2 stars 1 forks source link

Duration most times zero and lots of failed procedures #5

Open bluemelov1 opened 2 weeks ago

bluemelov1 commented 2 weeks ago

I am currently facing the issue that the duration is always 0.0 seconds. I just had once a run where the duration was 128 seconds. But alle the other tests gave no result? What could be the problem here?

I've also noticed that often the procedures (I just use registration and de-registration) fail or the core network itself fails, resulting in a restart of the AMF. Are there ways to prevent this?

Thanks a lot!

This is an example statistic: ┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Item ┃ Results ┃ ┣━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫ ┃ Duration ┃ 0.0 seconds ┃ ┃ N# of UEs ┃ 20 ┃ ┃ Successful procedures ┃ 18 UEs ┃ ┃ Failed procedures ┃ 2 UEs ┃ ┃ Min interval ┃ 0.0 seconds ┃ ┃ Avg interval ┃ 11.924201117621529 seconds ┃ ┃ Max interval ┃ 22.10416030883789 seconds ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

tariromukute commented 1 week ago

Hi @bluemelov1, apologies for the delayed response. I will look into the duration issue, I noticed it's always zero. I see you are having 18 out of 20 UEs, which doesn't seem to be a lots. Can you run with log level of 5, i.e., -vvvvv. This will print the procedures that failed. Please share and the file and I will look into it.

Also, which core network are you working with?

bluemelov1 commented 1 week ago

Hello @tariromukute,

Thanks for your response! I am using the OAI core network for my benchmark. I tried to put a service mesh on top of the OAI core network to have encryption between the different NFs. So far it works but is far more likely to crash.

I tried to use verbosity level of 4 -vvvv or 5 -vvvvv but there was always this error seen in the log file here that the compliance mapper does not recognize this key: "KeyError: '5GMMRegistrationRequest'" log_vvvv.txt

Unfortunately I couldn't manage to get another result with 2 UEs out of 20 failing. Either all 20 worked or didn't work. So I have provided you with both the logs. Thank you for looking into it!

log_vvv_fail.txt log_vvv.txt

tariromukute commented 3 days ago

Hi @bluemelov1 I checked the log_vvv_fail.txt file. it looks like the Registration Request is sent but the OAI core network does not respond. Can you please share the following:

  1. The logs for the OAI core network especially, oai-amf.
  2. Share the manifest file to reproduce your setup i.e., either docker compose and it's respective config or Kubernetes deployment files etc.

Also I see that can you try for 10 UEs or less and see if you get the same behavior.

tariromukute commented 3 days ago

I think I may have found what the issue is. The mysql database might take some time to finish initialising. So in some cases it maybe that when the traffic generator sends the Registration requests, when the UE details don't exist yet. If you check the logs for oai-ausf you might find an error for the UE not existing. If you are starting everything at once, this might be an issue. Try starting core-network-traffic-generator a minute after the OAI core network (likely less).

Let me know if that resolves your issue.