Sentry(pipelined): Validation error in SetSMFSessions server

[sentry-io[bot]]

Sentry Issue: LAB-AGWS-PYTHON-6S

Error : Parsing Error in SetInterface Message 4

[sebathomas]

I think there are two sides to this Sentry issue:

1. Find out why the client (sessiond) sends invalid requests.
2. A validation error in a gRPC request is not an unexpected error from the point of view of the server and should not be logged on level error.

[Neudrino]

11403 resolves point 2 on your list, right?

I think it should also resolve this issue, as well as the Sentry issue. For point 1 on your list, I think we should create another issue, if it is not easily resolvable as well. Thoughts?

[sebathomas]

I think it's an interesting case we haven't really had before.

From the point of view of keeping Sentry clean, it's solved after the PR is merged. However, it indicates that sessiond can send invalid messages to pipelined. So there's probably a bug or some edge case that's not handled there.

On the other hand, Sentry doesn't show you the invalid message so it's not really possible to debug without access to the logs of the server where it happened (TVM3-SA-regression).

I don't really have a preference about whether to create another issue or to leave this one open.

[sebathomas]

Might be relevant for the sessiond code owners. (I think it's the only caller of the interface.) @magma/approvers-agw-sessiond @uri200 @themarwhal

[themarwhal]

@GANESH-WAVELABS, @ulaskozat It looks like this issue is seen during 5G testing. Who would be the best person to get more context on this?

[ulaskozat]

cc @daljit-mahi could you steer this?

[GANESH-WAVELABS]

Hi All,

When I browse the code below error/warning message 4 means MANDATORY_IE_MISSING Error : Parsing Error in SetInterface Message 4

Please refer the below screen shot:

Also this could happen when new session operation is without any pdr group, below is the code snippet for the same.

# Check if the new session operation is without any pdr group
        if not new_session.set_gr_pdr:
            return CauseIE.MANDATORY_IE_MISSING

We could not reproduce this in our local setup. Tried with stub CLI magma/lte/gateway/python/scripts/pipelined_cli.py as well by missing some optional parameters but it was not reproduced. Not sure in which scenario pdr group absent.

@sebathomas Is it reproducing frequently ? If so Could you please follow below steps before reproducing the issue:

1) change the log_level to DEBUG and enable grpc messages by following below steps:
cat /etc/magma/sessiond.yml |grep -A 2 log_level
log_level: INFO --> DEBUG
print_grpc_payload: false --> true

2) Add the log_level to DEBUG and enable grpc messages by following below steps:
cat /etc/magma/pipelined.yml |grep -A 2 grpc
# Logs grpc payload content
magma_print_grpc_payload: false --> true
log_level: DEBUG ---> This line should add newly into this file, by default it was not present

3) restart all the services with below steps:
sudo service magma@* stop
sudo service magma@magmad restart

Once issue reproduces please share the file /var/log/syslog

[sebathomas]

It hasn't occurred frequently (and the issue is no longer monitored on the Python side). No way to reproduce it so far unfortunately.

[stale[bot]]

This pull request has been automatically marked as stale because it has not had any recent activity by the author or maintainer in the past 45 days. It will be closed if no further activity occurs in the next 7 days. Thank you for your contributions.