NC | NSFS | CLI | Events Improvements

shirady commented 1 month ago

Environment info

NooBaa Version: noobaa-core-5.15.4-20240710-5.15.el9.x86_64.rpm (but probably relevant also in 5.17.0)
Platform: NC

Actual behavior

I had an account that was not deleted due to an error InternalError, and there was not created event for it.

Expected behavior

To see an event when the account could not be deleted, even if the error is InternalError.

Steps to reproduce

I don't have specific steps, but one should create a situation where the account cannot be deleted due to InternalError and check the noobaa_events.

More information - Screenshots / Logs / Other output

I saw it during the investigation of account deletion in a loop.

Example of an account that was not deleted s3user-15793:

➜  app2-test1107_logs cat script-c83f1-app2.noobaa-core-5.15.0-20240710.el9.x86_64.test1107 | grep 'Failed to execute command for Account s3user-15793' -A 4 -B 14
Running Command /usr/lpp/mmfs/bin/mmlsconfig cesSharedRoot
CallerPath of File utils/utils.go
Running Command /usr/lpp/mmfs/bin/mmces service list
Running Command: env LC_ALL=C /usr/local/bin/noobaa-cli account delete --name s3user-15793 2>/dev/null
Execution of command failed:
Info: {
  "error": {
    "code": "InternalError",
    "message": "The server encountered an internal error. Please retry the request",
    "cause": "Error: No such file or directory"
  }
}

Error: exit status 1
Failed to execute command for Account s3user-15793: The server encountered an internal error. Please retry the request
Error Code: InternalError
  Error Details: <nil>

15793 is done

Example of an account that was not deleted s3user-15793 and s3user-15794 is an account that was deleted:

➜  ~ cat noobaa_events.log | grep s3user-15793
{"timestamp":"2024-07-11T05:51:22.795303-04:00","message":"{\"code\":\"noobaa_account_created\",\"message\":\"Account created\",\"description\":\"Noobaa Account created\",\"entity_type\":\"NODE\",\"event_type\":\"INFO\",\"scope\":\"NODE\",\"severity\":\"INFO\",\"state\":\"HEALTHY\",\"arguments\":{\"account\":\"s3user-15793\"},\"pid\":2748042}","host":"c83f1-app2","severity":"notice","facility":"local2","syslog-tag":"node[2748042]:","source":"node"}
➜  ~ cat /Users/shiradymnik/Downloads/test1107_logs/app2-test1107_logs/noobaa_events.log | grep s3user-15794
{"timestamp":"2024-07-11T05:51:31.409283-04:00","message":"{\"code\":\"noobaa_account_created\",\"message\":\"Account created\",\"description\":\"Noobaa Account created\",\"entity_type\":\"NODE\",\"event_type\":\"INFO\",\"scope\":\"NODE\",\"severity\":\"INFO\",\"state\":\"HEALTHY\",\"arguments\":{\"account\":\"s3user-15794\"},\"pid\":2749481}","host":"c83f1-app2","severity":"notice","facility":"local2","syslog-tag":"node[2749481]:","source":"node"}
{"timestamp":"2024-07-11T07:52:19.880822-04:00","message":"{\"code\":\"noobaa_account_deleted\",\"message\":\"Account deleted\",\"description\":\"Noobaa Account deleted\",\"entity_type\":\"NODE\",\"event_type\":\"INFO\",\"scope\":\"NODE\",\"severity\":\"INFO\",\"state\":\"HEALTHY\",\"arguments\":{\"account\":\"s3user-15794\"},\"pid\":3763051}","host":"c83f1-app2","severity":"notice","facility":"local2","syslog-tag":"node[3763051]:","source":"node"}

messages from the node doesn't contain any relevant data:

➜  ~ cat app2-test1107_logs/messages  | grep s3user-15793
Jul 11 05:51:47 c83f1-app2 mmsysmon[1534699]: [I] Event raised: S3 account s3user-15793 was found.
Jul 11 05:51:47 c83f1-app2 mmsysmon[1534699]: [I] Event raised: The state of S3_ACCOUNT s3user-15793 of the component S3 changed to CHECKING.
Jul 11 05:52:14 c83f1-app2 mmsysmon[1534699]: [I] Event raised: S3 account s3user-15793 is OK.
Jul 11 05:52:14 c83f1-app2 mmsysmon[1534699]: [I] Event raised: The state of S3_ACCOUNT s3user-15793 of the component S3 changed to HEALTHY.

shirady commented 1 month ago

In PR #8195 I added a GAP - to add events for this error: https://github.com/noobaa/noobaa-core/blob/a23512d13f2ce2b5561a22e184a75f5504bb7997/src/manage_nsfs/manage_nsfs_cli_errors.js#L435

And additional error was added with this mapping, because the event NoobaaEvent.UNAUTHORIZED is not a perfect match.

shirady commented 1 month ago

To test the events on the local machine please add this line: console.log('SDSD log_event', JSON.stringify(event)); //SDSD as the last line in function log_event(event): https://github.com/noobaa/noobaa-core/blob/a23512d13f2ce2b5561a22e184a75f5504bb7997/src/util/debug_module.js#L498-L507

noobaa / noobaa-core