search

mojaloop / project

Repo to track product development issues for the Mojaloop project.
Other
23 stars 15 forks source link

Helm Test Intermittent failure with 'Generic ID not found' #2925

Open mdebarros opened 1 year ago

mdebarros commented 1 year ago

Summary: Helm Test Intermittent failure with 'Generic ID not found' when executing transfers via the Mojaloop-Simulator's API for POST /scenarios requests related tests, failing with below error; possibly indicating an issue with lookup (either a provisioning issue or bug - after some investigation the source of the issue seems to occur at the Mojaloop-Simulator)

{ errorCode: '3200', errorDescription: 'Generic ID not found' }

Example reports:

Work-around:

Re-run the helm test, specifically re-run the following Test Collections:

helm -n $NAMESPACE test $DEPLOYMENT_NAME --filter 'name=$DEPLOYMENT_NAME-ml-ttk-test-setup, name=$DEPLOYMENT_NAME-ml-ttk-test-val-gp'

Severity: Low

Priority: Low

Expected Behavior

helm tests should consistently pass without the ID not found with the Parties being correctly provisioned on the Mojaloop Simulators.

Steps to Reproduce

  1. Deploy v14.0.0 Mojaloop Helm Release
  2. Wait for all services to report healthy (i.e. 6-8m)
  3. Execute several watch -n 300 helm test moja2 --filter "name=moja2-ml-ttk-test-setup,name=moja2-ml-ttk-test-val-gp"
  4. Monitor test results, and cancel the watch command to stop running further tests

Specifications

Dependency:

Tasks for fix:

Note: If issue not found move to closed or blocked until issue re-occurs and then revive the story

Notes:

elnyry-sam-k commented 1 year ago

Hey team! Please add your planning poker estimate with Zenhub @mdebarros @sri-miriyala @vijayg10 @kleyow

kleyow commented 1 year ago

https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-16T20:31:35.199Z.html https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T00:47:19.760Z.html https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T01:00:19.346Z.html https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T01:07:27.254Z.html https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T13:27:07.171Z.html https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T14%3A46%3A53.209Z.html https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T15%3A49%3A08.698Z.html

kleyow commented 1 year ago

After running the tests for multiple hours I have not observed the specific error labeled in this story. I've added failed reports that stem mostly from 5xx errors.

https://github.com/mojaloop/testing-toolkit-test-cases/pull/91 may have resolved this issue.

Moving to blocked. To be re assigned if observed in daily cron job again. To be closed if not observed for a length period of time.

elnyry-sam-k commented 1 year ago

WIl re-open this if there's a repro, but closed for now as it is not reproducible

kleyow commented 1 year ago

Issue seems to present as "Id not found" and not "Generic ID not found". We believe this issue is still present with the latest helm v14.1.1 release. Issue seems to be related to either/both provisioning onboarding of parties for the mojaloop simulators tesfsp1-4 or the simulators forgeting the addition of parties since they are stored in memory.

mdebarros commented 1 year ago

PartyId that is missing from the testfsp4's backend (i.e. mojaloop/mojaloop-simulator:v13.0.1):

17891239872

Log from moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc:

{"ts":"2023-03-07T14:55:56.235Z","msg":"Found handler","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-inbound-api","component":"api","request":{"id":"abandoned-orange-deafening-yak","path":"/parties/MSISDN/17891239872","method":"GET"},"handler":"[Function: getPartiesByTypeAndId]"}}
{"ts":"2023-03-07T14:55:56.237Z","msg":"Executing HTTP GET","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-inbound-api","component":"api","request":{"id":"abandoned-orange-deafening-yak","path":"/parties/MSISDN/17891239872","method":"GET"},"reqOpts":{"method":"GET","uri":"http://moja4-sim-testfsp4-backend:3000/parties/MSISDN/17891239872","headers":{"Content-Type":"application/json","Accept":"application/json","Date":"Tue, 07 Mar 2023 14:55:56 GMT"}}}}
{"ts":"2023-03-07T14:55:56.239Z","msg":"Request processed","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-inbound-api","component":"api","request":{"id":"abandoned-orange-deafening-yak","path":"/parties/MSISDN/17891239872","method":"GET"}}}
{"ts":"2023-03-07T14:55:56.240Z","msg":"Cache message received on channel __keyevent@0__:set. Making callback with id 0","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","component":"cache"}}
{"ts":"2023-03-07T14:55:56.240Z","msg":"Received Redis keyevent notification","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-test-api","component":"websocket-server","key":"request_17891239872","channel":"__keyevent@0__:set","id":"0"}}
{"ts":"2023-03-07T14:55:56.248Z","msg":"Error in getParties","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-inbound-api","component":"api","request":{"id":"abandoned-orange-deafening-yak","path":"/parties/MSISDN/17891239872","method":"GET"},"err":"{\"msg\":\"Request returned non-success status code 404\",\"res\":{\"statusCode\":404,\"headers\":{\"content-type\":\"application/json; charset=utf-8\",\"content-length\":\"46\",\"date\":\"Tue, 07 Mar 2023 14:55:56 GMT\",\"connection\":\"keep-alive\",\"keep-alive\":\"timeout=5\"},\"data\":{\"statusCode\":\"3200\",\"message\":\"ID not found\"},\"originalRequest\":{\"method\":\"GET\",\"host\":\"moja4-sim-testfsp4-backend\",\"port\":\"3000\",\"path\":\"/parties/MSISDN/17891239872\",\"headers\":{\"Content-Type\":\"application/json\",\"Accept\":\"application/json\",\"Date\":\"Tue, 07 Mar 2023 14:55:56 GMT\"},\"agent\":\"[REDACTED]\",\"body\":null}}}"}}

Request to backend to list all parties

curl 'http://moja4-sim-testfsp4-backend:3003/repository/parties' -H 'traceparent: 00-aabbe4cf5b5aa1b531408d4bbddbcd5c-0123456789abcdef0-00' -H 'user-agent: axios/0.27.2' --compressed

Response with missing 17891239872 party

{
  "status": 200,
  "statusText": "OK",
  "body": [
    {
      "displayName": "testfsp4BankFname testfsp4Bankname",
      "firstName": "testfsp4BankFname",
      "middleName": "",
      "lastName": "testfsp4BankLname",
      "dateOfBirth": "1985-05-13",
      "idType": "MSISDN",
      "idValue": "27713813917",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Bank"
        }
      ]
    },
    {
      "displayName": "fsp4WalletFname{{$timestamp}} fsp4WalletLname{{$timestamp}}",
      "firstName": "fsp4WalletFname{{$timestamp}}",
      "middleName": "",
      "lastName": "fsp4WalletLname{{$timestamp}}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "476036885579",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Wallet"
        }
      ]
    },
    {
      "displayName": "fsp4BankFname{$timestamp} fsp4BankLname{$timestamp}",
      "firstName": "fsp4BankFname{$timestamp}",
      "middleName": "",
      "lastName": "fsp4BankLname{$timestamp}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "124241848903",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Bank"
        }
      ]
    },
    {
      "displayName": "fsp4WalletFname{{$timestamp}} fsp4WalletLname{{$timestamp}}",
      "firstName": "fsp4WalletFname{{$timestamp}}",
      "middleName": "",
      "lastName": "fsp4WalletLname{{$timestamp}}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "258100991099",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Wallet"
        }
      ]
    },
    {
      "displayName": "fsp4BankFname{$timestamp} fsp4BankLname{$timestamp}",
      "firstName": "fsp4BankFname{$timestamp}",
      "middleName": "",
      "lastName": "fsp4BankLname{$timestamp}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "511367992055",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Bank"
        }
      ]
    },
    {
      "displayName": "fsp4WalletFname{{$timestamp}} fsp4WalletLname{{$timestamp}}",
      "firstName": "fsp4WalletFname{{$timestamp}}",
      "middleName": "",
      "lastName": "fsp4WalletLname{{$timestamp}}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "901216703790",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Wallet"
        }
      ]
    },
    {
      "displayName": "fsp4BankFname{$timestamp} fsp4BankLname{$timestamp}",
      "firstName": "fsp4BankFname{$timestamp}",
      "middleName": "",
      "lastName": "fsp4BankLname{$timestamp}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "654117426119",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Bank"
        }
      ]
    }
  ],
  "headers": {
    "vary": "Origin",
    "content-type": "application/json; charset=utf-8",
    "content-length": "2051",
    "date": "Tue, 07 Mar 2023 16:25:32 GMT",
    "connection": "keep-alive",
    "keep-alive": "timeout=5"
  }
}

GET http://moja4-sim-testfsp4-backend-9db946c89-49v5h:3000/parties/MSISDN/17891239872

{"statusCode":"3200","message":"ID not found"}

Log from moja4-sim-testfsp4-backend-9db946c89-49v5h:

{
  app: 'simulator',
  request: {
    id: 'steep-calm-helpful-furniture',
    path: '/parties/MSISDN/17891239872',
    method: 'GET'
  },
  response: {
    body: { statusCode: '3200', message: 'ID not found' },
    status: 404
  },
  msg: 'Request processed',
  timestamp: '2023-03-07T14:55:56.246Z'
}

Archive.zip

mdebarros commented 1 year ago

Re-running the API operation to create the Party on the Mojaloop-Simulator after a restart:

curl 'http://moja4-sim-testfsp4-backend:3003/repository/parties' -H 'content-type: application/json' -H 'traceparent: 00-aabbe4cf5b5aa1b531408d4bbddbcd90-0123456789abcdef0-00' -H 'user-agent: axios/0.27.2' --data-binary '{"displayName":"testfsp4WalletFname testfsp4WalletLname","firstName":"testfsp4WalletFname","middleName":"","lastName":"testfsp4WalletLname","dateOfBirth":"1985-05-13","idType":"MSISDN","idValue":"17891239872","extensionList":[{"key":"accountType","value":"Wallet"}]}' --compresse
{
  "status": 200,
  "statusText": "OK",
  "body": [
    {
      "displayName": "testfsp4WalletFname testfsp4WalletLname",
      "firstName": "testfsp4WalletFname",
      "middleName": "",
      "lastName": "testfsp4WalletLname",
      "dateOfBirth": "1985-05-13",
      "idType": "MSISDN",
      "idValue": "17891239872",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Wallet"
        }
      ]
    }
  ],
  "headers": {
    "vary": "Origin",
    "access-control-allow-origin": "",
    "content-type": "application/json; charset=utf-8",
    "content-length": "268",
    "date": "Tue, 07 Mar 2023 16:56:28 GMT",
    "connection": "keep-alive",
    "keep-alive": "timeout=5"
  }
}
mdebarros commented 1 year ago

Unable to determine the cause of the missing participant, except that it is seems to be occuring during the onboarding setup...

The provisioning script hub/provisioning/CGS_Specific/add-users-to-new-sims-and-als-registration.json still shows the operation as being successful: 

  Add Users to new Sims ; ALS registration -> [testfsp4, Wallet] POST /parties  {{SIM4_MSISDN}}
    [ SUCCESS ]   resposne code is 200

Will need to re-look at this once mojaloop-simulator/releases/tag/v13.1.0 or later is packaged in a release. This version will also include audits for incoming repository requests which will help identify the cause of the issue.