VAOS::AppointmentRequestsService#post_request - VAMF returning status 500

[grossb1]

As of this 02/10/2021 (02/03/2021-02/10/2021) there were 733 occurrences of this error.

• URL: /var/VeteranAppointmentRequestService/v4/rest/appointment-service/patient/ICN/xxx/appointments

Team will investigate Prometheus, Cloudwatch and the vets-api code-base to determine root-cause of this issue.

Deliverables

• A brief summary of the the results including any data that will help VAMF in identifying the issue.
• Include any supporting documents as attachments (e.g. spreadsheet, etc...).
• A list of any tickets that were opened in Jira.

Notes

• There are several different type of 500 responses, some with more information than others.
• Example Sentry URL with traceId: http://sentry.vfs.va.gov/organizations/vsp/issues/15043/events/83b09aef15f24787a497aa0c27207de6/?project=3&statsPeriod=14d
  • TraceId: af21a960b7d08869

[grossb1]

Here is an example of an error (cloudwatch query: {($.named_tags.request_id="fbb37738-2fa6-48f3-bbc3-a50c20d350c9")} ) :

{
    "host": "0661c2397a0b",
    "application": "vets-api-server",
    "timestamp": "2021-02-12T06:04:53.639991Z",
    "level": "error",
    "level_index": 4,
    "pid": 29,
    "thread": "puma threadpool 003",
    "file": "/srv/vets-api/src/lib/sentry_logging.rb",
    "line": 55,
    "named_tags": {
        "request_id": "fbb37738-2fa6-48f3-bbc3-a50c20d350c9",
        "remote_ip": "137.118.215.37",
        "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36",
        "ref": "269b6c0b218a90d345314f969d14582a02c6cea3",
        "csrf_token": "AuP2QLIVxO2YA19Cyr7FFaA9VLw89o7CpfeUAy4dB21tRzCeuig296nJ1/3gY9km7lZdr9QMNeOsf2PzxFuCsg=="
    },
    "name": "Rails",
    "message": "BackendServiceException: {:detail=>\"Could not fetch veteran identifiers from Mobile MVI Service\", :source=>{:vamf_url=>#<URI::HTTPS https://internal-dsva-vagov-prod-fwdproxy-2075821597.us-gov-west-1.elb.amazonaws.com:4463/var/VeteranAppointmentRequestService/v4/rest/appointment-service/patient/ICN/xxx/appointments?endDate=02%2F12%2F2021&startDate=01%2F13%2F2021>, :vamf_body=>\"{\\\"id\\\":\\\"6cb893e8-9e0f-4a4b-87a7-60b9ddad0725\\\",\\\"traceId\\\":\\\"b3fe6f7fbcc0c453\\\",\\\"code\\\":500,\\\"message\\\":\\\"Could not fetch veteran identifiers from Mobile MVI Service\\\",\\\"errorCode\\\":2000,\\\"detail\\\":\\\"500: {\\\\\\\"code\\\\\\\":500,\\\\\\\"message\\\\\\\":\\\\\\\"A server error occurred\\\\\\\",\\\\\\\"id\\\\\\\":\\\\\\\"8d928a58-cb9b-4427-af55-9a0aa93b2949\\\\\\\",\\\\\\\"traceId\\\\\\\":\\\\\\\"b3fe6f7fbcc0c453\\\\\\\"}\\\",\\\"meta\\\":{}}\", :vamf_status=>500}, :code=>\"VAOS_502\"}",
    "payload": {
        "title": "Bad Gateway",
        "detail": "Received an an invalid response from the upstream server",
        "code": "VAOS_502",
        "source": {
            "vamf_url": "https://internal-dsva-vagov-prod-fwdproxy-2075821597.us-gov-west-1.elb.amazonaws.com:4463/var/VeteranAppointmentRequestService/v4/rest/appointment-service/patient/ICN/xxx/appointments?endDate=02%2F12%2F2021&startDate=01%2F13%2F2021",
            "vamf_body": "{\"id\":\"6cb893e8-9e0f-4a4b-87a7-60b9ddad0725\",\"traceId\":\"b3fe6f7fbcc0c453\",\"code\":500,\"message\":\"Could not fetch veteran identifiers from Mobile MVI Service\",\"errorCode\":2000,\"detail\":\"500: {\\\"code\\\":500,\\\"message\\\":\\\"A server error occurred\\\",\\\"id\\\":\\\"8d928a58-cb9b-4427-af55-9a0aa93b2949\\\",\\\"traceId\\\":\\\"b3fe6f7fbcc0c453\\\"}\",\"meta\":{}}",
            "vamf_status": 500
        },
        "status": "502",
        "backtrace": [...