InferICD10CMRequest response issus

[f8y387512]

Describe the bug

Resource

Clinical Note

1.Left hip to left thigh swelling pain with limited of motion, because
  of left femur intertrochanteric comminuted fracture with displaced s/p
  ORIF with DHS in post-op 6th day
2.Moderate swelling pain with limited of motion and skin ecchymosis over 
  left wrist area, due to left distal radius intraarticular commintued
  fracture post ORIF with T-plate and left ulnar styloid fracture in 
  post-op 6th day-->keep short arm splint fixation
3.Previous history of left breast cancer s/p MRM on 106/08/28 with 
  FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology, 
  RECOMMENDATION: It is appropriate now to discontinue the use of 
  letrozole. I will see her again at my OPD 
4.Hypertension with LMD medication treatment
5.Type diabetes mellitus-->current keep RI scale and LMD medication: 
  Glubin 500mg 1tab bidpc po for blood sugar control
6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl

Diagnosis of Interest

left ulnar styloid fracture

Issue Description

AWS Comprehend API performs differently via C# and Python. In C# SDK there are uncatched diagnoses. For example "left ulnar styloid fracture" with ICD-10-CM code S52612A was missing with C# API call.

Responded JSON comparison

OK = Result with S52XXXX diagnoses (comprises S52612A) NG = Result without S52XXXX diagnoses

API	Clinical Note	Diagnosis
Aws Web	OK	OK
C#	NG	OK
Python	OK	OK

Responded JSON Differences

The left hand side C# SDK responded JSON is with three phrases: fracture, left, ulnar; No issue for Python or via the the Web interface.

When doing short clinical note, no issue found in C# SDK Call.

Expected Behavior

summaryResult json

"ICD10CMConcepts": [
          {
            "Code": "S52.612",
            "Description": "Displaced fracture of left ulna styloid process",
            "Score": 0.054005634
          },
          {
            "Code": "S52.611",
            "Description": "Displaced fracture of right ulna styloid process",
            "Score": 0.04087594
          },
          {
            "Code": "S52.615",
            "Description": "Nondisplaced fracture of left ulna styloid process",
            "Score": 0.040831503
          },
          {
            "Code": "S52.61",
            "Description": "Fracture of ulna styloid process",
            "Score": 0.026959484
          },
          {
            "Code": "S52.615D",
            "Description": "Nondisplaced fracture of left ulna styloid process, subsequent encounter for closed fracture with routine healing",
            "Score": 0.026853003
          }
        ]

Current Behavior

noteResult Json

      "Id": 22,
      "Score": 0.9441205,
      "Text": "fracture",
      "Traits": [
        {
          "Name": {
            "Value": "DIAGNOSIS"
          },
          "Score": 0.6549435
        }
      ],
      "Type": {
        "Value": "DX_NAME"
      }
    },
    {
      "Attributes": [
        {
          "BeginOffset": 367,
          "EndOffset": 371,
          "Id": 24,
          "RelationshipScore": 0.63903964,
          "Score": 0.9999672,
          "Text": "left",
          "Traits": [],
          "Type": {
            "Value": "DIRECTION"
          }
        },
        {
          "BeginOffset": 372,
          "EndOffset": 377,
          "Id": 25,
          "RelationshipScore": 0.9993754,
          "Score": 0.9940257,
          "Text": "ulnar",
          "Traits": [],
          "Type": {
            "Value": "SYSTEM_ORGAN_SITE"
          }
        }
      ],
      "BeginOffset": 386,
      "Category": {
        "Value": "MEDICAL_CONDITION"
      },
      "EndOffset": 394,
      "ICD10CMConcepts": [
        {
          "Code": "S62.601D",
          "Description": "Fracture of unspecified phalanx of left index finger, subsequent encounter for fracture with routine healing",
          "Score": 0.02694933
        },
        {
          "Code": "S22.32XD",
          "Description": "Fracture of one rib, left side, subsequent encounter for fracture with routine healing",
          "Score": 0.025233312
        },
        {
          "Code": "S62.101A",
          "Description": "Fracture of unspecified carpal bone, right wrist, initial encounter for closed fracture",
          "Score": 0.019765776
        },
        {
          "Code": "S62.102A",
          "Description": "Fracture of unspecified carpal bone, left wrist, initial encounter for closed fracture",
          "Score": 0.019586816
        },
        {
          "Code": "S12.121D",
          "Description": "Other nondisplaced dens fracture, subsequent encounter for fracture with routine healing",
          "Score": 0.013089595
        }
      ],
      "Id": 26,
      "Score": 0.93353933,
      "Text": "fracture",
      "Traits": [
        {
          "Name": {
            "Value": "DIAGNOSIS"
          },
          "Score": 0.6684849
        }
      ],
      "Type": {
        "Value": "DX_NAME"
      }
    },
    {
      "Attributes": [
        {
          "BeginOffset": 474,
          "EndOffset": 478,
          "Id": 29,
          "RelationshipScore": 0.99908006,
          "Score": 0.9999906,
          "Text": "left",
          "Traits": [],
          "Type": {
            "Value": "DIRECTION"
          }
        }

Reproduction Steps

C# sample code

    [Test]
    public async Task sampleCode()
    {
        var accessKey = "accessKey";
        var secretKey = "secretKey";

        string note = "1.Left hip to left thigh swelling pain with limited of motion, because\r\n  of left femur intertrochanteric comminuted fracture with displaced s/p\r\n  ORIF with DHS in post-op 6th day\r\n2.Moderate swelling pain with limited of motion and skin ecchymosis over \r\n  left wrist area, due to left distal radius intraarticular commintued\r\n  fracture post ORIF with T-plate and left ulnar styloid fracture in \r\n  post-op 6th day-->keep short arm splint fixation\r\n3.Previous history of left breast cancer s/p MRM on 106/08/28 with \r\n  FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology, \r\n  RECOMMENDATION: It is appropriate now to discontinue the use of \r\n  letrozole. I will see her again at my OPD \r\n4.Hypertension with LMD medication treatment\r\n5.Type diabetes mellitus-->current keep RI scale and LMD medication: \r\n  Glubin 500mg 1tab bidpc po for blood sugar control\r\n6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl";
        string summary = "left ulnar styloid fracture";

        var noteResult = await CreateClient().InferICD10CMAsync(CreatRequest(note));
        var summaryResult = await CreateClient().InferICD10CMAsync(CreatRequest(summary));

        IAmazonComprehendMedical CreateClient()
        {
            return new AmazonComprehendMedicalClient(
                new BasicAWSCredentials(accessKey, secretKey),
                new AmazonComprehendMedicalConfig { RegionEndpoint = RegionEndpoint.USEast2 });
        }

        InferICD10CMRequest CreatRequest(string context)
        {
            return new InferICD10CMRequest { Text = context };
        }
    }

python sample code

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

System Configuration

VS2022 Ent PreView 17.5.0 preview 1.0 TargetFramework net6.0 AWSSDK.ComprehendMedical 3.7.101.10

Targeted .NET Platform

.NET 6

Operating System and version

Windows 10 , Windows 11 , Ubuntu 22.04 LTS

[ashishdhingra]

Hi @f8y387512,

Good morning. I tried reproducing the issue using the package AWSSDK.ComprehendMedical version 3.7.101.10 using the below code:

using Amazon;
using Amazon.ComprehendMedical;
using Amazon.ComprehendMedical.Model;
using System.Text.Json;

string note = "1.Left hip to left thigh swelling pain with limited of motion, because\r\n  of left femur intertrochanteric comminuted fracture with displaced s/p\r\n  ORIF with DHS in post-op 6th day\r\n2.Moderate swelling pain with limited of motion and skin ecchymosis over \r\n  left wrist area, due to left distal radius intraarticular commintued\r\n  fracture post ORIF with T-plate and left ulnar styloid fracture in \r\n  post-op 6th day-->keep short arm splint fixation\r\n3.Previous history of left breast cancer s/p MRM on 106/08/28 with \r\n  FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology, \r\n  RECOMMENDATION: It is appropriate now to discontinue the use of \r\n  letrozole. I will see her again at my OPD \r\n4.Hypertension with LMD medication treatment\r\n5.Type diabetes mellitus-->current keep RI scale and LMD medication: \r\n  Glubin 500mg 1tab bidpc po for blood sugar control\r\n6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl";
string summary = "fracture、left、ulnar";

var noteResult = await CreateClient().InferICD10CMAsync(CreatRequest(note));
var summaryResult = await CreateClient().InferICD10CMAsync(CreatRequest(summary));

Console.WriteLine(JsonSerializer.Serialize(noteResult));
Console.WriteLine();
Console.WriteLine(JsonSerializer.Serialize(summaryResult));
Console.ReadKey();

IAmazonComprehendMedical CreateClient()
{
    return new AmazonComprehendMedicalClient(RegionEndpoint.USEast2);
}

InferICD10CMRequest CreatRequest(string context)
{
    return new InferICD10CMRequest { Text = context };
}

Attached is the serialized JSON result returned by the service" C#_Summary.txt C#_Note.txt

Executing the same command using AWS CLI:

aws comprehendmedical infer-icd10-cm --text "1.Left hip to left thigh swelling pain with limited of motion, because\r\n  of left femur intertrochanteric comminuted fracture with displaced s/p\r\n  ORIF with DHS in post-op 6th day\r\n2.Moderate swelling pain with limited of motion and skin ecchymosis over \r\n  left wrist area, due to left distal radius intraarticular commintued\r\n  fracture post ORIF with T-plate and left ulnar styloid fracture in \r\n  post-op 6th day-->keep short arm splint fixation\r\n3.Previous history of left breast cancer s/p MRM on 106/08/28 with \r\n  FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology, \r\n  RECOMMENDATION: It is appropriate now to discontinue the use of \r\n  letrozole. I will see her again at my OPD \r\n4.Hypertension with LMD medication treatment\r\n5.Type diabetes mellitus-->current keep RI scale and LMD medication: \r\n  Glubin 500mg 1tab bidpc po for blood sugar control\r\n6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl" > aws_cli_note.json

aws comprehendmedical infer-icd10-cm --text "fracture、left、ulnar" > aws_cli_summary.json

appears to return similar result (execpt for JSON serialization differences and order): aws_cli_summary.txt aws_cli_note.txt

I'm unsure how Python SDK is handling the scenario. The difference I could see in the C# code is the use of \r\n characters, I'm unsure if it would affect the result.

Also note that the SDK code for the ComprehendMedical service is auto-generated based on service models, SDK is simply returning the results of the service API call to the requestor. We are using AWS CLI for comparison since it should in effect have the similar behavior. Based on the comparison and the fact that the SDK is simply returning the output of service API operation, I do not see this as an AWS .NET SDK bug.

Please examine the attached C# and CLI results and let me know if I might have missed any differences. Also, could you try removing \r\n and may be try with \n or removing these characters all together.

MORE FINDINGS: If I replace \r\n characters in the middle of the sentence (for example first bullet point), I get the expected result:

{
            "Attributes": [],
            "BeginOffset": 353,
            "Category": {
                "Value": "MEDICAL_CONDITION"
            },
            "EndOffset": 380,
            "ICD10CMConcepts": [
                {
                    "Code": "S52.612",
                    "Description": "Displaced fracture of left ulna styloid process",
                    "Score": 0.018345313
                },
                {
                    "Code": "S52.615",
                    "Description": "Nondisplaced fracture of left ulna styloid process",
                    "Score": 0.013606193
                },
                {
                    "Code": "S52.615D",
                    "Description": "Nondisplaced fracture of left ulna styloid process, subsequent encounter for closed fracture with routine healing",
                    "Score": 0.012032543
                },
                {
                    "Code": "S52.61",
                    "Description": "Fracture of ulna styloid process",
                    "Score": 0.011936317
                },
                {
                    "Code": "S52.613",
                    "Description": "Displaced fracture of unspecified ulna styloid process",
                    "Score": 0.00960199
                }
            ],
            "Id": 25,
            "Score": 0.76039815,
            "Text": "left ulnar styloid fracture",
            "Traits": [
                {
                    "Name": {
                        "Value": "DIAGNOSIS"
                    },
                    "Score": 0.6071009
                }
            ],
            "Type": {
                "Value": "DX_NAME"
            }
        }

Thanks, Ashish

[f8y387512]

Conclusion

1. We verified the results responded from Web and CLI are the same, according to ashishdhingra's comments.
2. We noticed during the try-and-error, C# SDK handles return special characters is different than Web/CLI.
3. Comprehend NLP performs quite differently based on the same content with different ways of handling return characters.

The following approaches were used to compare how NLP responded differently. We then realized NLP SDK operates \r\n differently. If we used \n instead of \r\n, then the results equal to WEB/CLI.

Original Text

SDK possible operation

Operation with our replacement for characters

Content after our replacement for characters

1.Left hip to left thigh swelling pain with limited of motion, because  of left femur intertrochanteric comminuted fracture with displaced s/p  ORIF with DHS in post-op 6th day
2.Moderate swelling pain with limited of motion and skin ecchymosis over   left wrist area, due to left distal radius intraarticular commintued  fracture post ORIF with T-plate and left ulnar styloid fracture in   post-op 6th day-->keep short arm splint fixation
3.Previous history of left breast cancer s/p MRM on 106/08/28 with   FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology,   RECOMMENDATION: It is appropriate now to discontinue the use of   letrozole. I will see her again at my OPD 
4.Hypertension with LMD medication treatment
5.Type diabetes mellitus-->current keep RI scale and LMD medication:   Glubin 500mg 1tab bidpc po for blood sugar control
6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl

AWS CLI

aws comprehendmedical infer-icd10-cm --text "1.Left hip to left thigh swelling pain with limited of motion, because
  of left femur intertrochanteric comminuted fracture with displaced s/p
  ORIF with DHS in post-op 6th day
2.Moderate swelling pain with limited of motion and skin ecchymosis over 
  left wrist area, due to left distal radius intraarticular commintued
  fracture post ORIF with T-plate and left ulnar styloid fracture in 
  post-op 6th day-->keep short arm splint fixation
3.Previous history of left breast cancer s/p MRM on 106/08/28 with 
  FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology, 
  RECOMMENDATION: It is appropriate now to discontinue the use of 
  letrozole. I will see her again at my OPD 
4.Hypertension with LMD medication treatment
5.Type diabetes mellitus-->current keep RI scale and LMD medication: 
  Glubin 500mg 1tab bidpc po for blood sugar control
6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl" > aws_cli_note_result_with_cr.json

aws comprehendmedical infer-icd10-cm --text "1.Left hip to left thigh swelling pain with limited of motion, because  of left femur intertrochanteric comminuted fracture with displaced s/p  ORIF with DHS in post-op 6th day2.Moderate swelling pain with limited of motion and skin ecchymosis over   left wrist area, due to left distal radius intraarticular commintued  fracture post ORIF with T-plate and left ulnar styloid fracture in   post-op 6th day-->keep short arm splint fixation3.Previous history of left breast cancer s/p MRM on 106/08/28 with   FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology,   RECOMMENDATION: It is appropriate now to discontinue the use of   letrozole. I will see her again at my OPD 4.Hypertension with LMD medication treatment5.Type diabetes mellitus-->current keep RI scale and LMD medication:   Glubin 500mg 1tab bidpc po for blood sugar control6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl" > aws_cli_note_result_without_cr.json

C# sample code

    [Test]
    public async Task sampleCode()
    {
        var accessKey = AccessKey;
        var secretKey = SecretKey;

        string note = "1.Left hip to left thigh swelling pain with limited of motion, because\r\n  of left femur intertrochanteric comminuted fracture with displaced s/p\r\n  ORIF with DHS in post-op 6th day\r\n2.Moderate swelling pain with limited of motion and skin ecchymosis over \r\n  left wrist area, due to left distal radius intraarticular commintued\r\n  fracture post ORIF with T-plate and left ulnar styloid fracture in \r\n  post-op 6th day-->keep short arm splint fixation\r\n3.Previous history of left breast cancer s/p MRM on 106/08/28 with \r\n  FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology, \r\n  RECOMMENDATION: It is appropriate now to discontinue the use of \r\n  letrozole. I will see her again at my OPD \r\n4.Hypertension with LMD medication treatment\r\n5.Type diabetes mellitus-->current keep RI scale and LMD medication: \r\n  Glubin 500mg 1tab bidpc po for blood sugar control\r\n6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl";
        string note2 = "1.Left hip to left thigh swelling pain with limited of motion, because\n  of left femur intertrochanteric comminuted fracture with displaced s/p\n  ORIF with DHS in post-op 6th day\n2.Moderate swelling pain with limited of motion and skin ecchymosis over \n  left wrist area, due to left distal radius intraarticular commintued\n  fracture post ORIF with T-plate and left ulnar styloid fracture in \n  post-op 6th day-->keep short arm splint fixation\n3.Previous history of left breast cancer s/p MRM on 106/08/28 with \n  FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology, \n  RECOMMENDATION: It is appropriate now to discontinue the use of \n  letrozole. I will see her again at my OPD \n4.Hypertension with LMD medication treatment\n5.Type diabetes mellitus-->current keep RI scale and LMD medication: \n  Glubin 500mg 1tab bidpc po for blood sugar control\n6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl";

        string webtext = "1.Left hip to left thigh swelling pain with limited of motion, because  of left femur intertrochanteric comminuted fracture with displaced s/p  ORIF with DHS in post-op 6th day2.Moderate swelling pain with limited of motion and skin ecchymosis over   left wrist area, due to left distal radius intraarticular commintued  fracture post ORIF with T-plate and left ulnar styloid fracture in   post-op 6th day-->keep short arm splint fixation3.Previous history of left breast cancer s/p MRM on 106/08/28 with   FEMARA 2.5MG 1tab qd po, so 111/08/09 consultion Oncology,   RECOMMENDATION: It is appropriate now to discontinue the use of   letrozole. I will see her again at my OPD 4.Hypertension with LMD medication treatment5.Type diabetes mellitus-->current keep RI scale and LMD medication:   Glubin 500mg 1tab bidpc po for blood sugar control6.Anemia due to surgery blood loss, 111/08/13 recheck Hb: 10.8 gm/dl";
        string summary = "left ulnar styloid fracture";

        var noteResult = await CreateClient().InferICD10CMAsync(CreatRequest(note));
        noteResult.SaveTo($"R:\\awsNLP_NoteResult.json");

        var noteResult2 = await CreateClient().InferICD10CMAsync(CreatRequest(note2));
        noteResult2.SaveTo($"R:\\awsNLP_NoteResult2.json");

        var webtextResult = await CreateClient().InferICD10CMAsync(CreatRequest(webtext));
        webtextResult.SaveTo($"R:\\awsNLP_WebResult.json");

        var summaryResult = await CreateClient().InferICD10CMAsync(CreatRequest(summary));
        summaryResult.SaveTo($"R:\\awsNLP_summaryResult.json");

        IAmazonComprehendMedical CreateClient()
        {
            return new AmazonComprehendMedicalClient(
                new BasicAWSCredentials(accessKey, secretKey),
                new AmazonComprehendMedicalConfig { RegionEndpoint = RegionEndpoint.USEast2 });
        }

        InferICD10CMRequest CreatRequest(string context)
        {
            return new InferICD10CMRequest { Text = context };
        }
    }

note non-operated original text for SDK note2 replacement with \n for all \r\n webtext clearing all return characters

results

note NG results note2 results similar to Web/CLI score although ranking of translated diagnoses still a bit different. (Probably there are still unseen issues) webtext NG results

[ashishdhingra]

@f8y387512 Thanks for your detailed analysis. This doesn't appear to be SDK issue. Please advise if we could close this issue.

Thanks, Ashish

[github-actions[bot]]

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.