Closed AayushSameerShah closed 1 year ago
Hello @AayushSameerShah ,
I'm here to assist you. There seems to be a small error in the pipeline. You have been working at the document level by providing "document" as input to your model, but you should be working at the sentence level by providing your sentences instead. Similarly, for the NerConverter, you should be using "sentence" as input instead of "document". I have prepared a pipeline using one of our license NER models and the results for your better understanding.
🔎 SAMPLE PIPELINE :
documentAssembler = DocumentAssembler()\
.setInputCol("text")\
.setOutputCol("document")
sentenceDetector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\
.setInputCols(["document"])\
.setOutputCol("sentence")
tokenizer = Tokenizer()\
.setInputCols("sentence")\
.setOutputCol("token")
tokenClassifier = MedicalBertForTokenClassifier.pretrained("bert_token_classifier_ner_jsl", "en", "clinical/models")\
.setInputCols(["token", "sentence"])\
.setOutputCol("ner")\
.setCaseSensitive(True)\
.setMaxSentenceLength(512)
ner_converter = NerConverterInternal()\
.setInputCols(["sentence","token","ner"])\
.setOutputCol("ner_chunk")
pipeline = Pipeline(stages=[
documentAssembler,
sentenceDetector,
tokenizer,
tokenClassifier,
ner_converter])
sample_text = """
Mr. ABC is a 60-year-old gentleman who had a markedly abnormal stress test earlier today in my office with severe chest pain after 5 minutes of exercise on the standard Bruce with horizontal ST depressions and moderate apical ischemia on stress imaging only. He required 3 sublingual nitroglycerin in total (please see also admission history and physical for full details).
The patient underwent cardiac catheterization with myself today which showed mild-to-moderate left main distal disease of 30%, moderate proximal LAD with a severe mid-LAD lesion of 99%, and a mid-left circumflex lesion of 80% with normal LV function and some mild luminal irregularities in the right coronary artery with some moderate stenosis seen in the mid to distal right PDA.
I discussed these results with the patient, and he had been relating to me that he was having rest anginal symptoms, as well as nocturnal anginal symptoms, and especially given the severity of the mid left anterior descending lesion, with a markedly abnormal stress test, I felt he was best suited for transfer for PCI. I discussed the case with Dr. X at Medical Center who has kindly accepted the patient in transfer.
CONDITION ON TRANSFER: Stable but guarded. The patient is pain-free at this time.
MEDICATIONS ON TRANSFER:
1. Aspirin 325 mg once a day.
2. Metoprolol 50 mg once a day, but we have had to hold it because of relative bradycardia which he apparently has a history of.
3. Nexium 40 mg once a day.
4. Zocor 40 mg once a day, and there is a fasting lipid profile pending at the time of this dictation. I see that his LDL was 136 on May 3, 2002.
5. Plavix 600 mg p.o. x1 which I am giving him tonight.
"""
df = spark.createDataFrame([[sample_text]]).toDF("text")
result = pipeline.fit(df).transform(df)
🔎RESULTS : 👇🏻
+--------------------------+-----+----+---------------------------+----------+
|chunk |begin|end |ner_label |confidence|
+--------------------------+-----+----+---------------------------+----------+
|60-year-old |14 |24 |Age |0.94577295|
|gentleman |26 |34 |Gender |0.9995837 |
|markedly abnormal |46 |62 |Test_Result |0.9741745 |
|stress test |64 |74 |Test |0.9996149 |
|today |84 |88 |RelativeDate |0.99974406|
|severe |108 |113 |Modifier |0.999809 |
|chest pain |115 |124 |Symptom |0.9965067 |
|after 5 minutes |126 |140 |RelativeTime |0.9583209 |
|horizontal ST depressions |181 |205 |EKG_Findings |0.9952421 |
|moderate |211 |218 |Modifier |0.9997716 |
|apical ischemia |220 |234 |Heart_Disease |0.99728715|
|stress imaging |239 |252 |Test |0.99985504|
|He |260 |261 |Gender |0.99992895|
|3 |272 |272 |Dosage |0.99897873|
|sublingual |274 |283 |Route |0.9997238 |
|nitroglycerin |285 |297 |Drug_Ingredient |0.9997858 |
|admission |325 |333 |Admission_Discharge |0.9990115 |
|cardiac catheterization |398 |420 |Procedure |0.9975387 |
|today |434 |438 |RelativeDate |0.9995082 |
|mild-to-moderate |453 |468 |Modifier |0.9996182 |
|left |470 |473 |Direction |0.96920437|
|main |475 |478 |Internal_organ_or_component|0.18351443|
|distal |480 |485 |Direction |0.9167719 |
|disease |487 |493 |Disease_Syndrome_Disorder |0.51986647|
|moderate |503 |510 |Modifier |0.99935615|
|proximal |512 |519 |Direction |0.9985382 |
|LAD |521 |523 |Internal_organ_or_component|0.9988713 |
|severe |532 |537 |Modifier |0.99965113|
|mid-LAD |539 |545 |Direction |0.9281402 |
|lesion |547 |552 |Symptom |0.9457182 |
|mid-left |568 |575 |Direction |0.6427324 |
|circumflex lesion |577 |593 |Symptom |0.9425712 |
|normal |607 |612 |Test_Result |0.7443403 |
|LV |614 |615 |Internal_organ_or_component|0.7672348 |
|function |617 |624 |Test |0.592596 |
|mild |635 |638 |Modifier |0.999859 |
|luminal irregularities |640 |661 |Symptom |0.9989077 |
|right |670 |674 |Direction |0.8462074 |
|coronary artery |676 |690 |Internal_organ_or_component|0.98542917|
|moderate |702 |709 |Modifier |0.99986166|
|stenosis |711 |718 |Disease_Syndrome_Disorder |0.66917396|
|mid |732 |734 |Direction |0.9996561 |
|distal |739 |744 |Direction |0.9997109 |
|right |746 |750 |Direction |0.9980184 |
|PDA |752 |754 |Internal_organ_or_component|0.9953032 |
|he |806 |807 |Gender |0.99994254|
|he |838 |839 |Gender |0.99993706|
|rest |852 |855 |Modifier |0.48786902|
|anginal symptoms |857 |872 |Symptom |0.9804443 |
|nocturnal |886 |894 |Modifier |0.8196311 |
|anginal symptoms |896 |911 |Symptom |0.9273863 |
|mid |955 |957 |Direction |0.98741364|
|left |959 |962 |Direction |0.86374986|
|anterior descending lesion|964 |989 |Symptom |0.87919575|
|markedly abnormal |999 |1015|Test_Result |0.9932798 |
|stress test |1017 |1027|Test |0.99903107|
|he |1037 |1038|Gender |0.9999273 |
|PCI |1073 |1075|Procedure |0.99704456|
|Medical Center |1113 |1126|Clinical_Dept |0.9993375 |
|CONDITION ON TRANSFER: |1178 |1199|Section_Header |0.89329183|
|pain-free |1236 |1244|Symptom |0.9997884 |
|MEDICATIONS ON TRANSFER: |1261 |1284|Section_Header |0.9833642 |
|Aspirin |1289 |1295|Drug_Ingredient |0.88113344|
|325 mg |1297 |1302|Strength |0.9997001 |
|once a day |1304 |1313|Frequency |0.9996945 |
|Metoprolol |1319 |1328|Drug_BrandName |0.6567626 |
|50 mg |1330 |1334|Strength |0.99977183|
|once a day |1336 |1345|Frequency |0.9996777 |
|relative |1386 |1393|Modifier |0.9833679 |
|bradycardia |1395 |1405|VS_Finding |0.94523096|
|he |1413 |1414|Gender |0.99992585|
|Nexium |1448 |1453|Drug_BrandName |0.998147 |
|40 mg |1455 |1459|Strength |0.9997608 |
|once a day |1461 |1470|Frequency |0.99971056|
|Zocor |1476 |1480|Drug_BrandName |0.9998432 |
|40 mg |1482 |1486|Strength |0.9996531 |
|once a day |1488 |1497|Frequency |0.9995396 |
|fasting lipid profile |1515 |1535|Test |0.99982995|
|his |1587 |1589|Gender |0.99994373|
|LDL was 136 |1591 |1601|LDL |0.8410518 |
|May |1606 |1608|Date |0.99915946|
|2002 |1613 |1616|Date |0.9991523 |
|Plavix |1622 |1627|Drug_BrandName |0.9998543 |
|600 mg |1629 |1634|Strength |0.99934137|
|p.o |1636 |1638|Frequency |0.9889002 |
|. |1639 |1639|Frequency |0.97397804|
|x1 |1641 |1642|Frequency |0.9974308 |
|him |1662 |1664|Gender |0.99980974|
|tonight |1666 |1672|RelativeDate |0.95218736|
+--------------------------+-----+----+---------------------------+----------+
➮ As you can see, our model detected a total of 89 entities in our sample text.
I hope these explanations and the pipeline have been helpful to you. Have a great day!☘️
Thank you very much for your amazing response @Damla-Gurbaz 🤗 I also have one query if you can have a look: My question #1052
I would really appreciate your response. It is clean and so comprehensive 💯 Thank you 🙏🏻
Discussed in https://github.com/JohnSnowLabs/spark-nlp-workshop/discussions/1028