User: anonymous is not authorized to perform: execute-api:Invoke on resource:

[RajeshEk]

Hi I am trying to run the application after the stack is created. I used the output of API link?bucket=mybucketname/notes/

{"Message":"User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-east-2:****3413:06z8h6u812/PROD/GET/translateCM with an explicit deny"} Can you please let me know if I am missing something? Or, I need to update the yml with any iam related things. Really appreciate your time and help

Thanks

[mwweiler]

Rajesh-

Thanks for sending this along...let me take a look and get back to you.

-M

[mwweiler]

During creation of the solution, you are asked to provide the IP address list that can access the endpoint. Can you confirm that you are accessing it from an IP address that is in this range?

[RajeshEk]

Hi Thanks for responding to my request. I got the IP address from https://ip-ranges.amazonaws.com/ip-ranges.json Since my S3 and application are running in us-east-2, I used one of the us-east-2 region IP address. I didn't give the range, only picked one. Also, when I tried to access the endpoint, I copied the link from Resources tab in CloudFormation and paste in the chrome browser and then added ?bucket=MYBUCKETINFO since it requires the clinical notes for the input. Please let me know what am I missing. Thanks a lot.

  "ip_prefix": "54.239.0.224/28",
  "region": "us-east-2",
  "service": "AMAZON",
  "network_border_group": "us-east-2"

Thanks -Rajesh

On Mon, May 4, 2020 at 4:56 PM mwweiler notifications@github.com wrote:

During creation of the solution, you are asked to provide the IP address list that can access the endpoint. Can you confirm that you are accessing it from an IP address that is in this range?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-623768612, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQIOFBPYAC5V3PEKB6LRP5I3TANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

[mwweiler]

Hi Rajesh- What is the IP address of the machine that you are using to access the URL....and I am assuming from your comment that it is an EC2 instance...what is the OS and instance type? I want to try and repro on your configuration.

Thanks, -Mark

[RajeshEk]

Hi Mark No, it is not from EC2. I am accessing the URL from my Mac 71.198.137.97. Mac OS ver 10.15.3.

Thanks -Rajesh

On Mon, May 4, 2020 at 6:15 PM mwweiler notifications@github.com wrote:

Hi Rajesh- What is the IP address of the machine that you are using to access the URL....and I am assuming from your comment that it is an EC2 instance...what is the OS and instance type? I want to try and repro on your configuration.

Thanks, -Mark

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-623789026, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQMBXQWVEORT4TARNTLRP5SBZANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

[mwweiler]

Hi Rajesh-

Ok, then THAT is the IP address that you need to provide when deploying the solution. The IP range that you specify in the CloudFormation template is the IP range for the machines that will be able to access the API, so it needs to include your machine. You can go into API Gateway, find the "translate-cm" API, and navigate to the "Resource Policy" section. You will see the section of the resource policy that specifies the allowed IP addresses. Change that value for the IP address for your machine and give it another try.

NOTE: While investigating your issue report, I found a different issue that I resolved. It would be worth deploying the updated code

Thanks, -Mark

[RajeshEk]

Hi Mark Oh, I got it. That is what AllowedIP mean. I thought it was the IP ranges for Amazon. Thanks a lot for looking into this. I will get the latest package and try out and let me know.

Thank You -Rajesh

On Tue, May 5, 2020 at 10:54 AM mwweiler notifications@github.com wrote:

Hi Rajesh-

Ok, then THAT is the IP address that you need to provide when deploying the solution. The IP range that you specify in the CloudFormation template is the IP range for the machines that will be able to access the API, so it needs to include your machine. You can go into API Gateway, find the "translate-cm" API, and navigate to the "Resource Policy" section. You will see the section of the resource policy that specifies the allowed IP addresses. Change that value for the IP address for your machine and give it another try.

NOTE: While investigating your issue report, I found a different issue that I resolved. It would be worth deploying the updated code

Thanks, -Mark

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-624210882, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQOT6VR23MXPCHTEETLRQBHGTANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

[mwweiler]

No problem...I apologize for not making that clearer. Yes...let me know how it goes. If you get connected I can close out this issue. And if you run into any other issues, let me know. Also...please check out the README, I provided updated instructions on running the solution (specifically around how to specify the query string parameters)

[RajeshEk]

Mark, Not a problem. I am happy that you are actively providing responding dev questions. I am also new to Cloud Formation that might be the source of problem. I will try it out and let you know. Will go over the README.

Thanks -Rajesh

On Tue, May 5, 2020 at 3:50 PM mwweiler notifications@github.com wrote:

No problem...I apologize for not making that clearer. Yes...let me know how it goes. If you get connected I can close out this issue. And if you run into any other issues, let me know. Also...please check out the README, I provided updated instructions on running the solution (specifically around how to specify the query string parameters)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-624347032, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQOXOMK3WDRBKCJNTT3RQCJ25ANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

[RajeshEk]

Hi Mark I still see the error with {"Message":"User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-east-2:****3413:xqfku5x9pg/PROD/GET/translateCM with an explicit deny"}

This is what I did.

1. Downloaded the updated version from GitHub.
2. Copied the new two zip files and YAML file to my S3 bucket.
3. Copied the Raw Clinical notes to MyS3BucketName/notes
4. Went to CloudFormation and created Stack using the following AllowedIP: MyLaptop IP Address Bucket: MyS3BucketName
5. Stack created successfully
6. Copied the endpoint and added the bucket info at the end https://cvj7tt7n3h.execute-api.us-east-2.amazonaws.com/PROD/translateCM https://cvj7tt7n3h.execute-api.us-east-2.amazonaws.com/PROD/translateCM?bucket=MyS3BucketName/notes
7. Got the error message
8. In Athena, I saw translatecm database and results table created

9. In S3, I saw these two folders created ashstack-resultsbucket-1brh63lhxnzi https://s3.console.aws.amazon.com/s3/buckets/ashstack-resultsbucket-1brh63lhxnzi?region=us-east-2

   This one is empty folder ashstack-resultsbucket-1brh63lhxnzi-athenaresults https://s3.console.aws.amazon.com/s3/buckets/ashstack-resultsbucket-1brh63lhxnzi-athenaresults?region=us-east-2

   This one had two files with 0 bytes each

Not sure, why it still thinks the user is anonymous. Please let me know if I am missing something

Thanks -Rajesh

On Tue, May 5, 2020 at 3:58 PM Rajesh Ekambaram ashramraj@gmail.com wrote:

Mark, Not a problem. I am happy that you are actively providing responding dev questions. I am also new to Cloud Formation that might be the source of problem. I will try it out and let you know. Will go over the README.

Thanks -Rajesh

On Tue, May 5, 2020 at 3:50 PM mwweiler notifications@github.com wrote:

No problem...I apologize for not making that clearer. Yes...let me know how it goes. If you get connected I can close out this issue. And if you run into any other issues, let me know. Also...please check out the README, I provided updated instructions on running the solution (specifically around how to specify the query string parameters)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-624347032, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQOXOMK3WDRBKCJNTT3RQCJ25ANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

-- Thanks, -Rajesh

[mwweiler]

Hey Rajesh:

Everything looks okay there, except for execution. You would want to run: "https://cvj7tt7n3h.execute-api.us-east-2.amazonaws.com/PROD/translateCM?bucket=MyS3BucketName&prefix=notes" This would run on everything in the "notes" subfolder. If you want to go recursively further into notes, you would want to add "&recursive" to the command above.

But...that is not what is causing your current issues. Can you use a site like http://ip4.me/ to confirm your public IP address? I want to make sure NAT is not causing issues here.

Thanks, -Mark

[RajeshEk]

Mark Ok, let me check the site and make sure my IP address is correct. I will let you know

Thanks -Rajesh

On Wed, May 6, 2020 at 11:05 AM mwweiler notifications@github.com wrote:

Hey Rajesh:

Everything looks okay there, except for execution. You would want to run: "https://cvj7tt7n3h.execute-api.us-east-2.amazonaws.com/PROD/translateCM ?bucket=MyS3BucketName&prefix=notes" This would run on everything in the "notes" subfolder. If you want to go recursively further into notes, you would want to add "&recursive" to the command above.

But...that is not what is causing your current issues. Can you use a site like http://ip4.me/ to confirm your public IP address? I want to make sure NAT is not causing issues here.

Thanks, -Mark

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-624803104, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQOFFO247KLR2OV2CI3RQGRIBANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

[RajeshEk]

Hi Mark I was able to make some progress after using the proper IP address. Thanks for pointing to NAT, it was the issue. However, now after invoking the API endpoint, it runs for half minute or so and return with an 'Internal Server Error'. These are the messages inside the txt files that get created on the S3 bucket.

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop database: InvalidOperationException(message:Database translatecm is not empty.))

Thanks -Rajesh

On Wed, May 6, 2020 at 11:05 AM mwweiler notifications@github.com wrote:

Hey Rajesh:

Everything looks okay there, except for execution. You would want to run: "https://cvj7tt7n3h.execute-api.us-east-2.amazonaws.com/PROD/translateCM ?bucket=MyS3BucketName&prefix=notes" This would run on everything in the "notes" subfolder. If you want to go recursively further into notes, you would want to add "&recursive" to the command above.

But...that is not what is causing your current issues. Can you use a site like http://ip4.me/ to confirm your public IP address? I want to make sure NAT is not causing issues here.

Thanks, -Mark

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-624803104, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQOFFO247KLR2OV2CI3RQGRIBANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

[mwweiler]

Hi Rajesh-

Can you take a look at the Cloudwatch logs for the translate-cm Lambda function and for the API Gateway and send me any errors that you see?

Thanks, -Mark

[RajeshEk]

Hi Mark It looks like there could be some issue with raw clinical notes. I am also attaching the notes 'file10.txt' where the failure happened. I will remove this file and try running with one file 'file1.txt and see if things go fine. This is what I saw

On the aws/lambda/Translate-ComprehendMedical

An error occurred (ThrottlingException) when calling the TranslateText operation (reached max retries: 4): Rate exceeded: ClientError Traceback (most recent call last): File "/var/task/lambda_function.py", line 35, in lambda_handler english = translate.translate_text(Text=textToTranslate, SourceLanguageCode="auto", TargetLanguageCode="en") File "/var/runtime/botocore/client.py", line 316, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 626, in _make_api_call raise error_class(parsed_response, operation_name) ClientError: An error occurred (ThrottlingException) when calling the TranslateText operation (reached max retries: 4): Rate exceeded

On API Side in Cloudwatch, this is error I see.

(8b52aca0-bacf-4140-8956-f87d489295c3) Endpoint response body before transformations: { "stackTrace": [ [ "/var/task/lambda_function.py", 35, "lambda_handler", "english = translate.translate_text(Text=textToTranslate, SourceLanguageCode=\"auto\", TargetLanguageCode=\"en\")" ], [ "/var/runtime/botocore/client.py", 316, "_api_call", "return self._make_api_call(operation_name, kwargs)" ], [ "/var/runtime/botocore/client.py", 626, "_make_api_call", "raise error_class(parsed_response, operation_name)" ] ], "errorType": "ClientError", "errorMessage": "An error occurred (ThrottlingException) when calling the TranslateText operation (reached max retries: 4): Rate exceeded" } (8b52aca0-bacf-4140-8956-f87d489295c3) Lambda execution failed with status 200 due to customer function error: An error occurred (ThrottlingException) when calling the TranslateText operation (reached max retries: 4): Rate exceeded. Lambda request id: 818585ab-e47e-4795-b5c7-e2febff8fee6 (8b52aca0-bacf-4140-8956-f87d489295c3) Method completed with status: 502

On Thu, May 7, 2020 at 7:04 AM mwweiler notifications@github.com wrote:

Hi Rajesh-

Can you take a look at the Cloudwatch logs for the translate-cm Lambda function and for the API Gateway and send me any errors that you see?

Thanks, -Mark

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-625275495, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQJDKB5MEVULHUPN4QTRQK5V5ANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

Sample Type / Medical Specialty: Gastroenterology Sample Name: Admission History & Physical - Nausea Description: Patient status post gastric bypass surgery, developed nausea and right upper quadrant pain. (Medical Transcription Sample Report) CHIEF COMPLAINT: Nausea.

PRESENT ILLNESS: The patient is a 28-year-old, who is status post gastric bypass surgery nearly one year ago. He has lost about 200 pounds and was otherwise doing well until yesterday evening around 7:00-8:00 when he developed nausea and right upper quadrant pain, which apparently wrapped around toward his right side and back. He feels like he was on it but has not done so. He has overall malaise and a low-grade temperature of 100.3. He denies any prior similar or lesser symptoms. His last normal bowel movement was yesterday. He denies any outright chills or blood per rectum.

PAST MEDICAL HISTORY: Significant for hypertension and morbid obesity, now resolved.

PAST SURGICAL HISTORY: Gastric bypass surgery in December 2007.

MEDICATIONS: Multivitamins and calcium.

ALLERGIES: None known.

FAMILY HISTORY: Positive for diabetes mellitus in his father, who is now deceased.

SOCIAL HISTORY: He denies tobacco or alcohol. He has what sounds like a data entry computer job.

REVIEW OF SYSTEMS: Otherwise negative.

PHYSICAL EXAMINATION: His temperature is 100.3, blood pressure 129/59, respirations 16, heart rate 84. He is drowsy, but easily arousable and appropriate with conversation. He is oriented to person, place, and situation. He is normocephalic, atraumatic. His sclerae are anicteric. His mucous membranes are somewhat tacky. His neck is supple and symmetric. His respirations are unlabored and clear. He has a regular rate and rhythm. His abdomen is soft. He has diffuse right upper quadrant tenderness, worse focally, but no rebound or guarding. He otherwise has no organomegaly, masses, or abdominal hernias evident. His extremities are symmetrical with no edema. His posterior tibial pulses are palpable and symmetric. He is grossly nonfocal neurologically.

STUDIES: His white blood cell count is 8.4 with 79 segs. His hematocrit is 41. His electrolytes are normal. His bilirubin is 2.8. His AST 349, ALT 186, alk-phos 138 and lipase is normal at 239.

ASSESSMENT: Choledocholithiasis, ? cholecystitis.

PLAN: He will be admitted and placed on IV antibiotics. We will get an ultrasound this morning. He will need his gallbladder out, probably with intraoperative cholangiogram. Hopefully, the stone will pass this way. Due to his anatomy, an ERCP would prove quite difficult if not impossible unless laparoscopic assisted. Dr. X will see him later this morning and discuss the plan further. The patient understands. Sample Type / Medical Specialty: Lab Medicine - Pathology Sample Name: Lung Biopsy Pathology Report Description: Lung, wedge biopsy right lower lobe and resection right upper lobe. Lymph node, biopsy level 2 and 4 and biopsy level 7 subcarinal. PET scan demonstrated a mass in the right upper lobe and also a mass in the right lower lobe, which were also identified by CT scan. (Medical Transcription Sample Report) CLINICAL HISTORY: A 48-year-old smoker found to have a right upper lobe mass on chest x-ray and is being evaluated for chest pain. PET scan demonstrated a mass in the right upper lobe and also a mass in the right lower lobe, which were also identified by CT scan. The lower lobe mass was approximately 1 cm in diameter and the upper lobe mass was 4 cm to 5 cm in diameter. The patient was referred for surgical treatment.

SPECIMEN: A. Lung, wedge biopsy right lower lobe B. Lung, resection right upper lobe C. Lymph node, biopsy level 2 and 4 D. Lymph node, biopsy level 7 subcarinal

FINAL DIAGNOSIS: A. Wedge biopsy of right lower lobe showing: Adenocarcinoma, Grade 2, Measuring 1 cm in diameter with invasion of the overlying pleura and with free resection margin. B. Right upper lobe lung resection showing: Adenocarcinoma, grade 2, measuring 4 cm in diameter with invasion of the overlying pleura and with free bronchial margin. Two (2) hilar lymph nodes with no metastatic tumor. C. Lymph node biopsy at level 2 and 4 showing seven (7) lymph nodes with anthracosis and no metastatic tumor. D. Lymph node biopsy, level 7 subcarinal showing (5) lymph nodes with anthracosis and no metastatic tumor.

COMMENT: The morphology of the tumor seen in both lobes is similar and we feel that the smaller tumor involving the right lower lobe is most likely secondary to transbronchial spread from the main tumor involving the right upper lobe. This suggestion is supported by the fact that no obvious vascular or lymphatic invasion is demonstrated and adjacent to the smaller tumor, there is isolated nests of tumor cells within the air spaces. Furthermore, immunoperoxidase stain for Ck-7, CK-20 and TTF are performed on both the right lower and right upper lobe nodule. The immunohistochemical results confirm the lung origin of both tumors and we feel that the tumor involving the right lower lobe is due to transbronchial spread from the larger tumor nodule involving the right upper lobe.

[mwweiler]

Rajesh...thank you for the additional information, I will take a look at it shortly. One comment...the solution that I put together marries Amazon Translate and Amazon Comprehend Medical to process clinical notes that are in languages other than English. While I imagine my solution would simply pass it through (didn't test English), you could try using Comprehend Medical out of the box to parse this note.

Thanks, -Mark

[mwweiler]

Rajesh-

How many notes are you attempting to pass through the solution?

Thanks, -Mark

[RajeshEk]

Mark, I had 10 files of English text. I didn't realize this was meant for other languages to English. I will try using Comprehend directly.

Thank you for your support, -Rajesh

On Thu, May 7, 2020 at 10:57 AM mwweiler notifications@github.com wrote:

Rajesh-

How many notes are you attempting to pass through the solution?

Thanks, -Mark

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-625407781, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQKLXJE5FBY5ID6I2SDRQLZCFANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

[mwweiler]

Hi Rajesh-

Of course, and thank you for exercising the solution, as the issues that you bumped into are going to help me to make it better. The throttling that you ran into was API throttling on the Translate side, so I need to check my code and make it more efficient in its calls.

As for running Comprehend Medical, please review the following blog written by one of my colleagues:

https://aws.amazon.com/blogs/machine-learning/extract-and-visualize-clinical-entities-using-amazon-comprehend-medical/

The solution there will likely be helpful for you.

Thanks, -Mark

[RajeshEk]

Hi Mark I will check out the blog. Really appreciate your support. Happy my testing helped to make package better.

Thanks -Rajesh

On Thu, May 7, 2020 at 1:19 PM mwweiler notifications@github.com wrote:

Hi Rajesh-

Of course, and thank you for exercising the solution, as the issues that you bumped in to are going to help me to make it better. The throttling that you run into was API throttling on the Translate side, so I need to check my code and make it more efficient in its calls.

As for running Comprehend Medical, please review the following blog written by one of my colleagues:

https://aws.amazon.com/blogs/machine-learning/extract-and-visualize-clinical-entities-using-amazon-comprehend-medical/

The solution there will likely be helpful for you.

Thanks, -Mark

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#issuecomment-625475973, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQNIUKH7FSFE5PB5DGTRQMJWVANCNFSM4MX62BIQ .

-- Thanks, -Rajesh

[mwweiler]

Rajesh- Just as an FYI (and for anyone else that comes through this issue in the future), you were getting throttled on the calls to Amazon Translate. As part of deploying this solution, you would likely need to use Service Quotas to request an increase. I will add this to the README for others.

Thanks, -Mark

[RajeshEk]

Hi Mark Thanks for the info. Just FYI, I was able to complete the e2e testing with just one clinical note in my folder.

Thanks for your help, -Rajesh

On Mon, May 11, 2020 at 5:00 PM mwweiler notifications@github.com wrote:

Closed #2 https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2 .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/amazon-translate-with-comprehend-medical/issues/2#event-3325214007, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKSAPQKEHRGKMQAUJNBFEU3RRCGTFANCNFSM4MX62BIQ .

-- Thanks, -Rajesh