search

GoogleCloudPlatform / healthcare-data-harmonization-dataflow

Apache License 2.0
35 stars 24 forks source link

Dataflow job fails with message exec.go:66 #27

Closed skysada closed 11 months ago

skysada commented 11 months ago

Hello,

I've been running this pipeline fine since April, but around July 4th, fatal errors occur which causes the worker nodes to fall over repeatedly.

I upgraded the pipeline with the latest commit so Dataflow SDK 2.47.0 is being used, and the issue persists.

It appears that Go is throwing errors, but it is unclear what the issue is from the logs. Here's a screenshot of the errors:

Screenshot 2023-07-10 at 4 07 41 PM

The message for each of these errors is exec.go:66.

Any assistance on this would be greatly appreciated.

skysada commented 11 months ago

This error happens when you have your HL7 store set on SOFT_FAIL and you send an unparseable message to it. It still saves the message but in place of the 'data' key in the json, it has an 'error' key. I would expect this to be handled much more gracefully, especially since the json messages retrieved from the HL7 Store are managed (only accessible via API). But at least others can refer to this and suffer less than I did :)