search

google / fhir-data-pipes

A collection of tools for extracting FHIR resources and analytics services on top of that data.
https://google.github.io/fhir-data-pipes/
Apache License 2.0
154 stars 86 forks source link

Expected datetime: found "2024-12-30T16:01:05.978665" #1190

Open muhammad-levi opened 1 month ago

muhammad-levi commented 1 month ago

I do not know how to find the root cause(s), but it seems that some fields are required to be of type datetime when fhir-data-pipes tried to POST the transaction bundle into a Google Healthcare API FHIR Data Store.

The error log was:

07:24:27.092 [CHAIN DataSource (at Create.Values2/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources2/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources2/ParDo(Search)/ParMultiDo(Search).output) (2/2)#0] WARN  o.a.flink.runtime.taskmanager.Task org.apache.flink.runtime.taskmanager.Task.transitionState:1097 - CHAIN DataSource (at Create.Values2/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources2/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources2/ParDo(Search)/ParMultiDo(Search).output) (2/2)#0 (ee03a40d89eb96ab43031516685244d0) switched from RUNNING to FAILED with failure cause: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-12-30T16:01:05.978665"
cause: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-12-30T16:01:05.978665"
...
Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-12-30T16:01:05.978665"

Full stack trace:

2024-09-20 09:34:42 
2024-09-20 09:34:42   .   ____          _            __ _ _
2024-09-20 09:34:42  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
2024-09-20 09:34:42 ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
2024-09-20 09:34:42  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
2024-09-20 09:34:42   '  |____| .__|_| |_|_| |_\__, | / / / /
2024-09-20 09:34:42  =========|_|==============|___/=/_/_/_/
2024-09-20 09:34:42  :: Spring Boot ::                (v3.1.5)
2024-09-20 09:34:42 
2024-09-20 09:34:42 02:34:42.667 [main] INFO  c.g.f.a.ControlPanelApplication org.springframework.boot.StartupInfoLogger.logStarting:51 - Starting ControlPanelApplication using Java 17.0.6 with PID 8 (/app/controller-bundled.jar started by root in /app)
2024-09-20 09:34:42 02:34:42.670 [main] INFO  c.g.f.a.ControlPanelApplication org.springframework.boot.SpringApplication.logStartupProfileInfo:638 - No active profile set, falling back to 1 default profile: "default"
2024-09-20 09:34:43 Sep 20, 2024 2:34:43 AM org.apache.coyote.AbstractProtocol init
2024-09-20 09:34:43 INFO: Initializing ProtocolHandler ["http-nio-8080"]
2024-09-20 09:34:43 Sep 20, 2024 2:34:43 AM org.apache.catalina.core.StandardService startInternal
2024-09-20 09:34:43 INFO: Starting service [Tomcat]
2024-09-20 09:34:43 Sep 20, 2024 2:34:43 AM org.apache.catalina.core.StandardEngine startInternal
2024-09-20 09:34:43 INFO: Starting Servlet engine: [Apache Tomcat/10.1.15]
2024-09-20 09:34:44 Sep 20, 2024 2:34:44 AM org.apache.catalina.core.ApplicationContext log
2024-09-20 09:34:44 INFO: Initializing Spring embedded WebApplicationContext
2024-09-20 09:34:44 02:34:44.132 [main] WARN  o.s.boot.actuate.endpoint.EndpointId o.springframework.boot.actuate.endpoint.EndpointId.logWarning:155 - Endpoint ID 'pipeline-metrics' contains invalid characters, please migrate to a valid format.
2024-09-20 09:34:44 02:34:44.186 [main] WARN  o.s.c.LocalVariableTableParameterNameDiscoverer o.s.core.LocalVariableTableParameterNameDiscoverer.inspectClass:123 - Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: com.google.fhir.analytics.DataProperties$ConfigFields
2024-09-20 09:34:44 02:34:44.208 [main] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.validateProperties:138 - Using FHIR-search mode since dbConfig is not set.
2024-09-20 09:34:44 02:34:44.222 [main] INFO  c.g.f.analytics.FlinkConfiguration com.google.fhir.analytics.FlinkConfiguration.initialiseFlinkConfiguration:122 - Flink configuration will be initialised from the directory at /app/config
2024-09-20 09:34:44 02:34:44.290 [main] INFO  ca.uhn.fhir.util.VersionUtil ca.uhn.fhir.util.VersionUtil.initialize:84 - HAPI FHIR version 6.10.4 - Rev 5d6bb47853
2024-09-20 09:34:44 02:34:44.301 [main] INFO  ca.uhn.fhir.context.FhirContext ca.uhn.fhir.context.FhirContext.<init>:226 - Creating new FHIR context for FHIR version [R4]
2024-09-20 09:34:44 02:34:44.356 [main] INFO  c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:368 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-resources.xml
2024-09-20 09:34:44 02:34:44.389 [main] INFO  ca.uhn.fhir.util.XmlUtil ca.uhn.fhir.util.jar.DependencyLogImpl.logStaxImplementation:76 - FHIR XML procesing will use StAX implementation 'Woodstox' version '5.4.0'
2024-09-20 09:34:45 02:34:45.437 [main] INFO  c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:368 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-types.xml
2024-09-20 09:34:45 02:34:45.472 [main] INFO  c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:368 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-others.xml
2024-09-20 09:34:45 02:34:45.566 [main] INFO  c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:368 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/extension/extension-definitions.xml
2024-09-20 09:34:46 02:34:46.526 [main] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:34:47 02:34:47.519 [main] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:34:47 02:34:47.522 [main] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.testFhirConnection:324 - Validating FHIR connection
2024-09-20 09:34:48 02:34:48.562 [main] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.testFhirConnection:335 - Validating FHIR connection successful
2024-09-20 09:34:48 02:34:48.622 [main] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.getAllChildDirectories:378 - Child directories : [/dwh/controller_DWH_TIMESTAMP_2024_09_19T06_34_31_302626884Z/, /dwh/controller_DWH_TIMESTAMP_2024_09_19T03_03_51_972577779Z/]
2024-09-20 09:34:48 02:34:48.655 [main] INFO  c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.initDwhStatus:232 - Initializing with most recent DWH controller_DWH_TIMESTAMP_2024_09_19T06_34_31_302626884Z
2024-09-20 09:34:48 02:34:48.941 [main] WARN  o.s.c.LocalVariableTableParameterNameDiscoverer o.s.core.LocalVariableTableParameterNameDiscoverer.inspectClass:123 - Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: com.google.fhir.analytics.metrics.PipelineMetricsEndpoint
2024-09-20 09:34:49 Sep 20, 2024 2:34:49 AM org.apache.coyote.AbstractProtocol start
2024-09-20 09:34:49 INFO: Starting ProtocolHandler ["http-nio-8080"]
2024-09-20 09:34:49 02:34:49.113 [scheduling-1] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:119 - Last purge run was at null next run is at 2024-09-20T02:34:49.113309176
2024-09-20 09:34:49 02:34:49.113 [scheduling-1] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:121 - Purge run triggered at 2024-09-20T02:34:49.113862832
2024-09-20 09:34:49 02:34:49.114 [main] INFO  c.g.f.a.ControlPanelApplication org.springframework.boot.StartupInfoLogger.logStarted:57 - Started ControlPanelApplication in 7.082 seconds (process running for 7.985)
2024-09-20 09:34:49 02:34:49.144 [main] INFO  c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.onApplicationEvent:501 - createHiveResourceTables is false; skipping Hive table creation.
2024-09-20 09:34:49 02:34:49.166 [scheduling-1] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.getAllChildDirectories:378 - Child directories : [/dwh/controller_DWH_TIMESTAMP_2024_09_19T06_34_31_302626884Z/, /dwh/controller_DWH_TIMESTAMP_2024_09_19T03_03_51_972577779Z/]
2024-09-20 09:34:49 02:34:49.183 [scheduling-1] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:123 - Purge run completed at 2024-09-20T02:34:49.183598639
2024-09-20 09:34:49 02:34:49.184 [scheduling-1] INFO  c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:373 - Last run was at 2024-09-20T02:34:48.655980377 next run is at 2024-09-20T02:45
2024-09-20 09:35:01 Sep 20, 2024 2:35:01 AM org.apache.catalina.core.ApplicationContext log
2024-09-20 09:35:01 INFO: Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-09-20 09:35:01 02:35:01.637 [http-nio-8080-exec-1] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:35:01 02:35:01.661 [http-nio-8080-exec-1] INFO  c.g.fhir.analytics.MainController com.google.fhir.analytics.MainController.index:76 - Done serving /
2024-09-20 09:35:03 02:35:03.005 [http-nio-8080-exec-3] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:35:03 02:35:03.007 [http-nio-8080-exec-3] INFO  c.g.fhir.analytics.MainController com.google.fhir.analytics.MainController.index:76 - Done serving /
2024-09-20 09:35:06 02:35:06.629 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.ApiController com.google.fhir.analytics.ApiController.runBatch:66 - Received request to start the pipeline ...
2024-09-20 09:35:06 02:35:06.630 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:35:06 02:35:06.640 [http-nio-8080-exec-5] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:06 02:35:06.841 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.createSegments:221 - Fetching first batch of Flag
2024-09-20 09:35:06 02:35:06.996 [http-nio-8080-exec-5] INFO  com.google.fhir.analytics.FetchUtil c.u.f.rest.client.interceptor.LoggingInterceptor.interceptRequest:82 - Client request: GET https://fhir-server.dev.sid-indonesia.org/fhir/Flag?_count=100&_total=accurate&_summary=data HTTP/1.1
2024-09-20 09:35:07 02:35:07.417 [http-nio-8080-exec-5] INFO  com.google.fhir.analytics.FetchUtil c.u.f.rest.client.interceptor.LoggingInterceptor.interceptResponse:127 - Client response: HTTP 200 OK in 420ms
2024-09-20 09:35:07 02:35:07.495 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.createSegments:239 - Number of resources for Flag search is 177
2024-09-20 09:35:07 02:35:07.497 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.createSegments:255 - Total number of segments for resource Flag is 2
2024-09-20 09:35:08 02:35:08.648 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.runBatchPipeline:418 - Running full pipeline for DWH /dwh/controller_DWH_TIMESTAMP_2024_09_20T02_35_06_630731647Z
2024-09-20 09:35:08 02:35:08.648 [Thread-3] INFO  c.g.fhir.analytics.PipelineManager c.g.fhir.analytics.PipelineManager$PipelineThread.run:653 - Pipelines execution started with a new thread; number of threads is 20
2024-09-20 09:35:08 02:35:08.681 [pool-5-thread-1] WARN  o.a.beam.runners.flink.FlinkRunner org.apache.beam.runners.flink.FlinkRunner.run:93 - For maximum performance you should set the 'fasterCopy' option. See more at https://issues.apache.org/jira/browse/BEAM-11146
2024-09-20 09:35:10 02:35:10.577 [pool-5-thread-1] WARN  o.a.f.r.webmonitor.WebMonitorUtils o.a.f.r.webmonitor.WebMonitorUtils$LogFileLocation.find:82 - Log file environment variable 'log.file' is not set.
2024-09-20 09:35:10 02:35:10.577 [pool-5-thread-1] WARN  o.a.f.r.webmonitor.WebMonitorUtils o.a.f.r.webmonitor.WebMonitorUtils$LogFileLocation.find:88 - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'web.log.path'.
2024-09-20 09:35:10 02:35:10.664 [http-nio-8080-exec-8] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:35:10 02:35:10.668 [http-nio-8080-exec-8] INFO  c.g.fhir.analytics.MainController com.google.fhir.analytics.MainController.index:76 - Done serving /
2024-09-20 09:35:11 02:35:11.045 [pool-5-thread-1] INFO  c.g.f.a.metrics.FlinkJobListener c.google.fhir.analytics.metrics.FlinkJobListener.onJobSubmitted:37 - Submitting the job with ID com.google.fhir.analytics.metrics.FlinkJobListener@1344cf21 
2024-09-20 09:35:11 02:35:11.406 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (2/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.406 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.406 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (3/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.406 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (5/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.407 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (1/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.449 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] INFO  c.g.fhir.analytics.FetchResources com.google.fhir.analytics.FetchResources$SearchFn.processElement:138 - Fetching 100 resources for state Flag_main; URL= https://fhir-server.dev.sid-indonesia.org/fhir?_getpages=2e246ea1-ae96-46b6-a15a-44f485e6dea9&_getpagesoffset=100
2024-09-20 09:35:11 02:35:11.451 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (5/5)#0] INFO  c.g.fhir.analytics.FetchResources com.google.fhir.analytics.FetchResources$SearchFn.processElement:138 - Fetching 100 resources for state Flag_main; URL= https://fhir-server.dev.sid-indonesia.org/fhir?_getpages=2e246ea1-ae96-46b6-a15a-44f485e6dea9&_getpagesoffset=0
2024-09-20 09:35:12 02:35:12.438 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (5/5)#0] INFO  c.google.fhir.analytics.GcpStoreUtil com.google.fhir.analytics.GcpStoreUtil.uploadBundle:79 - Full URL is: https://healthcare.googleapis.com/v1/projects/nextgen-398301/locations/asia-southeast2/datasets/team-based-care.dev/fhirStores/team-based-care.dev.data-store/fhir
2024-09-20 09:35:12 02:35:12.438 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] INFO  c.google.fhir.analytics.GcpStoreUtil com.google.fhir.analytics.GcpStoreUtil.uploadBundle:79 - Full URL is: https://healthcare.googleapis.com/v1/projects/nextgen-398301/locations/asia-southeast2/datasets/team-based-care.dev/fhirStores/team-based-care.dev.data-store/fhir
2024-09-20 09:35:15 02:35:15.300 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] ERROR o.a.f.runtime.operators.BatchTask org.apache.flink.runtime.operators.BatchTask.logAndThrowException:1325 - Error in task code:  CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)
2024-09-20 09:35:15 org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.305 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] WARN  o.a.flink.runtime.taskmanager.Task org.apache.flink.runtime.taskmanager.Task.transitionState:1097 - CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0 (c4ac18626093cabc98c1ac4eab54d3ac) switched from RUNNING to FAILED with failure cause: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 
2024-09-20 09:35:15 02:35:15.601 [pool-5-thread-1] ERROR c.g.f.a.metrics.FlinkJobListener c.google.fhir.analytics.metrics.FlinkJobListener.onJobExecuted:49 - Error while executing the job
2024-09-20 09:35:15 org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2024-09-20 09:35:15     at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:300)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:297)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2024-09-20 09:35:15     at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
2024-09-20 09:35:15     at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2024-09-20 09:35:15     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
2024-09-20 09:35:15     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
2024-09-20 09:35:15     at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive(Actor.scala:537)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2024-09-20 09:35:15     at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
2024-09-20 09:35:15     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2024-09-20 09:35:15     at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2024-09-20 09:35:15     ... 5 common frames omitted
2024-09-20 09:35:15 Caused by: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.601 [pool-5-thread-1] ERROR o.a.beam.runners.flink.FlinkRunner org.apache.beam.runners.flink.FlinkRunner.run:106 - Pipeline execution failed
2024-09-20 09:35:15 org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2024-09-20 09:35:15     at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:300)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:297)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2024-09-20 09:35:15     at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
2024-09-20 09:35:15     at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2024-09-20 09:35:15     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
2024-09-20 09:35:15     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
2024-09-20 09:35:15     at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive(Actor.scala:537)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2024-09-20 09:35:15     at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
2024-09-20 09:35:15     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2024-09-20 09:35:15     at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2024-09-20 09:35:15     ... 5 common frames omitted
2024-09-20 09:35:15 Caused by: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.602 [Thread-3] ERROR com.google.fhir.analytics.EtlUtils com.google.fhir.analytics.EtlUtils.runMultiplePipelines:141 - Error while executing the pipelines
2024-09-20 09:35:15 java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:129)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelinesWithTimestamp(EtlUtils.java:78)
2024-09-20 09:35:15     at com.google.fhir.analytics.PipelineManager$PipelineThread.run(PipelineManager.java:667)
2024-09-20 09:35:15 Caused by: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:107)
2024-09-20 09:35:15     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
2024-09-20 09:35:15     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:163)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:153)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2024-09-20 09:35:15     at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:300)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:297)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2024-09-20 09:35:15     at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
2024-09-20 09:35:15     at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2024-09-20 09:35:15     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
2024-09-20 09:35:15     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
2024-09-20 09:35:15     at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive(Actor.scala:537)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2024-09-20 09:35:15     at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
2024-09-20 09:35:15     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2024-09-20 09:35:15     at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2024-09-20 09:35:15     ... 5 common frames omitted
2024-09-20 09:35:15 Caused by: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.602 [Thread-3] ERROR c.g.fhir.analytics.PipelineManager c.g.fhir.analytics.PipelineManager$PipelineThread.run:701 - exception while running pipeline: 
2024-09-20 09:35:15 java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:142)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelinesWithTimestamp(EtlUtils.java:78)
2024-09-20 09:35:15     at com.google.fhir.analytics.PipelineManager$PipelineThread.run(PipelineManager.java:667)
2024-09-20 09:35:15 Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:129)
2024-09-20 09:35:15     ... 2 common frames omitted
2024-09-20 09:35:15 Caused by: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:107)
2024-09-20 09:35:15     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
2024-09-20 09:35:15     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:163)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:153)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2024-09-20 09:35:15     at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:300)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:297)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2024-09-20 09:35:15     at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
2024-09-20 09:35:15     at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2024-09-20 09:35:15     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
2024-09-20 09:35:15     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
2024-09-20 09:35:15     at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive(Actor.scala:537)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2024-09-20 09:35:15     at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
2024-09-20 09:35:15     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2024-09-20 09:35:15     at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2024-09-20 09:35:15     ... 5 common frames omitted
2024-09-20 09:35:15 Caused by: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.692 [Thread-3] INFO  c.g.fhir.analytics.PipelineManager c.g.fhir.analytics.PipelineManager$PipelineThread.run:708 - Total time taken for the pipelines = 7 secs
bashir2 commented 1 month ago

Can you please provide a sample config (or command line if you are running the pipelines directly and not through the controller) to reproduce this? Also please provide the full stack-trace of the errors if possible.

muhammad-levi commented 1 month ago

@bashir2 After further investigation yesterday, it seems that our data values were not compliant with what FHIR spec described on dateTime as our datetime values were lacking time zone.

From the spec:

If hours and minutes are specified, a time zone SHALL be populated.