search

srdc / tofhir

Mapping toolset to migrate/transform existing datasets to HL7 FHIR
Apache License 2.0
9 stars 1 forks source link

Bug related to execution monitoring (EFK) #230

Open YemreGurses opened 3 weeks ago

YemreGurses commented 3 weeks ago

A similar error has been reported from multiple places on the EFK side. After running an execution, execution monitoring screen stucks like below: image

Screencast from 19-09-2024 02_17_19 ΜΜ.webm

We suspect duplicate records as the reason. This error can be received after running mapping with a dataset containing duplicate records.

Probably related to this: https://github.com/srdc/tofhir/issues/198

dogukan10 commented 2 days ago

First, I can confirm that duplicate records are not the cause of this issue. I tested several scenarios and different environments (Docker, IDE, etc.), but I was unable to reproduce the error.

Previously, we generated @timestamp using System.currentTimeMillis() as shown below:

markerMap.put("@timestamp", System.currentTimeMillis.toString)

This produced a number like 1729258250148, which could have been interpreted by Kibana as a string rather than a date. To address this, I’ve modified it to generate a properly formatted string like 2024-10-18T13:38:53.921Z, which should resolve the issue.

I’m keeping this ticket open for now, in case we encounter the problem again. If it does reoccur, we should retrieve the logs directly from Elasticsearch using its API and investigate the specific @timestamp values. If the issue doesn’t reappear, we can close the ticket.