search

Baltic-RCC / EMF

Repository for Open Source EMF implementation
Mozilla Public License 2.0
8 stars 3 forks source link

Schedules to ELK document id selection #32

Closed m-karo closed 2 months ago

m-karo commented 6 months ago

Need to figure out how properly select document id in order to be able to overwrite document by each revision number.

Now id is generated from mRID, revisionNumber, TimeSeries.mRID and position. This is not good as revision number is included makes it different for each revision.

image

Haigutus commented 3 months ago

Options:

  1. Keep all PEVF version (intraday is updated each hour) - add version in the _id string
  2. Keep only latest version - remove version from _id string

In case 1, we can use document "created time" as filter to get latest version

Decision needed: To we keep all PEVF versions or not?

In option 1. we keep 2 versions of D-1 and 32 versions of ID In option 2. we keep only one copy od D-1 and ID

Haigutus commented 3 months ago

2024-03-14 Decision: Keep only latest version for now, change if real need arises.

TODO: Remove "revisionNumber" from the _id

Haigutus commented 3 months ago

Options:

  1. Keep all PEVF version (intraday is updated each hour) - add version in the _id string
  2. Keep only latest version - remove version from _id string (keeps data per curve)
  3. Keep data per MTU with unique ID

Options storage amount (per month):

  1. 9-10gb of storage
  2. 900MB
  3. 900MB

One file is estimated to take 10MB

We might need to separate indexes to weeks as we are storing too much data per one index

2024-03-25 Decision: Keep only latest version for now, change if real need arises. + Add to A03 parsing that for each MTU there would be a document.

For version 3. change is needed here - https://github.com/Baltic-RCC/EMF/blob/d90f8b5a882d3bae3a6aab6014805933e7ae0fb3/emf/common/converters/iec_schedule_to_ndjson.py#L124

VeikoAunapuu commented 2 months ago

IEC conversion to MTU implemented