Options for merging datasets

Open-Telecoms-Data / ofds_consolidation_tool

A tool to consolidate multiple data sets formatted using the Open Fibre Data Standard

https://open-telecoms-data.github.io/ofds_consolidation_tool/

GNU General Public License v3.0

1 stars 0 forks source link

Options for merging datasets #17

Closed rhiaro closed 3 months ago

rhiaro commented 8 months ago

Simplest is to ask the user to pick one to retain (all data from network A, or all data from network B, no merging/combining).

Set the option for the whole network, or allow user to choose per node/span?

More complicated, so determine if this is useful enough:

Fill in missing data from one network which is present in the other network (always? sometimes? depends on field? ask user?)
Merge fields which are amenable to being merged, eg. arrays (always? sometimes? depends? ask?)
How to combine fields which are single values? (depends? ask?)

rhiaro commented 8 months ago

Related: #10

R2ZER0 commented 7 months ago

I'm thinking, for a simple first implementation, of allowing the user to choose which node/span info to keep whenever they decide as duplicate? E.g. three buttons "Keep A" "Keep Both" "Keep B"

R2ZER0 commented 7 months ago

As for merging amenable fields i.e. arrays, I guess we could do that by default. Maybe have a checkbox to not?

rhiaro commented 7 months ago

Discussion with @R2ZER0 and @odscjen:

We should combine arrays in the consolidated output for:

networkProviders
technologies
transmissionMedium
internationalConnections

If we're saying this is one cable, these make sense to merge.