Adds a script to remove carbonDB duplicates

[ribalba]

I went for a KISS version which needs to be called via cron. As we will be reworking carbonDB soonish I didn't want to do something overly complex like copying DBs around. We can call this every n minutes. We could consider setting a flag in redis if something has changed but I think we should think about this bigger. Maybe we need a general queuing system like jobs. But more flexible. Not something I see vital for now though

[github-actions[bot]]

Old Energy Estimation

Eco-CI Output: |Label|🖥 avg. CPU utilization [%]|🔋 Total Energy [Joules]|🔌 avg. Power [Watts]|Duration [Seconds]| |---|---|---|---|---| |Total Run|22.9131|1568.94|3.48653|458| |Measurement #1|23.1277|1568.94|3.48653|450| 📈 Energy graph: ```bash 8.18 ┤ ╭───╮ 7.54 ┤ │ │ 6.90 ┤ ╭╮ ╭╮│ │ 6.26 ┤ ││ ╭╮ │││ │ 5.62 ┤ ││ ╭╯│ │╰╯ │ 4.97 ┤ ╭──╮ ││ │ ╰╮ │ │ 4.33 ┤ │ ╰╮ ╭╮ ╭╮ ╭╮ ╭╮ ╭╮ │╰╮╭───╮ ╭╯ ╰─╯ │ ╭╮ ╭╮ ╭╮ ╭─────╮ ╭╮ ╭ 3.69 ┤ ╭────────╯ ╰────╯╰────╯╰──────────────────────────────────────────────╯╰─╯╰───────────────────────╯╰─────────╯ ╰╯ ╰─╯ │ ╭───────╯╰─╮ ╭╮ ╭╮ ╭╯╰─╮ ╭╮ ╭╮ ╭─────╮╭───────────────────────────────────────╯╰─╯ ╰───╮ ╭──────────╮ ╭──╮ ╭──╮ ╭────╮ ╭──╮ ╭───────────────────────────────────────────────────╮ ╭──────────────────────────────────╯╰────────────────╯ 3.05 ┤ │ ╰╮ ╭╯ │ ╭╯╰╮ ││ │ │ │╰╮ ││ │ ││ │ │ │ │ │ │ ╰╮ ╭╯ │ │ │ │ │ │ 2.41 ┤ │ │ │ │ ╭╯ │ ││ │ ╰╮ ╭─╯ │ ││ │ ││ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 1.77 ┼────╯ ╰───╯ ╰───────╯ ╰────────╯╰─╯ ╰───────╯ ╰────────╯╰─╯ ╰╯ ╰─────────╯ ╰───────╯ ╰───────╯ ╰────────╯ ╰──╯ ╰─────────╯ ╰─╯ Watts over time ``` 🌳 CO2 Data: City: Boydton, Lat: 36.677696, Lon: -78.37471 Carbon Intensity for this location: 361 gCO₂eq/kWh SCI: 0.566387 gCO₂eq / pipeline run emitted

[ArneTR]

Update on this: The script cannot finish in it's current form. Taking longer than 12 hours ... then I stopped.

Might be due to the nature of the script or because of the resource limitation on the server.

[ribalba]

Renamed and changed the logic. Now the query takes 2 seconds and with the old DB full of duplicates 16 secs. So I think this is something we can run every hour so so.

[github-actions[bot]]

Eco-CI Output:	Label	🖥 avg. CPU utilization [%]	🔋 Total Energy [Joules]	🔌 avg. Power [Watts]	Duration [Seconds]
Total Run	22.3409	1671.2	3.46721	490
Measurement #1	22.4811	1671.2	3.46721	483

📈 Energy graph:

 8.18 ┤                                                                                                                                                        ╭──╮
 7.54 ┤                                                                                                                                                        │  │
 6.90 ┤                                                                                                                                                       ╭╯  │
 6.26 ┤                                                                                                                                                 ╭╮  ╭╮│   │
 5.62 ┤                                                                                                                                     ╭─╮       ╭╮│╰╮ │╰╯   ╰╮
 4.97 ┤                                                                                          ╭╮                                         │ │       │╰╯ │ │      │
 4.33 ┤                          ╭─╮                                                         ╭╮  ││  ╭╮                          ╭╮         │ │╭────╮ │   ╰╮│      │     ╭╮    ╭╮╭╮                           ╭╮                                                                   ╭╮     ╭╮  ╭─────╮                                                                                                                                                                                      ╭╮
 3.69 ┤    ╭──────╮╭─────────────╯ ╰───────╮  ╭╮  ╭──────────────────────────╮╭─╮  ╭─────────╯╰─╮│╰╮ │╰──────────╮╭──────────────╯╰─────────╯ ╰╯    ╰╮│    ╰╯      │    ╭╯╰────╯││╰─╮         ╭╮         ╭╮ ╭─╯╰─╮         ╭╮            ╭─────────────────────────────────────────╯╰────╮│╰──╯     ╰───╮         ╭────────────╮       ╭──╮       ╭──╮          ╭────╮  ╭──╮         ╭─────────────────────────────────────────────────────╮ ╭──────────────────────╮╭───╮╭────────────────╯╰───────╮╭──
 3.05 ┤    │      ││                       ╰╮╭╯│  │                          ││ │╭─╯            ╰╯ ╰─╯           ╰╯                                  ╰╯            │    │       ││  │        ╭╯│╭╮       ││ │    │         ││╭╮       ╭─╮│                                               ││             │         │            │      ╭╯  │       │  │         ╭╯    │  │  │         │                                                     │ │                      ││   ╰╯                         ╰╯
 2.41 ┤    │      ╰╯                        ││ ╰╮ │                          ╰╯ ││                                                                                 │    │       ││  │        │ │││       │╰╮│    ╰╮       ╭╯╰╯│       │ ││                                               ╰╯             ╰╮        │            │      │   │       │  │         │     │  │  │         │                                                     ╰╮│                      ╰╯
 1.77 ┼────╯                                ╰╯  ╰─╯                             ╰╯                                                                                 ╰────╯       ╰╯  ╰────────╯ ╰╯╰───────╯ ╰╯     ╰───────╯   ╰───────╯ ╰╯                                                               ╰────────╯            ╰──────╯   ╰───────╯  ╰─────────╯     ╰──╯  ╰─────────╯                                                      ╰╯
                                                                                                                                                                                                                                                Watts over time

🌳 CO2 Data: City: Chicago, Lat: 41.8819, Lon: -87.6278 Carbon Intensity for this location: 384 gCO₂eq/kWh SCI: 0.641741 gCO₂eq / pipeline run emitted

[ArneTR]

Looks good