search

electricitymaps / electricitymaps-contrib

A real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.43k stars 918 forks source link

Remove usage of arrow #6135

Open VIKTORVAV99 opened 8 months ago

VIKTORVAV99 commented 8 months ago

Description

The built in datetime class is faster and less error prone than arrow so we should migrate to fully using datetime instead of arrow in our parsers.

This probably needs to be done in steps so it's easier to review as it can have a big impact on how parsers are structured.

Parsers:

Barissa-Imran commented 8 months ago

Hi @VIKTORVAV99 , I would like to work on this.

VIKTORVAV99 commented 8 months ago

Hi @VIKTORVAV99 , I would like to work on this.

Sure thing!

We use arrow in a lot of places so just pick a parser/file and remove it. It should be replaced by datetime objects and I would prefer one PR per parser as it will allow for much faster reviews.

Barissa-Imran commented 8 months ago

I've compiled the following list to keep track of all parsers that have been refactored and one's the have not. The command I used is:

grep -El '(import arrow|arrow)' parsers/*.py

or the below to get files without arrow:

grep -L -E '(import arrow|arrow)' parsers/*.py

Edit by @VIKTORVAV99: The list has been moved to the top level issue comment so the tracking works correctly.

yujia21 commented 7 months ago

Hello, would like to help too! This is my first time contributing, started with the parser for my home country: https://github.com/electricitymaps/electricitymaps-contrib/pull/6246

VIKTORVAV99 commented 7 months ago

Hello, would like to help too! This is my first time contributing, started with the parser for my home country: #6246

Thats great, thanks!

gianantoniopini commented 5 months ago

Hi, I have just created PR #6481 to remove usage of the arrow module from the PE parser.

gianantoniopini commented 5 months ago

Hallo! I was planning to start trying to remove usage of the arrow module from the CL parser, unless somebody else is already working on it.

VIKTORVAV99 commented 5 months ago

Hallo! I was planning to start trying to remove usage of the arrow module from the CL parser, unless somebody else is already working on it.

I don't think anyone is working on that particular parser so go right ahead! And thanks!

gianantoniopini commented 5 months ago

Thanks!

nboswell216 commented 4 months ago

I tried to start on CA_BC this evening, but ran into an issue in attempting an initial run test_parser. Here is the full output: poetry run test_parser CA-BC Traceback (most recent call last): File "<string>", line 1, in <module> File "/workspaces/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 1134, in __call__ return self.main(*args, **kwargs) File "/workspaces/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 1059, in main rv = self.invoke(ctx) File "/workspaces/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 1401, in invoke return ctx.invoke(self.callback, **ctx.params) File "/workspaces/electricitymaps-contrib/.venv/lib/python3.10/site-packages/click/core.py", line 767, in invoke return __callback(*args, **kwargs) File "/workspaces/electricitymaps-contrib/test_parser.py", line 61, in test_parser parser: Callable[..., list[dict[str, Any]] | dict[str, Any]] = PARSER_KEY_TO_DICT[ KeyError: 'CA-BC'

I've tried all permutations of CA-BC with underscores and spaces, and still can't seem to find the key. The web app shows 'CA-BC', and the same zone key is listed multiple times in the parser code. Any idea on what might be causing this?

shuuji3 commented 4 months ago

@nboswell216 When running poetry run test_parser without the second DATA_TYPE option, it looks like it will be the default production. But CA_BC.py doesn't have a fetch_production() function so it failed to run. CA_BC.py instead defines fetch_consumption() so consumption option can be run:

> poetry run test_parser CA-BC consumption
Parser result:
[{'consumption': 7194.0,
  'datetime': datetime.datetime(2024, 2, 20, 5, 0, tzinfo=zoneinfo.ZoneInfo(key='America/Vancouver')),
  'source': 'bchydro.com',
  'sourceType': <EventSourceType.measured: 'measured'>,
  'zoneKey': 'CA-BC'},
lin0110 commented 3 months ago

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

VIKTORVAV99 commented 3 months ago

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

There are still a few parsers that need updating I think but if you want a before and after fix I would look at some of the merged PRs. There you should be able to see both the code before and after as well as the diff between the two.

lin0110 commented 3 months ago

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

There are still a few parsers that need updating I think but if you want a before and after fix I would look at some of the merged PRs. There you should be able to see both the code before and after as well as the diff between the two.

Thank you for clarifying! If no one is working on [parsers/IN.py], can I work on [parsers/IN.py]? Or is there any other suggested parser for beginner? :)

nboswell216 commented 3 months ago

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

There are still a few parsers that need updating I think but if you want a before and after fix I would look at some of the merged PRs. There you should be able to see both the code before and after as well as the diff between the two.

Thank you for clarifying! If no one is working on [parsers/IN.py], can I work on [parsers/IN.py]? Or is there any other suggested parser for beginner? :)

@lin0110 Any parser will be good for a beginner. If you get stuck, check out how the conversion has been handled in other parsers, especially in the same region. No need to ask to work on any of the parsers for this issue - if you don't see any open PR for a parser, just go for it!

lin0110 commented 3 months ago

Hi, just wondering if I can still contribute to this issue. I'm first time trying to contribute, very beginner level. Do I just need to change parser to date time object? Would it be possible to see any before fix version and after fix version?

There are still a few parsers that need updating I think but if you want a before and after fix I would look at some of the merged PRs. There you should be able to see both the code before and after as well as the diff between the two.

Thank you for clarifying! If no one is working on [parsers/IN.py], can I work on [parsers/IN.py]? Or is there any other suggested parser for beginner? :)

@lin0110 Any parser will be good for a beginner. If you get stuck, check out how the conversion has been handled in other parsers, especially in the same region. No need to ask to work on any of the parsers for this issue - if you don't see any open PR for a parser, just go for it!

Thank you guys for responding and encouraging me, I really appreciate it! I'm currently working on IN.py, but I'm encountering a ConnectTimeoutError while running 'poetry run test_parser IN'. How can I fix this so I can continue testing my code?