electricitymaps / electricitymaps-contrib

A real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.48k stars 925 forks source link

Bulgaria is missing #890

Closed corradio closed 6 years ago

systemcatch commented 6 years ago

Belgium and Italy have also been missing for a while.

stgram commented 6 years ago

As far as I can see in the BG parser the values are obtained from tso.bg, which is no longer an active domain. Instead the Bulgarian Operator is currently housed under eso.bg, with the operational data provided here: http://www.eso.bg/?did=124 . Looking at the python code, the parser looks for a table that looks identical to the one in the active link, from which it extracts data to send to the website. In my opinion simply changing the url would fix the problem, but I am not competent in Python to verify that, if somebody could help that would be greatly appreciated.

systemcatch commented 6 years ago

Ah I thought Bulgaria was part of the ENTSOE parser but clearly it's separate.

@stgram I quickly tried the new url in the parser but it throws errors with the datetime. Looking at the page source the data is still in a table but certain parts have changed enough that altering the parser is needed. If you wanted to have a go at fixing it we could help out if you get stuck, shouldn't need any extra modules or more complex code.

stgram commented 6 years ago

@systemcatch Thank you for the response. Unfortunately I am not familiar with Py, I only understand the general concept of the code. Looking at the source of the page the datetime is on line 81:

<div class='dashboardDiv'>
<div class='dashboardCaptionDiv'>Генерация и товар на ЕЕС за <b>
04.12.2017 20:53:14  </b></div>
<center><small style='color:#555;'>

The parser used to look for a tag with the id lblTecDate, which does not exist anymore.

datetime = None
    local_time = soup.find(id='lblTecDate')
    if local_time:
        time = local_time.string.split(':')
        if len(time) == 3:
            datetime = arrow.now('Europe/Sofia').floor('day').\
                replace(hour=int(time[0]), minute=int(time[1]), second=int(time[2]))
    if not datetime:
        raise Exception('No datetime')

Instead would it be possible to fetch the content of the div tag, which includes the whole date & time information? I found this documentation on the issue, searching for the bulgarian text "Генерация и товар на ЕЕС за" in the tag should return the whole tag, from which the datetime could be extracted, or perhaps a regex expression for the datetime format? I cannot understand how the arrow commands work. Please help me with the issue, if you need assistance with the bulgarian language I am here to translate.

systemcatch commented 6 years ago

I would do the following to find the tag. The div class attribute is not unique so we also search by text. Then we find the b tag and get the text within.

div = soup.find('div', 'class' = 'dashboardCaptionDiv', text = 'Генерация и товар на ЕЕС за')
b = div.find('b')
time_string = b.contents

We can use the strptime method to convert the time string into a datetime object.

stgram commented 6 years ago

Could somebody try this? Sounds like a simple thing, lets get one more country on the map!