Open alixunderplatz opened 5 years ago
jarek
commented
5 years ago It would be great to ask them for figures per hour (and if possible in text format), if someone can.
If no other options come up, I don't mind setting up a proxy service that would save previous hours' values and calculate the most recent figure.
systemcatch
commented
5 years ago With some resizing I can get OCR to work.
chris@ThinkPad:~/Downloads$ convert image019.png -resize 100% ec_resized.png
chris@ThinkPad:~/Downloads$ tesseract ec_resized.png ec_read
It would be fun to use a proxy to make this a real time source, but trying to get a better format from them is probably the first step.
zonepilot
commented
5 years ago Hello I speak fluent Spanish having worked in S. America many years, I would be happy to ask some questions. As I am new to this, but find it very interesting, please can you help med explain how the data is presented and captured to work with the live map? Also, if you are in Ecuador, could you please find out contact details to whom the questions wouild be asked? BR Michael
systemcatch
commented
5 years ago Hi @zonepilot, thank you very much for offering to help with this.
Contact details are at http://www.cenace.org.ec/index.php?option=com_contact&view=contact&id=1:name&catid=12:contacts
What we would like is for the data in those images to be made available in a text format (json, csv, xml for example). We would also like it in an hourly format rather than cumulative for the entire day.
Simply put the map uses a number of parsers to read/scrape data from many different sources. That data is transformed into carbon intensities for every country which then colours the map in real time. There are some parsers (e.g. Chile (SIC)) that get data retrospectively which does not show on the live map.
I hope that makes sense but please ask any questions you need to.
zonepilot
commented
5 years ago OK, I have sent an inquiry asking for hourly data in text format, while referring to this project with links to website etc, let's see what comes out of it!,
BR
lorrieq
commented
5 years ago Hey guys, I had emailed them before in Spanish as well and finally got a response. Looks like @zonepilot's follow up email got the ball rolling. Thanks! My translation of the response is below:
"The CENACE maintains information on its portal that has the approval of the Dirección Ejecutiva. I do not see an inconvenience in having a space on the web page where one can download the requested information, once we have the authorization of the Dirección Ejecutiva. CENACE does not have hourly measurements but they could be obtained through estimates with a periodicity that we would have to analyse. Considering the request, we will communicate with the authorities and after their authorization the implementation of the requested information will be carried out."
It looks like they are happy to provide an area on the webpage where one can download the numbers in the image. I think the second paragraph wants to say they don't get the hourly values but they could estimate them by continuously comparing the totals they receive (as @jarek previously suggested we could do to work out hourly values).
How long this will all take remains to be seen.
po5i
commented
4 years ago Hey everyone! Any updates on this (at least from the CENACE side)? I'm currently living in Quito (where CENACE is located) and I'm willing to help.
systemcatch
commented
4 years ago Hey @po5i nothing yet, maybe you'd like to send a follow up email to the contact details in https://github.com/tmrowco/electricitymap-contrib/issues/1558#issuecomment-436667201
corradio
commented
4 years ago Any news here?
q--
commented
3 years ago By the way, I think this could be tagged "parser buildable". Even if doing the math with the combined generation to figure out the hourly generation isn't an option, it is still possible to measure the height of the various categories in the Curva de Generatción graph. (The downside being that otra generación isn't further broken down there, and it'll probably be less exact.)
Also, have you (Tomorrow) ever considered offering some way to parsers to read ElectricityMap-saved data from earlier in the day? That would probably be a lot easier than setting up a proxy like @jarek proposed.
Edit: also note that there are a few more screens on that web page (links at the bottom). One of them is as real-time demand ("DEMANDA TIEMPO REAL"), and this one has data for the current hour.
Edit 2: As an aside, when implementing the parser be careful: the "otra generación" at the left of the screen does not include renewables, but the stacked "otra generacion" to the right of the center does.
IV1T3
commented
2 years ago Found new sources, since all links above did not work anymore:
1) Generation capacities: https://www.celec.gob.ec/index.php/115-informacion-general/1770-generacion 2) Real-time production (?) per power unit including its capacity: https://potencia.celec.gob.ec/reportepotencia.html 3) Another potentially useful chart: https://potencia.celec.gob.ec/TRSEP.html
The title for 2) is "Power", whereas the title for 3) is "potencia entregada sector electrico petrolero" which translates to "power delivered in the oil and gas sector" according to DeepL. One might think that 3) is a subset of 2), however, this is not the case, since 3) also contains the unit "SNI" and thus has usually a higher "POTENCIA ACTIVA TOTAL" (MW).
SNI refers to the National Interconnected System: https://www.celec.gob.ec/transelectric/index.php/18-institucional/contenido-documentos/260-2017-09-05-22-40-46.
Update 1: Good presentation from 2017: https://www.celec.gob.ec/images/rendicion2017/InforRendCtas2017.pdf
The unit to type mapping is in progress...
IV1T3
commented
2 years ago Any idea how we could parse the data from link (2) mentioned above? The current production value is the number on top of each bar, as far as I have understood the interface.
Real-time production (?) per power unit including its capacity: https://potencia.celec.gob.ec/reportepotencia.html
jarek
commented
1 year ago https://potencia.celec.gob.ec/reportepotencia.html uses code and API that seems pretty hacky, but it works, and it should be easy enough to replicate.
The main chart ("potencia") seems to have MW data columns coming from a POST request to https://potencia.celec.gob.ec/GeneradorConsultas3.php with form data columna=KW, MVAR data columns coming from a POST request to same URL with form data columna=KVAR, and label coming from POST request with form data columna=DESCRIPCION. Data in the responses is a comma-separated list and order of the list is the same for each response, so the 6th element in DESCRIPCION maps to 6th element in KW and so on.
If the data seems reliable, we should be able to parse it fairly easily.
IV1T3
commented
1 year ago According to (1) https://www.celec.gob.ec/index.php/115-informacion-general/1770-generacion, the installed capacity of Ecuador is about 70% hydro, 30% oil and <1% wind. However, (2) https://potencia.celec.gob.ec/reportepotencia.html only reports thermal, i.e. oil, production - with the exception of Lumbaqui, which accounts for roughly 0.2% of current production. Given that 70% of the installed capacity is hydro, it seems suspicious that only 0.2% of the reported production comes from hydro. We should probably try to find other sources which also include more hydro production. Additionally, the currently reported total output (2) is 51.23MW, which seems fairly low...
Also found the new URL of the comprehensive dashboard which @alixunderplatz already mentioned back in 2018 but was moved: http://www.cenace.gob.ec/info-operativa/InformacionOperativa.htm.
alixunderplatz
commented
1 year ago Building a consumption-style parser for Ecuador 🇪🇨 is still possible from the available data!
This image shows the hourly demand as of 2 hours ago (and as of 3 hours ago just below): http://www.cenace.gob.ec/info-operativa/InformacionOperativa.htm under "DEMANDA TIEMPO REAL"
The demand-image is located at: http://www.cenace.gob.ec/info-operativa/InformacionOperativa_archivos/image099.png
Only downside: the need to use OCR / tesseract, but the content to read is simple and we managed worse image quality before 😄 I just can't get tesseract running properly on my own, so, someone go ahead and write this parser! =)))
VIKTORVAV99
commented
1 year ago Wouldn't a production parser also be possible with the data from http://[www.cenace.gob.ec/info-operativa/InformacionOperativa.htm](http://www.cenace.gob.ec/info-operativa/InformacionOperativa.htm)? It looks like it only has data for hydro and "unknown" but that should be more than adequate for adding it on the map. One problem might be to map it to the correct hour as I can't see any specific datetime on that page? Another problem might be that it seems to be accumulated production/consumption for the whole day?
alixunderplatz
commented
1 year ago @VIKTORVAV99 the datetime is specified by the little blue bar just under curva de generacion on produccion tiempo real. new level of OCR skills required :D as the graph is updated hourly, likely at a fixed minute-range, datetime should be easy to determine. For a production parser, we'd need to subtract the latest values of production from the value before. there is no way to do this without a workaround and caching the previous values (like, putting it in an invisible electricitymap zone and re-scraping it from there :P) which is needed to be subtracted from the latest value. See my first comment. If there is a way to do this with the help of your backend or an auxiliary JSON or sth like that which is hosted on your side, that would be great. (The dashboard probably will stay like this forever, it's been 4.5 years since I've opened this^^ )
VIKTORVAV99
commented
1 year ago @VIKTORVAV99 the datetime is specified by the little blue bar just under curva de generacion on produccion tiempo real. new level of OCR skills required :D as the graph is updated hourly, likely at a fixed minute-range, datetime should be easy to determine. For a production parser, we'd need to subtract the latest values of production from the value before. there is no way to do this without a workaround and caching the previous values (like, putting it in an invisible electricitymap zone and re-scraping it from there :P) which is needed to be subtracted from the latest value. See my first comment. If there is a way to do this with the help of your backend or an auxiliary JSON or sth like that which is hosted on your side, that would be great. (The dashboard probably will stay like this forever, it's been 4.5 years since I've opened this^^ )
This is going to be one complicated parser if we manage to do it 😅 Right now I don't think there is any way to store data between parser runs. I and @unitrium have been brainstorming a little on how we can improve the parsers and being able to save data between parser runs would solve a lot of issues and work arounds in several parsers. But the scope of saving data between parser runs is a lot bigger than what we discussed.
q--
commented
6 months ago I tried to find a "Curva de generación" with export on it, if someone ever tries to build a parser based on that graph.
Here's a graph that has it: https://web.archive.org/web/20230517214209/http://www.cenace.gob.ec/info-operativa/InformacionOperativa.htm
q--
commented
5 months ago Also worth noting: the tabs "INFORMACIÓN OPERATIVA DIARIA", "ACUMULADA MENSUAL" and "ACUMULADA ANUAL" also have a generation graph of yesterday or the day before (updated 13:00 local time). It uses less color gradients than the graph for the current day and therefore might be easier to work with.
q--
commented
5 months ago I did a bit of research into the VML tags I found in the web page source. As VML is a vector language (like SVG) I figured it might have the underlying data for the graphs in an easier to parse format, but unfortunately it seems most or all VML is the equivalent of an SVG with <image ... xlink:href="image037.png"/>, so just a reference to a PNG image wrapped inside a VML.
As an aside, I wrote this little piece of JavaScript to link all the images like https://www.cenace.gob.ec/info-operativa/InformacionOperativa_archivos/image002.png found on the server:
<script>addEventListener('DOMContentLoaded',e=>{var element = document.createElement('div');for(var i=1;i<270;i++){
element.innerHTML+=`<img src="https://www.cenace.gob.ec/info-operativa/InformacionOperativa_archivos/image${i.toString().padStart(3,0)}.png" alt=#${i} title=#${i}>`
};document.body.appendChild(element)})</script>The images often contain sub-sections of the larger images also found with these URLs.
Ugzuzg
commented
5 months ago I wrote a simple parser to get the data out for hydro, gas (taken from a stacked "OTRA GENERACION" bar chart), and unknown. However, there might be a problem with using image names directly to get the data out.
This comment states that the demand chart is image099.png. But if I open that image today, it only shows an icon for "DEMANDA CNEL". I assume, the numbers change depending on how many items are shown on "HIDRO" bar chart, or something similar.
There is generation data per type for Ecuador, which is updated hourly (around minutes :30 to :40). http://www.cenace.org.ec/docs/InformacionOperativa.htm
Two issues:
1.) OCR is required to read the images with generation figures (actually not really a problem)
2.) The data represents the cumulative generation per type and day. The generation of each hour is added to the previous generation for that day. After 24 hours, it is set to 0 again. This means, a difference has to be calculated to obtain the hourly generation in MW. I assume, this is impossible to use for the real-time map, but it could be possible to calculate the hourly generation in your database.
For 04:00:
For 05:00:
For 06:00
So the difference between these for hour 4-5 will be hydro: 1695 MW other: 340 MW (mix of oil and gas) renewable (non-conventional): 68 MW
A breakdown of the "other" fraction could be possible by using the figures of the "otra generacion" bar. gas: 85 MW ("gas natural") oil: 255 MW (I assumed that "calidad de servicio" goes in that category because of the generation shares on the IEA website.)
And for hour 5-6: hydro: 1720 MW other: 344 MW (of which gas: 100 MW and oil: 242 MW (minor mismatch of 2 MW)) renewable: 74 MW
Is it worth a try to read-out the values using OCR?
Here are the image-URLs:
hydro: http://www.cenace.org.ec/docs/InformacionOperativa_archivos/image036.png renewable: http://www.cenace.org.ec/docs/InformacionOperativa_archivos/image038.png other: http://www.cenace.org.ec/docs/InformacionOperativa_archivos/image037.png breakdown of other: http://www.cenace.org.ec/docs/InformacionOperativa_archivos/image015.png
I assume the "other" breakdown for oil/gas seperation will be hard to use OCR on, due to varying heights of the lower two categories. But I guess just having "other" will be enough.
Maybe someone with decent Spanish skills could ask them for the non-cumulative hourly values? That would make things easier :-)
Kind regards, Alex