search

electricitymaps / electricitymaps-contrib

A real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.46k stars 923 forks source link

Data for India #304

Closed corradio closed 2 years ago

corradio commented 7 years ago

Possible data source:

hectorespert commented 7 years ago

I found the demand and production of the State of Punjab, but to implement the parser we require split India in states.

I tried it but I don't found the India subunits.

corradio commented 7 years ago

That's a pretty cool milestone if we can start adding parts of India. If you want to look at states (and not subunits), you should try to add IND to https://github.com/tmrowco/electricitymap/blob/master/web/topogen.sh#L9, and then re-run that script. It will re-generate world.json, and another file, world_states_props.json (https://github.com/tmrowco/electricitymap/blob/master/web/topogen.sh#L35) that you can inspect in order to find the proper state. Just ping me if you have troubles, but this might help get you started.

jarek commented 7 years ago

Excellent catch @blackleg! It hasn't occured to me to check for data by state.

Looks like Gujarat also has data: https://www.sldcguj.com/RealTimeData/RealTimeDemand.php

Maharashtra as well, but I only found the image format so far: http://mahasldc.in/wp-content/themes/mahasldc/report2.html

Google searches for "(state_name) electricity generation" have found these for me and might find subsequent ones

systemcatch commented 7 years ago

Generation data I've found so far.

Andhra Pradesh http://www.core.ap.gov.in/CMDashBoard/UserInterface/Power/PowerReport.aspx

Capital Territory of Delhi http://www.delhisldc.org/Redirect.aspx?Loc=0804 Unsure if there is a breakdown by type.

Karnataka http://kptclsldc.com/StateGen.aspx The plants will need to be mapped to type.

jarek commented 7 years ago

I wonder how much power goes over interchanges... Karnataka http://kptclsldc.com/Snapshot.aspx looks to be about 10%?

We could really use someone well-versed in India electricity situation

systemcatch commented 7 years ago

Wikipedia has some interesting information. The states seem very well connected. https://en.wikipedia.org/wiki/Electricity_sector_in_India https://en.wikipedia.org/wiki/National_Grid_(India)

hectorespert commented 7 years ago

India installed capacity: http://www.cea.nic.in/reports/monthly/installedcapacity/2016/installed_capacity-03.pdf

prson commented 7 years ago

I am slightly aware of Indian power scenario and have been following this amazing initiative since sometime. For India, there is a lack of data sources and a lot of manual parsing has to be done. @blackleg the link you mentioned (http://www.cea.nic.in/reports/monthly/installedcapacity/2016/installed_capacity-03.pdf) is the best source for capacity. They release at the end of each month. More data is available as pdfs mostly. @systemcatch Unfortunately, wikipedia is mostly outdated.

acomets commented 6 years ago

Hi, I just found out about your initiative, great job! If I may chip in on a little research I did:

Capital Territory of Delhi I saw most of the power stations in the parser (https://github.com/tmrowco/electricitymap/blob/master/parsers/IN_DL.py) were missing fuel type delhisdldc provides us with a list of generation companies codes (https://www.delhisldc.org/psgencocodes.aspx) from which here is what I could get

Karnataka https://github.com/tmrowco/electricitymap/blob/master/parsers/IN_KA.py As for what CGS stands for, my guess would be Conventional/Coal Generating Stations, wdyt?

Keep up the good work!

systemcatch commented 6 years ago

@acomets thanks for doing research into those power stations, I think we should be able to improve our breakdown by type there. As for CGS I tend to think it means conventional generation systems while NCEP means non conventional electricity production.

anirudh-ramesh commented 6 years ago
  1. State Load Dispatch Centre of Gujarat (GJ) - the power plants indicated on the dashboard have a corresponding description in the following file (CSV): gujarat.txt
  2. State Load Dispatch Centre of Uttar Pradesh (UP) - this doesn't have a scrapable webpage, but the data could possibly be retrieved through an API.
  3. Western Regional Load Dispatch Centre - doesn't provide a breakdown of plants, so this can't be used to determine carbon emissions, but it provides a breakdown of states within the region, so the fuels for the generation can currently be marked as "unknown".
anirudh-ramesh commented 6 years ago

Folks, "CGS" stands for "Central Government Share". The logic is that the power plants are owned by one or more of three parties: The state government, the central (identical to federal) government, and/or private enterprises. Any amount of energy flowing through a transmission line operated and audited by the load dispatch centre (LDC) that was generated from one of the plants owned by the central government is labelled as CGS. @systemcatch @acomets @blackleg @jarek @corradio

systemcatch commented 6 years ago

@anirudh-ramesh Thank you for digging up these new sources. It's useful to know that CGS refers to ownership rather than generation type.

Gujarat should be ready to go based on what I see, do you want to have a go at writing the parser?

Uttar Pradesh cannot be scraped directly but watching the developer console I see POST requests that appear to be updating the data.

anirudh-ramesh commented 6 years ago

Yes, I've been moulding the GJ parser. It doesn't look very professional, so I'll need help refactoring after I have a fully functional version, @systemcatch.

corradio commented 6 years ago

@anirudh-ramesh the GJ parser has an issue: The date doesn't have a timezone, so it falls back to UTC, and therefore doesn't get shown in the UI Also, it would be nice to print out to the console all the units that do not match your list of units (so we can add them later on)

acomets commented 6 years ago

On a more general note, I was wondering if there was any way to account for off-grid/non-utility generation. Indeed, I found the following:

Much of the country does not have an electrical grid, so one of the first applications of solar power has been for water pumping, to begin replacing India's four to five million diesel powered water pumps, each consuming about 3.5 kilowatts, and off-grid lighting.

which represents 4-5 million x 3.5 kW ≈ 15 GW of capacity to be replaced.

anirudh-ramesh commented 6 years ago

@corradio, credits for the current version of the GJ parser to @sahitya-pavurala. I've replied to #1079 with details on IN-GJ.

anirudh-ramesh commented 6 years ago

@acomets, good point!

  1. Off-grid: Most of these small-scale off-grid deployments don't have a meter, so it's practically impossible to retrieve data from these.
  2. Non-utility: The parsers we have right now retrieve data from government-owned transmission line operators - there are private lines as well, and they all don't offer a public real-time data access interface - that's another problem to accurately quantify CO2 footprint. I'll look for these sources of information and the worst case is retrieving data from periodic reports.
  3. Solar PV in specific: Several hundred MW have been installed off-grid or on-grid, but several have been idle for reasons such as grid failure, no PPAs, poor maintenance, frequency mismatch etc (I've been studying these of late). I have access to info (hard copies) on who installed what, when and where - should that be of use.
brunolajoie commented 6 years ago

Yeah, offgrid is a very important topic indeed...There are two aims to the electricity map:

1/ Inform citizen/policymakers on their country's power carbon footprint, in which case offgrid is important to account for (but we don't have data to account for it)

2/ Inform electricity consumer of where the power he consumes comes from. Either he is offgrid and he knows it. Else he's connected to the power grid and off-grid does not impact its carbon footprint!

So, all in all, for the moment, let's continue adding on-grid values only until we find good, reliable data for off-grid

alixunderplatz commented 6 years ago

@brunolajoie For my master thesis, I have simulated power generation of PV systems based only on measured global horizontal irradiation (GHI) and ambient temperature using the package solaR for R, which can seperate GHI in direct and diffuse shares and does all the other related yield calculaltion, too. Don't know if something similar exists for python/JSON/... . The live solar potential layer (equal to GHI) is available, so there is a way to calculate estimated generation using date and time, geographic coordinates, and GHI. This could be used as indicator "Wgen/Winstalled" or yield factor in %, so users will see how much may be produced from their off-grid PV in that hour in their area.

brunolajoie commented 6 years ago

Wow nice! In the short term, estimations to fill live data gaps is not on the plan of electricity map, but lets keep that in mind. That could be added later on as adds on indeed

tmslaine commented 6 years ago

Some additional resources:

In addition Western Region Load Dispatch Centre has generator level real-time data: http://www.wrldc.in/onlinegenvsact.aspx Most of these are covered by existing parsers (once OCR for Maharasthra is done), however two of these generators are in Madhya Pradesh: Vindhyachal NTPC (coal 4760MW) and SASAN (coal 3960MW). The generation data for Western Region states on http://www.wrldc.in/onlinestate.aspx is inconsistent with generator level data. The last time I checked both of these coal power plants exceeded the generation listed for Madhya Pradesh.

alixunderplatz commented 6 years ago

Very good news! I found two wonderful maps with capacity data for all the Indian federal states :)

  1. This one is as of Nov 2017 (IP as a link ... ) http://172.104.62.103:8080/CAPACITY/capacity.html The breakdown is not the best, as it covers hydro, nuclear, thermal and renewables. image

  2. This dashboard has 10 GW more, so probably more recent data. https://npp.gov.in/dashBoard/cp-map-dashboard The breakdown also includes wind, biomass and solar :) VERY PROMISING!!! Select the colored larger areas first, then you can select the smaller states: image

EDIT: I had overlooked there is a thermal capacity breakdown in the orange stripe above the map :))))) https://npp.gov.in/dashBoard/cp-map-dashboard

image

alixunderplatz commented 6 years ago

The website of the national power portal https://npp.gov.in/ has some interesting links and apps.

Let's hope one day things like live generation will be centralized here in another dashboard! 🙏

alixunderplatz commented 6 years ago

Market prices for all areas hooray!

The displayed market prices are for the recent quarter hourly block (Don't get confused by the weird 30 minute offset!!! GMT+5:30) http://www.vidyutpravah.in/

image

edit: I just saw Oli added this link in his very first comment in this issue 👍

alixunderplatz commented 6 years ago

To keep the ball rolling:

On the website for marginal costs of generation per state http://meritindia.in/

I may have found a very exciting thing: Live generation for India as a whole:

At least I hope it is:

image

Unfortunately, the renewables section has no breakdown to wind, bio and solar :/ but the other numbers look very good! That thermal stuff is mainly coal, luckily gas is a seperate category.

The aggregated data is there, which means it's got to be somewhere else, too! Struggling to find it finally!

plz hlp! 😸

systemcatch commented 6 years ago

picard_clap

When you need something found, put @alixunderplatz on the case!!!!

alixunderplatz commented 6 years ago

For all the interconnectors, some grid maps of the Central Energy Authority of India may be useful: http://www.cea.nic.in/powermaps_hindi.html

alixunderplatz commented 6 years ago

I think it's worth writing a little script to collect data for some days or a week from this website http://meritindia.in/ to gather some info on the data presented in the table "ALL INDIA POWER POSITION (MW) [CURRENT]"

image

Although a higher spatial granularity for the data is desirable in the long run, I have some concerns this will be possible in the near future for all the states in India. The states that are currently featured are all lacking electricity import and export data (some could be added using the maps to determine interconnectors). And the complexity of the Indian power system is extremely high in general.

So I suggest checking this source out and possibly switching to an India-wide parser. The "Renewable Generation" field is the only con that I see, lacking a breakdown for wind, solar and biomass. "Thermal generation" is almost 100% coal, so no issues with that.

After collecting some data, we can see if assigning the "mixed renewables" to "unknown" or even minimizing the "unknown" by keeping solar, wind and biomass data from the other parsers is an option.

alixunderplatz commented 6 years ago

Here are the results of some manual tracking I did on Friday for http://meritindia.in.

I watched the website in irregular intervals and copied the figures once I had noticed some changes. This does not necessarily represent the interval the numbers are updated. I may have skipped a few datapoints, especially when not tracking it for longer intervals.

In general, the data quality is very good and precise! The update-intervall itself is a bit irregular and goes from around 2 min to maybe 5 to 10 min, but that's not an issue. I seperated the types in the graphs because of the great gap in the dimensions of "coal and demand" vs. rest.

As previously mentioned, everything that is considered "thermal" can be assigned to coal, which covers around 80% of the demand. Demand can vary by a few GW in a short interval of 2-4 min. During the period of tracking, it was somewhere between 140 to 160 GW. Time on the x-axis is middle european summer time, so add 3:30 h to get local time.

image

The green line for renewables (RES) is declining in the graph, because of solar component getting smaller and smaller towards the night as expected. Nuclear and gas were on constant levels. Hydro was peaking to cover the rising evening demand peak. Time on the x-axis is middle european summer time, so add 3:30 h to get local time.

image

Combined view of all types of generation:

image

And everything stacked (note: the time interval is a mess):

image

jarek commented 6 years ago

A couple of thoughts:

alixunderplatz commented 6 years ago

@jarek good points!

  1. to avoid a new category and still getting the "% renewable" right, what about making that "other renewables" type a "hidden export zone" with carbon intensity x and renewable share 100%. Like for the country specific export carbon intensity. It would be stacked on top in a bright green and could be labeled "Wind, Solar & Biomass" or "Renewables" (and get a special tooltip):

image

1.1 Or is there a way to collect all the wind/solar/biomass data as of now in a seperate parser and then apply it to the reduce the "other renewable" category?

  1. I agree, it's a pity loosing details for some regions. When I compare Karnataka's production with the "renewables" generation for India as a whole, around 1/5 of it is coming from there right now. I guess subtracting the output of a single state from the entire mix will be too complex and increase the risk to lose data due to errors?

  2. On the other hand, unless there will be a reliable dashboard by state showing imports, exports and all the other relevant info, there is no chance of getting every state totally right because of the lack of info for other states. Exchanges are not considered for any of the recently covered states in India. The state of Delhi generates about 900 MW, while the demand is beyond 4 000 MW, but we cannot really determine where that energy is coming from at how it is composed.

  3. Power plants belong either to "state sector", to "central sector" or to "IPPs". They are located in a certain state, but other states own and report some generation shares, too. There is a really good map to determine the location of every single power plant, although it's plenty of work. But it's very hard to find relevant interconnector data on state levels and to try to assign and combine all that info without double or triple accounting for something.

  4. If the Western/Northern/Southern/Eastern/North Eastern regional dispatch centres had reliable generation and interconnector real-time data for their areas, this was probably the "safest" granularity to go for at the moment. At least the real-time power flows between them are published here: http://wbs.nldc.in:82/Web_TTC_ATC.aspx , as was pointed out above (it's stuck at the moment^^).

alixunderplatz commented 6 years ago

I had a little script running in the last days which was collecting around 36 hours of data. Scrape-interval was 30 seconds. Mostly, the data would change each 2-3 minutes for all categories. Here are the results: In the area with dotted lines, I lost my internet connection at night. The peak on the right is an occasional data issue (coal + hydro).

Stacked view: image

Categories on their own: image

You can clearly see the green renewables curve peaking due to PV (between 6 and 7 UTC = 12:30-13:30 IST), and some RES generation increasing during night from wind power. There were low wind speeds recently, so not much was generated from it at night. As of March 2018, India has 21 GW solar, 34 GW of wind power and 8.7 GW of biomass capacity installed. I'm curious to see what it looks like when it's really windy.

Renewables image

120 GW of coal running constantly ... that's

OMG 😭 😞 🤢 🌡 🌏 📈

alixunderplatz commented 6 years ago

I just finished collecting data from meritindia.in, and here are the final results for the period that I observed in the last last 3 days. There were some more value peaks due to minor data quality issues yesterday, but nothing to really worry about. This happens everywhere from time to time and could be fixed by setting some limits to skip them. Other than that, the data is unbelievably smoooooooth and updates are very frequent.

Stacked:

image

Seperated:

image

@corradio and @brunolajoie What do you guys think about it? Any comments on the suggestions and thoughts from above? (These data cover about 1/6 of the recent world's population ;-) )

corradio commented 6 years ago

@alixunderplatz it seems like you found A LOT of things. Sorry for not having replied before. The website also seems to have breakdowns by regions also! I would favor continuying by region. Could you make a short summary of what needs to be done so we can more easily try to get people who want to contribute on the case?

alixunderplatz commented 6 years ago

@corradio What would you do with the "renewable" category (most likely consisting of wind & solar, maybe biomass)?

Unfortunately, the regional data on http://meritindia.in for India has no breakdown of types. It just gives "demand met", "own generation" as a total, and "import". When you go through all of the states, all show "import" because of the difference of "demand met" and "own generation". Would be nice to have export numbers included to calculate the balance, but that figure is totally missing everywhere. No idea if everything is included in "own generation" or if it is just a part (state owned, IPP, central generation).

image

This site has current demand and current price per state http://www.vidyutpravah.in/ http://vidyutpravah.in/state-data/gujarat

I'd first create a simple parser which collects the breakdown for India as a whole from http://meritindia.in:

image

No idea on how to continue or apply this to the other states, though. Is it possible to apply stripes with a color/transparent/color/transparent layer to the states are covered already?

image

corradio commented 6 years ago

Maybe the plantwise energy summary can be used to derive breakdown, but I realise it's going to be a lot of work: image

If we can't split wind/solar from biomass, then unfortunately the only solution is to put it in "unknown" with a decent carbon intensity based on the average yearly mix of that category.

alixunderplatz commented 6 years ago

@corradio the plantwise stuff looks like it's including plants that are outside of the respective states, too.

Check Dehli and expand the details below. http://meritindia.in/state-data/delhi For instance, some of the hydro power is located in Kashmir in the NW of India. See "capacity allocated to state" column for all where this is the case. Yes, assigning all units to the respective states and accounting for the correct amount of energy is muuuuch work. In addition, these numbers summarize the generated energy on a daily level for the previous day :-/

corradio commented 6 years ago

You are completely right. I've been looking at this too quickly.

On Fri 4 May 2018 at 18:20, Alex notifications@github.com wrote:

@corradio https://github.com/corradio the plantwise stuff looks like it's including plants that are outside of the respective states, too.

Check Dehli and expand the details below. http://meritindia.in/state-data/delhi For instance, some of the hydro power is located in Kashmir in the NW of India. See "capacity allocated to state" column for all where this is the case. Yes, assigning all units to the respective states and accounting for the correct amount of energy is muuuuch work. In addition, these numbers summarize the generated energy on a daily level for the previous day :-/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/304#issuecomment-386652225, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKLKGQ2XkeRd28y0YhJKwlPU_DcH8ks5tvH_BgaJpZM4LpACJ .

alixunderplatz commented 5 years ago

Live state-wide generation for West Bengal, no fuel type specified: http://www.wbsldc.in/ image

Live generation of WBPDCL, which owns [five coal power plants with 4.865 MW]:(https://en.wikipedia.org/wiki/List_of_power_stations_in_West_Bengal#Coal): https://www.wbpdcl.co.in/cgis/ image

No data on hydro or renewables,

systemcatch commented 5 years ago

@alixunderplatz maybe we could build a parser that just splits everything into coal and unknown?

anirudh-ramesh commented 5 years ago

@systemcatch and @alixunderplatz, for any region in India, we need to implement a merit-order-of-dispatch algorithm.

In Delhi, for example, we consider all plants located within Delhi and the ones elsewhere that have allocated a portion of power to the Delhi market. There is a legislation that power from renewable energy sources must not be curtailed when available i.e. substituted by a fossil fuel source regardless of cost. Next, we keep adding power from other fuel sources e.g. coal thermal or nuclear or hydroelectric power plants until the real-time demand is met.

Unless we have this algorithm in place, the models calculating carbon emissions intensity will be inaccurate. Right now, the Delhi parser only accounts for <700 MW generation of the >5000 MW production since we are not considering imports from power plants located elsewhere.

vidyakalai commented 4 years ago

Data on the generation of power from thermal stations in Tamil Nadu is present in https://www.tangedco.gov.in/thermal-stations.html Data on generation of power from renewable sources like hydro stations and gas-turbines : https://www.tangedco.gov.in/hydro-stations.html https://www.tangedco.gov.in/gas-turbine-stations.html

q-- commented 4 years ago

https://www.nerldc.org/ has, besides for its own area, also a demand figure for all of India (which is not rounded to GW as it is on http://vidyutpravah.in/)

Day-ahead electricity prices URL has moved to https://www.iexindia.com/maps.aspx

DineshCyanam commented 4 years ago

The base map for India is incorrect. Andhra Pradesh was split into two states in 2014. One being Andhra Pradesh (IN-AP) and the other is Telangana (IN-TG). The data we have here for IN-AP is for the new state of Andhra Pradesh.

IN-AP: http://vidyutpravah.in/state-data/andhra-pradesh IN-TG: http://vidyutpravah.in/state-data/telangana

jarek commented 4 years ago

See also #1097 and #234 regarding Telangana. Thanks for the confirmation that the data is only for the new borders of IN-AP!

willbeaufoy commented 3 years ago

Hi there, I'd like to contribute a parser. You do not yet have one for Himachal Pradesh, and I have found real time data. It does not list generation by source, but does list it by power plant (GENERATION OF HP(Z) table), and from googling them they are all hydropower (though I'm not sure yet what the last three in the list are). Is this an acceptable data source? If so I can build the parser.

FelixDQ commented 3 years ago

Hi guys. @willbeaufoy made a PR with data from Himachal Pradesh. It would be awesome if some of you have any input for the interpretation on the data. Check it out here: https://github.com/tmrowco/electricitymap-contrib/pull/2654

anselm94 commented 2 years ago

Hello everyone. Glad I found this interesting project.

For IN-TN (Tamil nadu), TNEB-SLDC doesn't seem to list this - http://tnebsldc.org/tnercreports.htm in their main site, but they publish daily generation/consumption for TN SLDC as a PDF here. I'm not an expert here. Will this data work?

Also, TNEB (a state company) publishes live generation data here - http://mail.tneb.in/gisapp/generationstationpub.jsp#7/10.596/79.058 and the corresponding JSON API - http://mail.tneb.in/geoserver/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=ssalert%3AGenerating_Station&outputFormat=application%2Fjson. With a quick read, I understand this list doesn't include private power companies.

Kongkille commented 2 years ago

Thanks for sharing these data sources, I'm adding it to our main issue for missing countries.

Feel free to comment on this issue if anything changes then we can reopen if we believe a parser can be built!