Add Japan

[corradio]

Seems like real-time data source is available here: http://agora.ex.nii.ac.jp/earthquake/201103-eastjapan/energy/electrical-japan/

[jarek]

From brief exploration on 2017-05-11, I didn't see a real-time generation breakdown by type.

From http://agora.ex.nii.ac.jp/earthquake/201103-eastjapan/energy/electrical-japan/stat/01/ (aggregate, not real time) it looks majority thermal/fossil.

http://agora.ex.nii.ac.jp/earthquake/201103-eastjapan/energy/electrical-japan/stat/25/ (again aggregate over past few years) says thermal gen is ~55% gas, 40% coal.

Maybe we can ask via Twitter if they know of real-time breakdowns: https://twitter.com/electricaljapan

[corradio]

Great. I'll ask from the Tomorrow account!

[corradio]

Twitter thread: https://twitter.com/electricalJapan/status/864240082764546048

[corradio]

I've sent an email to one of the professors at NIL.

[alixunderplatz]

I may have found a data source for Kyushu, the southern main island of Japan: It seems that data is mainly displayed in "ten thousands" kilowatts in the graphs and charts.

http://www.kyuden.co.jp/power_usages/pc.html Here is demand, solar generation and other lines (forecast?).

Some installed capacities for RES are here: http://www.kyuden.co.jp/effort_renewable-energy_application

I also managed to find some .csv-files that you can download, but we'd need somebody knowing the Japanese language to help us search that site for info for generation by type, exports and imports, etc...

There are many buttons that cannot be translated that easily, so I OCRed some and had them translated afterwards^^

Maybe @happysalada, who was involved in the Japanese translation, could help?

[happysalada]

ok so quick update on the page you mentioned there is a big button to download the csv of the day data. Next to it is a button with all past data. The only data given though is power consumption. Past data is given by day. A daily csv includes data by hour and by 5 minutes.

On their graph they show the percentage of power production that is solar, however that data didn't show up on the csv.

I looked on the site but really couldn't find stuff. I sent an email to them asking them if they had other data. (in japan you don't really ask for stuff without a reason, so I'm pretty sure that will seem suspect), but let's see what they say. (I remembered how I love japanese UX designer, their website is the worst I have seen in a while)

[alixunderplatz]

@happysalada Thank you very much for taking a look!!! 👍 Yeah, that website ... is entertaining 🤣 I think I found something that hasn't been there before!!

Here http://www.kyuden.co.jp/power_index is "live data" from nuclear, solar and wind.

SOLAR: http://www.kyuden.co.jp/megasolar-monitor/MegaSolar_Omuta.html

WIND: http://www.kyuden.co.jp/effort_renewable_windpower_nagashima.html

NUCLEAR (...which button is the right one? ^^): http://www.kyuden.co.jp/nuclear_real.html

[alixunderplatz]

https://www.occto.or.jp/index.html

That site seems to be from a transmission system coordinator. Go here http://occtonet.occto.or.jp/public/dfw/RP11/OCCTO/SD/LOGIN_login# and select the very last category on the left and then you'll see "CSV"

This will lead to another window where you can select dates and other checkboxes. Maybe there are some power flows between the regions?

[brunolajoie]

Very interesting! Given that we have live data for solar, nuclear and wind, PLUS daily CSV with 5-minute granular data for all fuels, it's probably worth parsing the CSV and add Kuyden to the map.

Usually, it's when we start adding a subset of a country that people start to react and help us make data become more and more transparent

[alixunderplatz]

I'm afraid the live solar and wind data on the seperate pages is for only 1 particular wind farm (~50 MW) and 1 particular solar farm (~3 MW) and not for total generation. 😭

[tmslaine]

Consumption data is available in a machine readable format on 5-minute granularity for all areas (Hokkaido, Tohoku&Niigata, Tokyo, Chubu, Hokuriku, Kansai, Chugoku, Kyushu and Okinawa), but there is a lag of up to one hour for some of these.

http://occtonet.occto.or.jp/public/dfw/RP11/OCCTO/SD/LOGIN_login# has the realized inter-area flows on 5-minute granularity and with seemingly no lag under 地域間連系線情報 / 連系線 潮流情報参照 / 連系線潮流実績 with possibility for csv download. The problem is writing a parser that can deal with the User Interface.

A number of nuclear power plants report their status and power output, but writing a parser to collect this data individually from every site would be very arduous. Other than that I think the real-time production data is pretty much limited to the solar farm and the wind farm mentioned above.

Assuming parsing consumption and flow data for each area succeeds, one can deduce total (unknown) production for the areas. http://www.enecho.meti.go.jp/statistics/electric_power/ep002/results.html has prefecture level generation statistics under 都道府県別発電実績 (xlsx形式:72KB). It should in principle be possible to guesstimate unknown generation structure in each area using that and for example https://en.wikipedia.org/wiki/Electricity_sector_in_Japan#/media/File:Power_Grid_of_Japan.svg as a reference.

[tmslaine]

At least history is available and published once a quarter: https://www.occto.or.jp/oshirase/sonotaoshirase/2016/170106_juyojisseki.html has links to hourly demand and supply by generation type from 2016/04 to 2017/12 for all areas

For Kansai there seems to be only solar and wind.

For Okinawa url has changed to: http://www.okiden.co.jp/business-support/service/supply-and-demand/index.html

[brunolajoie]

Ok, so we have found

• historical hourly data, hourly granularity, quarterly updated
• live exchanges data, 5 min granularity
• live consumption data, 5-minute granularity, hourly to sub-hourly updated

But not enough on real-time production per fuel to start writing parsers right now. Let's keep looking on that. Most def, given the complexity of Japan grid, it will not be solved in one issue: Feel free to create region-specific issues as soon as one of us find enough production data for one region to have it added on the map

[happysalada]

Kyuden came back with an answer, they pointed me to historical data that are updated every year http://www.kyuden.co.jp/wheeling_disclosure.html I asked them about real time data and other data sources in case they know it.

[alixunderplatz]

I found an Android app named "E-power" which links to the websites of all the regional Japanese electricity companies. In the play store, I was actually searching for "Kepco" which is actually a Korean utility, but luckily, I ended up finding that app.

Seems like disclosing info on load 📉 and nuclear output ☢️ are obligatory for all the EPCOs.

---

For the following links, first link is load, second is nuclear output (if available - for most regions still 0 MW a.t.m., see end of this comment).

Note: I just took the names suggested in the App, but they might differ from the website name.

Okinawa https://www.okiden.co.jp/denki/

For Kyushu-EPCO, see comments and links above. http://www.kyuden.co.jp/power_usages/pc.html http://www.kyuden.co.jp/php/nuclear/genkai/g_power.php (Genkai NPP) http://www.kyuden.co.jp/php/nuclear/sendai/s_power.php (Sendai / Kawachi NPP)

Shikoku-EPCO http://www.yonden.co.jp/denkiyoho/index.html http://www.yonden.co.jp/energy/atom/ikata/page_03.html

Chugoku-EPCO http://www.energia.co.jp/jukyuu/index.html http://www.energia.co.jp/atom/real_time/hatsuden/index.html

Kansai-EPCO (some nuclear plants running!) http://www.kepco.co.jp/energy_supply/supply/denkiyoho/index.html http://www.kepco.co.jp/energy_supply/energy/nuclear_power/info/monitor/live_unten/index.html

Hokuriku-EPCO http://www.rikuden.co.jp/denki-yoho/index.html http://atomic-monitoring.rikuden.co.jp/power?1519233523

Chubu-EPCO http://denki-yoho.chuden.jp/index.html

Tokyo-EPCO http://www.tepco.co.jp/forecast/index-j.html

Tohuku-EPCO http://setsuden.tohoku-epco.co.jp/graph.html http://www.tohoku-epco.co.jp/electr/genshi/jyokyo1/graph.html (Onagawa NPP) http://www.tohoku-epco.co.jp/electr/genshi/higashi/hd.html (Higashidori NPP)

Hokkaido-EPCO http://denkiyoho.hepco.co.jp/forecast.html http://www10.hepco.co.jp/gen_out_realtime.html

Note: All these diagrams show power output or demand in "ten thousands kW".

---

Here is a list of existing and operating NPPs ("restarted" after Fukushima): Seems like only five are in active operation:

http://www.fepc.or.jp/english/nuclear/power_generation/plants/index.html

[alixunderplatz]

FYI: All these "!" should actually show the load in these grid regions, but something seems to be broken in that app.

[tmslaine]

I think Occtonet has actually some useful information as under 地内基幹送電線関連情報 / 地内基幹送電線の現在潮流・潮流実績参照 / 地内基幹送電線潮流実績 there is live information on intra-area high-voltage lines with 30 min granularity. Some are just flows between substations that do not convey particularly useful info about generation but for example in the case of Okinawa there are flows for transformers of at least four thermal power plants: 吉の浦火力変電所(LNG), 金武火力変電所(coal), 具志川火力変電所(coal) and 石川火力変電所 (oil). Other areas have less specific info.

Occtonet also has grid maps for the intra-area transmission lines under 地内基幹送電線関連情報 / 地内基幹送電線の系統図・関連潮流参照 / 地内基幹送電線運用容量・予想潮流. These might help in figuring out flows from power plants.

[happysalada]

I asked the guys at kyuden if there were any data available appart from theirs. They said that there is an obligation under japanese law to publish production data (for each electricity producing company) every quarter. He said that data are uploaded at the begining of january, april, June, September on each of the electricity producing company homepage. Here is a list of all the homepage of the electricity companies http://www.tepco.co.jp/en/useful/links-e.html Here is the link of all the data for tepco http://www.tepco.co.jp/forecast/index-j.html The hourly data for 2018 for example http://www.tepco.co.jp/forecast/html/images/juyo-2018.csv they even have the 5 minutes data http://www.tepco.co.jp/forecast/html/images/juyo-j.csv (all these are updated every 5 minutes it seems, so real time consumption data is possible with tepco)

Looking at previous comment it seems that some of that data was clear. As an update, I couldn't find electricity production per source on tepco's site. I have just sent them an email asking if they have it somewhere.

[brunolajoie]

Thanks @happysalada, this is good news for total consumption/production data but unfortunately not broken down by fuel so no way to put a color for japan on the map with this at the moment

[gomatsuo]

In Japan, the electricity flow and the generation situation by fuel type are not disclosed in real time. Generators who have power plants with an output of 100 MW or more must enter stop / operation information in HJKS operated by the Japan Electric Power Exchange. https://hjks.jepx.or.jp/hjks/unit_status

The information published on this site is displayed every hour, but it is displayed by type of fuel.

[happysalada]

@gomto86 it seems there are dates for the futur on this graph. Is it a forecast ? also, do you know if it's possible to get the raw data somewhere ?

[tmslaine]

@happysalada I think that the linked site contains power plant availability information and planned outages for the next month. That in itself does not tell much about what the actual generation mix will be.

I have been testing OCR inspired by the Namibia parser for some of the nuclear power plants, seems to work:) Only the facilities listed in blue at http://www.genanshin.jp/facility/map/ need to be checked. Still leads to checking images at roughly 10 sources to get real-time nuclear production for all of Japan.

As for splitting Japan into control areas of the electric companies, Kyushu would be attractive as in addition to nuclear they report a real-time estimate for solar production. Solar and nuclear sometimes cover more than 50% of the consumption there, however based on occtonet the occasional export of 3 GW is quite large compared to the demand of ~10 GW.

For areas other than Kyushu I have not seen estimates of any production type for the whole area, but at least Japanese Wikipedia has detailed information on the geography of the control areas.

[alixunderplatz]

@tmslaine Japan looks great on the map!!! 👍

These are the sources for adding the remaining operating NPPs:

KANSAI: Oi (or Ohi or Ooi) NPP , lowest in the list: http://www.kepco.co.jp/energy_supply/energy/nuclear_power/info/monitor/live_unten/index.html with seperate gifs for unit 3 and unit 4 http://www.kepco.co.jp/interchange/nuclear_power/monitor/live_images/unten5.gif http://www.kepco.co.jp/interchange/nuclear_power/monitor/live_images/unten6.gif Same info here: http://www.kepco.co.jp/energy_supply/energy/nuclear_power/info/monitor/live_unten/ooi_real.html

Takahama NPP, currently under maintenance: http://www.kepco.co.jp/energy_supply/energy/nuclear_power/info/monitor/live_unten/index.html, center of the list with seperate gifs for unit 3 and unit 4 http://www.kepco.co.jp/interchange/nuclear_power/monitor/live_images/unten9.gif http://www.kepco.co.jp/interchange/nuclear_power/monitor/live_images/unten10.gif Same info here: http://www.kepco.co.jp/energy_supply/energy/nuclear_power/info/monitor/live_unten/takahama_real.html

SHIKOKU: Ikata NPP Looks like the only remaining unit 3 is planned to be shut down in September 2018. http://www.yonden.co.jp/energy/atom/ikata/ikt725.html I don't see a seperate figure in the graph for the actual output, but looks like we don't have to consider this one in the future anyways.

---

The other 4 operating units of the list are already included in Kyushu, right (Genkai 3+4 and Sendai 1+2)?

[brunolajoie]

Open question here: wouldn't it be interesting to add a fixed carbon intensity of "unknown" for Japan, in order to get at least the color right on the map while we look for live data for these remaining regions?

• For Kyushi, the unknown intensity would be equal to the annual average carbon intensity of the non-solar non-nuclear output?
• For all other region, the unknown intensity would be equal to that of "Japan minus Kyushi" annual average mix?

[tmslaine]

@corradio My assumption was that areas with only unknown generation would appear grey on eMap, like New Brunswick, Canada. What will happen if every generation type except unknown is removed from the dictionaries returned by the JP parser?

@brunolajoie From the pull request: Japanese Ministry of Economy, Trade and Industry has statistics for the fiscal year 2017 (Apr 2017-Mar 2018) for the national generation by fuel (Table 2-(1)) and generation by type in the 7 prefectures that make up Kyushu (Table 2-(2), rows 44-50) link

-From table 2-(1) I get 56% gas, 39% coal, 5% oil for fossil generation in Japan(626.7 gCO2eq/kWh). An alternative, openly available English source for this would be the BP statistical review link -From table 2-(2) I get 86% fossil, 3.3% bio, 8.5% hydro, 1.5% geothermal, 0.7% wind for the non-solar, non-nuclear generation in Kyushu. Thus the unknown CO2-intensity for Kyushu would be roughly 550gCO2eq/kWh.

[corradio]

Actually CA-NB is manually excluded. The conditions for having a color (i.e. having a carbon intensity) is just that there is production and import (with some exceptions for Bornholm and others).

Olivier

On Thu, Aug 16, 2018 at 6:17 PM tmslaine notifications@github.com wrote:

@corradio https://github.com/corradio My assumption was that areas with only unknown generation would appear grey on eMap, like New Brunswick, Canada. What will happen if every generation type except unknown is removed from the dictionaries returned by the JP parser?

@brunolajoie https://github.com/brunolajoie From the pull request: Japanese Ministry of Economy, Trade and Industry has statistics for the fiscal year 2017 (Apr 2017-Mar 2018) for the national generation by fuel (Table 2-(1)) and generation by type in the 7 prefectures that make up Kyushu (Table 2-(2), rows 44-50) link http://www.enecho.meti.go.jp/statistics/electric_power/ep002/results_archive.html

-From table 2-(1) I get 56% gas, 39% coal, 5% oil for fossil generation in Japan(626.7 gCO2eq/kWh). An alternative, openly available English source for this would be the BP statistical review link http://bp.com/statisticalreview -From table 2-(2) I get 86% fossil, 3.3% bio, 8.5% hydro, 1.5% geothermal, 0.7% wind for the non-solar, non-nuclear generation in Kyushu. Thus the unknown CO2-intensity for Kyushu would be roughly 550gCO2eq/kWh.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap-contrib/issues/541#issuecomment-413601812, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKCrdcWSpcVCpvjSgBlnrDuNKg3VRks5uRZsYgaJpZM4NXsnD .

[jarek]

IMO JP regions other than JP-KY should be excluded until we map nuclear production to region. Colouring based on 100% unknown from yearly breakdown is really pushing the purpose of Electricity Map.

[brunolajoie]

You're probably right

[corradio]

will do

On Thu, Aug 16, 2018 at 7:32 PM Bruno Lajoie notifications@github.com wrote:

You're probably right

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap-contrib/issues/541#issuecomment-413624664, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKAG1LSRl375UhTIFb67R2CFiBSAYks5uRay2gaJpZM4NXsnD .

[alixunderplatz]

We could also add Okinawa electricity demand data: https://www.okiden.co.jp/denki/ No breakdown of production types, this is what fuel mix looked like in 2014 (from http://www.tj.kyushu-u.ac.jp/leading/en/c_publication/EG2016-3-2content/Pages%2036-44.pdf):

The Okinawa Electric Power Company covers only Okinawa prefecture, the smallest supply region in Japan. Electric power generation was 8643 GWh in 2014. The ratio is coal: 65%, oil: 13%, liquefied natural gas (LNG): 18%, and others: 5%.

Historical 60-min and 5-min load data: https://www.okiden.co.jp/denki/dl/index.html

[alixunderplatz]

There is another source in English for Japan by ISEP for all zones with a historical breakdown per type :))))) https://isep-energychart.com/en/

It seems lik there only is historical data at the moment as well (going back to June 2018). It is possible to download .xls data :) !!!

https://isep-energychart.com/en/graphics/electricityproduction/?region=all&period_year=2016&period_month=4&period_day=1&period_length=1+week&display_format=residual_demand Select the "+" sign below the graphs to get more info about the displayed data.

In their blog, it seems like they got in touch with German Agora Energiewende (https://www.agora-energiewende.de/service/agorameter/) for some inspiration on the graphs. So there is hope we could see live-data with a breakdown in the future! :))) @corradio @brunolajoie you should contact them for more info! :) They're on twitter as well: https://twitter.com/isepjapan

Most interesting fact for me: The pumped storages are often charged during the day in summer, when there is a very high solar share (up to 35 GW)

---

Here's some more info on OCCTONET publishing data: https://isep-energychart.com/859/ (Not sure if there is any new content? It was mentioned in this issue previously that occtonet publishes quarterly data.)

---

Some inspiration I find interesting: they got the "residual load" view and the "baseload view" which is changing the position of renewables and fossil fuels in the diagrams. Maybe that's a possible feature for the electricitymap as well.

---

By the way, here is a production mix breakdown for 2014 for the control areas of Japan: https://www.researchgate.net/figure/Electricity-mix-in-2010-and-2014_fig2_318597636

It is also possible to obtain this from the historical data per region from the ISEP website to refine the unknown shares :) https://isep-energychart.com/en/graphics/piechart/?region=all&period_year=2018&period_month=6&period_day=1&period_length=12+months&display_format=demand_percentage

---

[systemcatch]

I can see data all the way back to May 2016, definitely worth scraping this.

[brunolajoie]

That's huge! Would be a very useful addition to the eMap ecosystem. Japan is such an important economy to add for comparison purpose to followup progress within the COP21 framework!

[alixunderplatz]

@brunolajoie Any news from ISEP or a reply to your tweet ?

[brunolajoie]

Nothing unfortunately!

On Wed, Feb 6, 2019 at 8:24 PM Alex notifications@github.com wrote:

@brunolajoie https://github.com/brunolajoie Any news from ISEP or a reply to your tweet ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap-contrib/issues/541#issuecomment-461153317, or mute the thread https://github.com/notifications/unsubscribe-auth/AVEnG18HjKUZo8CPwMWu7jn9IuxCLU5Gks5vKyvbgaJpZM4NXsnD .

[tmslaine]

Just noting here that real-time transparency has slightly improved lately in Western and Central Japan:

• Chubu region added solar 5-min data to their real-time consumption files (PR pending).
• Chugoku, Hokuriku and Tokyo regions have the solar production info on their website, latest value only. This is what we already had for Kyushu.
• Kansai region has solar production as graph on the website, but the raw numbers seem to be unavailable.

[alixunderplatz]

Hey @tmslaine We seem to experience issues with data for the following Japanese prefectures:

• Hokkaido
• Hokuriku
• Kansai
• Chugoko
• Shikoku The production isn't displayed in the diagram on the map anymore and the kibana logger shows this error: "Length mismatch: Expected axis has 6 elements, new values have 3 elements"

Should you have some time, could you please have a look at the issue?

[gomatsuo]

This is the real-time performance value for the Tokyo area. In Japan, real-time operation results of hydroelectric, thermal and nuclear power plants are not disclosed. Only real-time demand and photovoltaic output are published in real time. http://www.tepco.co.jp/forecast/

[gomatsuo]

Any data that is updated one month late can be downloaded from this link. http://www.tepco.co.jp/forecast/html/area_data-j.html

[alixunderplatz]

Hi @gomto86 and thank your for your comments. :) We already fetch the solar production of the Tokio prefecture as indicated in your first comment. To view the electricity production graph, you can select the Tokio area on the map. For the "unknown" production category, we use some other data by occtonet (regional load, flows) and make calculations with it. Best historical data overview for Japan is here: https://isep-energychart.com/en/graphics/electricityproduction/?region=all&period_year=2018&period_month=4&period_day=1&period_length=1+week&display_format=residual_demand

[gomatsuo]

I'm sorry. I overlooked the solar power performance in the Tokyo area. In the Hokkaido area, the actual solar power generation is displayed in the image. http://denkiyoho.hepco.co.jp/area_forecast.html

[gomatsuo]

In the Kansai area, the photovoltaic power output and demand are indicated by numbers. https://www.kepco.co.jp/energy_supply/supply/denkiyoho/index.html

[alixunderplatz]

Hi @gomatsuo :) Thank you for digging up this data 👍 We should be able to implement solar + nuclear for Kansai area :). Can you confirm the following website also displays the data for the three nuclear plants?: https://www.kepco.co.jp/energy_supply/energy/nuclear_power/info/monitor/live_unten/index.html

[gomatsuo]

[gomatsuo]

I'm sorry in poor English, but I can do this kind of cooperation.

[alixunderplatz]

@gomatsuo no worries, this is very helpful!!! :) 👍 👍 👍 Thank you!

[gomatsuo]

The other areas also mark the location of solar and nuclear power data in a similar manner.

This is the current demand and solar output of the Hokuriku area. http://www.rikuden.co.jp/denki-yoho/index.html

[gomatsuo]

The operation status of Hokuriku Electric Power's nuclear power plant. http://atomic-monitoring.rikuden.co.jp/power?1578225795

[Kongkille]

Hi guys, I've just written a parser for the Takahama plant and I end up with the total KW generated, but in which time period it is? My Japanese extent to the use of Google Translate, and for some reason everything on the page is pictures.

I can see that something at the top which might be the time, but having a hard time deciphering it and I want to be certain.

Can anyone help?

here is the website parsed (also referenced throughout the issue): https://www.kepco.co.jp/energy_supply/energy/nuclear_power/info/monitor/live_unten/

cc @gomatsuo @alixunderplatz @FelixDQ

[happysalada]

From the link you gave

• the takahama　（高浜） power plant is the second one from the top. (the first one being mihama (美浜））
• it has four parts (号機), each labeled with a power output
• the time is at a particular point in time. for example (when I checked) 7月24日12時30分　is july 24th 12:30
• it seems though that you will have to look at what is in the image on the right though. When I checked, it says that the output power is 87.0万kW (870,000kW) but on the image on the right it says it's operating at 104%, so you probably need to multiply one by the other.