ENTSOE data retrieval for the Netherlands keeps failing due to negative values for "unknown"

[q--]

Kibana&_a=(columns:!(message,extra.key,level),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:level,negate:!f,params:(query:ERROR,type:phrase),type:phrase,value:ERROR),query:(match:(level:(query:ERROR,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:extra.key,negate:!f,params:(query:NL,type:phrase),type:phrase,value:NL),query:(match:(extra.key:(query:NL,type:phrase))))),index:'96f67170-0c49-11e9-85c1-1d63df8c862c',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc)))

The Netherlands have been gray most of the time for the last few days.

Values are roughly in the -100 to -250 range

[FelixDQ]

That is weird actually. When looking at the transparency platform the values doesn't seem to be negative. Am I looking at the wrong thing, or do we have a bug in the parser?

[q--]

I don't know, I'm just going by what's logged in Kibana.

I don't have an ENTSOE API key so I can't help with debugging

[jha749]

Would this PR fix this issue? Sounds like the PR will be merged soon. #2561

[q--]

Well, from quickly going through the code it seems https://github.com/tmrowco/electricitymap-contrib/pull/2561/files#diff-ddbeef7a3989d3ddde3446632507009dR152 will result in the parser returning nothing at all if unknown is zero or less. Which is even less helpful than having an error message like we do now, and should probably be changed before the PR is merged

So no, but it appears it will get rid of the error message.

[PaulCornelissen]

I've had a look at both the old parser and the new parser from #2561. With the ENTSO-E parser I do get the negative values, but not with the new parser. I've used a debugger to step through the ENTSO-E parser, but it's not directly clear why it's returning negative values for unknown. It seems to be a timing related issue, because when I step through the code with the debugger the problem doesn't appear, where as when I run the ENTSO-E parser regularly it does return negative values.

Btw. the problem doesn't appear with the new parser, because that uses the consumption, production and import values to calculate the unknown production if that results in a larger unknown value than the reported unknown value from the ENTSO-E parser.

[PaulCornelissen]

I've now done some more digging and this seems to be the cause:

Other Consumption is larger than Other Generation.

The ENTSOE parser substracts consumption from generation, so it reaches a negative value.

I will look further if I can find documentation what consumption of a specifiek generation type means and if the way the ENTSOE parser works is correct.

Edit: I've checked all other countries and no other country reports any significant consumption as actual generation.

[alixunderplatz]

@PaulCornelissen I've wrote a couple of lines regarding these negative "... consumption" values in https://github.com/tmrowco/electricitymap-contrib/issues/78#issuecomment-537269673 and https://github.com/tmrowco/electricitymap-contrib/issues/78#issuecomment-589267374 The magnitude of these values for something like a "self-consumption" of these plants is just too large, just like you said in comparison to the other countries. Usually its a few MW per unit when they are not running, nothing significant. No chance this will add up to so many MW in the NL. I assume these negative values should actually be included in the respective positive production categories, but due to some misreporting by the producers or whatsoever (probably just the negative sign...) they end up in consumption.

[q--]

I did some research into what's going on. Turns out 51,8% of the total generated electricity in the Netherlands is coming from CHP plants (2011)

Many of these plants are greenhouses. This site says greenhouses produce about 10% of electricity since 2009

Greenhouses also consume a lot of the electricity (light for growing plants). This 2019 report by a Dutch university on behalf of the government says the following:

Production

After 2016, production grew again to over 9 billion kWh in 2018. This was due to the increase in the useful life due to a more favorable market for the sale of electricity and an increase in the use of grow light. In addition to natural gas CHP, greenhouse horticulture produced a small amount of sustainable electricity with bio CHP and solar cells (Chapter 3). The total electricity production by greenhouse horticulture in 2018 provided 7.8% of the total Dutch electricity consumption. The total production per m2 of greenhouse in 2018 came to an average of approximately 105 kWh.

Sales

Sales declined from 8.4 to 5.0 billion kWh in the period 2010-2016. This decrease of almost 40% was mainly in the years 2010-2014. In 2018, sales grew to approximately 5.6 billion kWh due to a better spark spread for CHP.

Purchase

The purchase of electricity showed an upward trend in the period 2010-2017. Purchases amounted to approximately 2.2 billion kWh in 2010 and 2.8 billion kWh in 2017. This is an increase of almost 30%. This development is related to the increase in the area with grow light, but especially the intensification of grow light (W / m2). In order to make maximum use of heat from the CHP, not all the necessary electricity was generated ourselves, but partially purchased. Purchases remained the same in 2018.

Net supplier

Since 2006, more electricity has been sold per year than purchased and greenhouse horticulture is a net supplier of electricity. Net sales increased to 6.2 billion kWh in the period up to 2010 and decreased to 2.3 billion kWh in the period 2010-2016. This is a decrease of approximately 36%. In 2017, both sales and purchases increased. Sales also increased in 2018, but purchasing remained the same. On balance, net sales in 2018 were approximately 2.8 billion kWh.

Consumption

Electricity consumption (production plus purchasing minus sales) of the sector was estimated at 6.6 billion kWh in 2018. This is 5.5% of the national consumption. Consumption per m2 of greenhouse was an average of 74 kWh. Despite the shrinkage of the acreage in the period 2010-2018, the absolute electricity consumption of greenhouse horticulture grew so much that it almost doubled. Electricity consumption per m2 greenhouse grew by 127% in the period 2010-2018

So, if the electricity consumed by these greenhouses counts as 'self-consumption', then these figures might actually be plausible.

[alixunderplatz]

@q-- nice catch! sounds very legit at a first glance: take 6.6 TWh and divide it by 8760 hours, you get about 750 MW of average consumption. that coincides with what we see at the moment, and probably it increases in winter times? it just feels strange that this kind of self-consumption is disclosed on entsoe - is it included in "total load" figures of the country or just in the generation graphs?

[q--]

Not sure where I could find the answer to that on ENTSO-E, but if it helps, the Dutch TSO also reports production and consumption on its own website (no generation breakdown).

Production

Consumption

[PaulCornelissen]

Thanks @q--, that's a nice report. I just got an e-mail back from ENTSO-E basically confirming what you suspected (see #2676).

[PaulCornelissen]

@FelixDQ Since #2676 had been merged, this issue can be closed. I'll open a new issue about fixing the fetch_consumption.

[FelixDQ]

👍