Closed etofi closed 1 year ago
ah ... that's a nastier one, I see: pvlib removed pvlib.forecast, which was used for converting cloud coverage to irradiance (in case you only have cloud coverage ...). Will need check how to work around that.
As a quick hack, I created a branch v2.11.02_dev: pvmodel.py has been updated to contain the part of the deprecated pvlib.forecast which is needed to maintain irradiance modelling from cloud coverage.
On a quick check, this seems to work, but more validation is needed: Forecasts for my system the coming days, using GHI (dc_disc) and cloud coverage (dc_clearsky_scaling, dc_campbell_norman)
So for me, the calculation runs again with the dev version - but it throws a lot of errors in the log. I don't know if they are relevant.
root@07847541f0b1:/pvforecast# python3 PVForecasts.py
------------------------- Start (config.ini at 2023-08-06, 07:01:08 - local)
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
Warning - New columns found in incoming data for table 'dwd' at 2023-08-06 03:00:00+00:00
ac_campbell_norman ac_clearsky ac_clearsky_scaling ac_dirindex ac_dirint ac_disc ac_erbs clouds dc_campbell_norman dc_clearsky dc_clearsky_scaling dc_dirindex dc_dirint dc_disc dc_erbs dhi_campbell_norman dhi_clearsky dhi_clearsky_scaling dhi_dirindex dhi_dirint dhi_disc dhi_erbs dni_campbell_norman dni_clearsky dni_clearsky_scaling dni_dirindex dni_dirint dni_disc dni_erbs ghi ghi_campbell_norman ghi_clearsky ghi_clearsky_scaling kt kt_disc kt_erbs pressure temp_air temp_dew wind_speed zenith
Message - IssueTime 2023-08-06 03:00:00+00:00 already exists in table 'dwd', no data to add to DB
Traceback (most recent call last):
File "/pvforecast/PVForecasts.py", line 34, in <module>
myForecastManager.runForecasts()
File "/pvforecast/PVForecast/forecast_manager.py", line 305, in runForecasts
if 'MOSMIX_L' in runList: self.processDWDFile('L') # gets latest forecast (MOSMIX_L - DWD)
File "/pvforecast/PVForecast/forecast_manager.py", line 101, in processDWDFile
myInflux.loadData(myWeather)
File "/pvforecast/PVForecast/influx.py", line 91, in loadData
client.write_points(df, data.SQLTable)
File "/usr/lib/python3/dist-packages/influxdb/_dataframe_client.py", line 133, in write_points
super(DataFrameClient, self).write_points(
File "/usr/lib/python3/dist-packages/influxdb/client.py", line 525, in write_points
return self._write_points(points=points,
File "/usr/lib/python3/dist-packages/influxdb/client.py", line 594, in _write_points
self.write(
File "/usr/lib/python3/dist-packages/influxdb/client.py", line 337, in write
self.request(
File "/usr/lib/python3/dist-packages/influxdb/client.py", line 302, in request
raise InfluxDBClientError(response.content, response.status_code)
influxdb.exceptions.InfluxDBClientError: 400: {"error":"partial write: unable to parse 'dwd dc_disc=139.61510214919252,dc_dirint=271.45501130046324,dc_dirindex=-inf,dc_erbs=65.94757820238189,dc_clearsky_scaling=109.84354456867581,dc_campbell_norman=233.2918671011652,dc_clearsky=141.88519960385793 1691431200000000000': invalid number dropped=0"}
Hmm ... that's a number of things. Let's unpack:
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
refers to the remark here
that the irradiation model dirindex
has numerical instabilities.
Warning - New columns found in incoming data for table 'dwd' at 2023-08-06 03:00:00+00:00
ac_campbell_norman ac_clearsky ac_clearsky_scaling ac_dirindex ac_dirint ac_disc ac_erbs clouds dc_campbell_norman dc_clearsky dc_clearsky_scaling dc_dirindex dc_dirint dc_disc dc_erbs dhi_campbell_norman dhi_clearsky dhi_clearsky_scaling dhi_dirindex dhi_dirint dhi_disc dhi_erbs dni_campbell_norman dni_clearsky dni_clearsky_scaling dni_dirindex dni_dirint dni_disc dni_erbs ghi ghi_campbell_norman ghi_clearsky ghi_clearsky_scaling kt kt_disc kt_erbs pressure temp_air temp_dew wind_speed zenith
You seem to have switched from no irradiance
statement in config.ini
(which then defaults to disc
) to all
. You are also storing into an SQLite database. As you increase the calculated irradiance
models, you create more columns. However, columns (fields) are not added dynamically.
Message - IssueTime 2023-08-06 03:00:00+00:00 already exists in table 'dwd', no data to add to DB
That's just a message to tell us that nothing changed in dwd
since the last call - normal if you call very often during debugging.
Traceback (most recent call last):
File "/pvforecast/PVForecasts.py", line 34, in <module>
myForecastManager.runForecasts()
File "/pvforecast/PVForecast/forecast_manager.py", line 305, in runForecasts
if 'MOSMIX_L' in runList: self.processDWDFile('L') # gets latest forecast (MOSMIX_L - DWD)
File "/pvforecast/PVForecast/forecast_manager.py", line 101, in processDWDFile
myInflux.loadData(myWeather)
File "/pvforecast/PVForecast/influx.py", line 91, in loadData
client.write_points(df, data.SQLTable)
File "/usr/lib/python3/dist-packages/influxdb/_dataframe_client.py", line 133, in write_points
super(DataFrameClient, self).write_points(
File "/usr/lib/python3/dist-packages/influxdb/client.py", line 525, in write_points
return self._write_points(points=points,
File "/usr/lib/python3/dist-packages/influxdb/client.py", line 594, in _write_points
self.write(
File "/usr/lib/python3/dist-packages/influxdb/client.py", line 337, in write
self.request(
File "/usr/lib/python3/dist-packages/influxdb/client.py", line 302, in request
raise InfluxDBClientError(response.content, response.status_code)
influxdb.exceptions.InfluxDBClientError: 400: {"error":"partial write: unable to parse 'dwd dc_disc=139.61510214919252,dc_dirint=271.45501130046324,dc_dirindex=-inf,dc_erbs=65.94757820238189,dc_clearsky_scaling=109.84354456867581,dc_campbell_norman=233.2918671011652,dc_clearsky=141.88519960385793 1691431200000000000': invalid number dropped=0"}
Here we have a problem writing to Influx
... my assumption is that dc_dirindex=-inf
is the culprit. But then, this should be handled gracefully by the influx library. I'll look into this.
Stefan
I can't reproduce the issue with partial write
, but I find this link. I've the current version 5.3.1 installed - This issue seems to be closed for 5.3.0.
Let me know if this is the problem - then I'll update requirements.txt
accordingly.
python3 -m pip show influxdb
Name: influxdb
Version: 5.3.1
Summary: InfluxDB client
Home-page: https://github.com/influxdb/influxdb-python
Author:
Author-email:
License: MIT License
Location: /home/eichenbe/.local/lib/python3.10/site-packages
Requires: msgpack, python-dateutil, pytz, requests, six
Required-by:
After creating the fields in the SQLite DB and updating the InfluxDB client to version 5.3.1, I get the following output. I'll leave out the umpteen messages of this kind for now.
/usr/local/lib/python3.9/dist-packages/pvlib/singlediode.py:763: RuntimeWarning: overflow encountered in exp
np.exp((Rs[idx_p] * (IL[idx_p] + I0[idx_p]) + V[idx_p]) /
These are then the remaining messages:
Message - IssueTime 2023-08-06 09:00:00+00:00 already exists in table 'dwd', no data to add to DB
Message - downloading SolCast data at (UTC): 2023-08-06 10:28:38.612652+00:00
Traceback (most recent call last):
File "/pvforecast/PVForecasts.py", line 34, in <module>
myForecastManager.runForecasts()
File "/pvforecast/PVForecast/forecast_manager.py", line 307, in runForecasts
if 'SolCast' in runList: self.processSolCast() # get / post solcast
File "/pvforecast/PVForecast/forecast_manager.py", line 116, in processSolCast
mySolCast.getSolCast()
File "/pvforecast/PVForecast/solcast.py", line 169, in getSolCast
if self.config['SolCast'].getboolean('post', 0):
File "/usr/lib/python3.9/configparser.py", line 1303, in get
return _impl(self._name, option, raw=raw, vars=vars,
File "/usr/lib/python3.9/configparser.py", line 828, in getboolean
return self._get_conv(section, option, self._convert_to_boolean,
File "/usr/lib/python3.9/configparser.py", line 808, in _get_conv
return self._get(section, conv, option, raw=raw, vars=vars,
File "/usr/lib/python3.9/configparser.py", line 803, in _get
return conv(self.get(section, option, **kwargs))
File "/usr/lib/python3.9/configparser.py", line 1163, in _convert_to_boolean
raise ValueError('Not a boolean: %s' % value)
ValueError: Not a boolean: 1
interval = 30
The latter seems to be an error in the config.ini
? It looks to me that somehow 1 interval = 30
is picked up as a value?
(That said: you seem to try to set post = 1
- but posting is no longer supported by SolCast (they will silently drop data submitted)
I can reproduce your error by writing the following into config.ini
:
[SolCast]
post = 1
interval = 30
See first answer here about deeper indenting
I uploaded a new pvmodel.py
to the development branch, which aims at catching and suppressing the RuntimeWarnings. Not sure it works, since I don't get the warnings here right now (they depend on the DWD input data)
I still have the RuntimeWarnings. Unfortunately, the update did not help. I'll check the ini to see if I can find the excess space.
hmm ... can you write out the results to .csv and share the .csv? Then I can run on your data ...
storeCSV = 1
storePath = ./temp/ # assuming ./temp exists
dropWeather = 0
Enclosed are the two CSV files MOSMIX_L_2023080609_10738.csv.gz solcast_2023-08-06_14-30.csv.gz
... thx, etofi.
Getting a little desperate here. Still can't reproduce it. I know I've seen the same in the past, but can't get it even running my old input data of the last couple of years ...
If you don't mind copying [PVSystem]
section of your config file here (not other sections as they may contain API keys) - this is kind of my last hope ...
Last time I saw these messages (but just few every now and then ...) was in 2021 when I was running Python 3.7, pvlib 0.8.1 ... Rerunning that old data no longer produces the error here ... difference could be numpy version (1.24.1 here)
If you don't mind copying
[PVSystem]
section of your config file here (not other sections as they may contain API keys) - this is kind of my last hope ...
[PVSystem] # PV system to be modeled (for DWD, OpenWeatherMap based forecasts)
Model = CEC # modeling strategy for PV: 'PVWatts' or 'CEC'
TemperatureModel = open_rack_glass_glass # https://pvlib-python.readthedocs.io/en/stable/generated/pvlib.temperature.sapm_cell.html
clearsky_model = simplified_solis # model in pvlib.location.get_clearsky (note: 'haurwitz' not supported)
# ----------------------------------------------------- physical definition of PV System, using CEC database
# based on .csv files at ~/.local/lib/python3.8/site-packages/pvlib/data, special characters to be replaced by '_'
ModuleName = Aleo_Solar_S19Y305 # select your panel (or same Wp as yours, eg. 325Wp)
InverterName = SolarEdge_Technologies_Ltd___SE5000__240V_ # select an inverter comparable name plate power (eg. 10kW)
NumStrings = 1 # number of strings
NumPanels = 16 # number of panels per string
# ----------------------------------------------------- PVWatts definition
InverterPower = 4600 # name-plate inverter max. power
NominalEfficiency = 0.965 # nominal European inverter efficiency
SystemPower = 4880 # system power [Wp]
TemperatureCoeff = -0.0040 # temperature coefficient (efficiency loss per 1C)
# ----------------------------------------------------- orientation of solar panels
# location (Latitude, Longitude) defined in [DEFAULT] section above
Tilt = 32
Azimuth = 180 # 270=West, 180=South, 90=East
This is the current numpy version in the Solaranzeige Image.
root@15fe19caa105:/pvforecast# pip show numpy
Name: numpy
Version: 1.19.5
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email:
License: BSD
Location: /usr/lib/python3/dist-packages
Requires:
Required-by: pvlib, siphon
Can't reproduce ... must be in underlying libraries. Although Solaranzeige doesn't use python (numpy), I'd not change without a backup plan ...
One last try with an updated warning ignore
in pvmodel.py
in the development branch ...
Looks good - no more RuntimeWarnings. Are you going to put this in the main branch?
Thx., etofi, for your continued help over this rainy weekend :-)
The development branch is merged into main and release Version-v2.11.02 created
Who maintains the Solaranzeige Docker image? Can you help ensure that this gets merged back into it?
Thank you for the perseverance. The PVForecast files are always re-read automatically when the Docker image is started - therefore no ToDo here. I have created a post and asked DeBaschdi to make the two updates required
pip install packaging
pip install influxdb --upgrade
on the Solaranzeige Docker image.
Thx - then I close this one (hopefully not as prematurely as #22 ;-) )
Thank you for the perseverance. The PVForecast files are always re-read automatically when the Docker image is started - therefore no ToDo here. I have created a post and asked DeBaschdi to make the two updates required
pip install packaging pip install influxdb --upgrade
on the Solaranzeige Docker image.
Thx, done in Image 0.2.2 https://github.com/DeBaschdi/docker.solaranzeige/commit/61ed22ecf0f0e779f067fee6d6e4607d254a8764
I have made the update (Issue 22) and also installed the "packaging" pip package because it is missing in Solaranzeige Docker image.
Unfortunately, I still get the same error message.
Error: Can't run MOSMIX_L - pvlib library installation missing or old version (required: >=0.9.0)