Raw Data

[lmoscattini]

Hi Mate, do you happen to have a csv file for the raw data you used? If so, would you be able to post it, cheers!

[foscoj]

Would be great to see a sample structure, as the output from my discovergy smartmeters needs to be converted!

[Habedere]

i´m also interested ina a sample file! I can´t run the script in jupyter - :( "[Errno 44] No such file or directory" The path must be correct - so the problem is perhaps the File?!

I would have a File with more than 70k Input Lines to test the script.

[Habedere]

Upload was possible - CSV seems correct:

But it fails...

[foscoj]

According to https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html the default delimiter assumed is ','. My discovergy files are comma-separated, so this situation did not occur to me. In your screenshot, the delimiter seems to be ';'. Either you need to change the "sep" parameter in the read_csv call or replace your delimiter in your source files form ';' to ','.

[foscoj]

As this issue was targeted at my Fork, the example for the fork is uploaded here: https://github.com/foscoj/Battery-Simulation/blob/master/example.csv

[PV-Soft]

@foscoj @Habedere @lmoscattini I've included sample data in the repo and updated the notebooks to explain the data structure. Hope this helps, even though my reply is quite late.

[kiryph]

Is it possible to use a custom report from Fronius Solarweb Premium?

As far as I see, only energy (kWh) is exported but only power for the different phases?

See this file (header row is in German): Detailed_Daily_Report_2024_05_11.csv

[PV-Soft]

@kiryph Looking at the data it seems like you already have a battery installed (e.g. column "Energie aus Batterie bezogen"). Are you trying to calculate how much more energy you could use from your PV system with a larger battery?

As you mentioned, the file also contains the power for the different phases (colums "Wirkleistung L1/2/3 Einspeisepunkt | PowerMeter"). The sum of these thee columns should be the total power that you were providing to the grid or taking from the grid at a given time. This could be used for the script to calculate different batteries. The format of the input file is given in sample_data.csv file here in the repo: https://github.com/PV-Soft/Battery-Simulation/blob/91b433e6214ddfd24271bed6fe01aafb844436ce/sample_data.csv

I noticed that your data has a time resolution of 5 minutes so sharp power peaks might be averaged out. But I think that the script should still give you meaningful results.

[kiryph]

@PV-Soft Thanks for the quick response.

Looking at the data it seems like you already have a battery installed (e.g. column "Energie aus Batterie bezogen"). Are you trying to calculate how much more energy you could use from your PV system with a larger battery?

Yes, I have already installed a battery with a capacity of 5.12kWh. I am wondering whether it is financially sensible to upgrade the battery pack by one module (2.56 kWh).

The Fronius Solarweb Battery Simulator also does not provide an option for extending the system by one module of 2.56kWh. One can only choose the smallest battery pack 5.12kWh (here the same as I actually have):

So I cannot use this tool to simulate the smallest possible battery upgrade. Only 2x additional modules which the pictures shows with an additional self-consumption of 130kWh. The next size (3x additional battery modules) already means only an even more minor improvement by 18kWh self-consumption during more than 6 months compared to two additional modules (148kWh improvement compared to the existing battery pack).

As you mentioned, the file also contains the power for the different phases (colums "Wirkleistung L1/2/3 Einspeisepunkt | PowerMeter"). The sum of these thee columns should be the total power that you were providing to the grid or taking from the grid at a given time.

I was assuming this. I will try it. I checked out also the fork https://github.com/foscoj/Battery-Simulation which uses a slightly different csv file, in particular a similarly formatted date and time string (German locale).

My modified Jupyter notebook cell for reading the csv file:

dateTimeFormatString = '%d.%m.%Y %H:%M' #German Time Format String, needed for Discovergy .csv
data = pd.read_csv(r'Solarkraftwerk_01112023-21062024.csv', encoding='latin-1', header=[0,1])
df2 = pd.DataFrame.from_records(data, columns = ['Datum und Uhrzeit', 'Wirkleistung L1 Einspeisepunkt | PowerMeter','Wirkleistung L2 Einspeisepunkt | PowerMeter', 'Wirkleistung L3 Einspeisepunkt | PowerMeter'])
df2['timestamp'] = df2['Datum und Uhrzeit']
df2['power'] = df2['Wirkleistung L1 Einspeisepunkt | PowerMeter'] + df2['Wirkleistung L2 Einspeisepunkt | PowerMeter'] + df2['Wirkleistung L3 Einspeisepunkt | PowerMeter']
df = pd.DataFrame.from_records(df2, columns = ['timestamp', 'power'])
df.tail()

The next cell "Calculate Power and Energy usable by Battery" runs without issues.

However, the cell "Run Battery Simulation" throws an error. Since I am not proficient with Pandas, maybe you know what the error tells me:

Simulating battery of size  1  kWh.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/var/folders/2z/bljkcy1j2cbf7l9zf3dk0nqc0000gn/T/ipykernel_61892/219161310.py](http://localhost:57644/var/folders/2z/bljkcy1j2cbf7l9zf3dk0nqc0000gn/T/ipykernel_61892/219161310.py) in ?()
      1 # Go through all batteries and start simulation for each one. Add results as columns to df.
      2 for i in range(len(battery_sizes)):
----> 3     energy_in_battery, energy_excess_to_grid, power_provided_by_battery, total_power_provided_by_battery = simulate_battery(df, battery_sizes[i])
      4     #battery_charge, battery_charge_excess, battery_discharge, battery_total_discharge = simulate_battery(df, battery_sizes[i])
      5 
      6     df['energy_in_battery_' + battery_names[i]] = energy_in_battery #battery_charge

[/var/folders/2z/bljkcy1j2cbf7l9zf3dk0nqc0000gn/T/ipykernel_61892/1721990933.py](http://localhost:57644/var/folders/2z/bljkcy1j2cbf7l9zf3dk0nqc0000gn/T/ipykernel_61892/1721990933.py) in ?(df, battery_size)
     16         # In this case, set battery state to battery_size or zero.
     17         # Copy excess energy, not taken by battery to this_energy_excess_to_grid
     18         this_energy_in_battery =  row[1]['energy_battery'] + last_energy_in_battery
     19         # If battery is full, copy excess to this_energy_excess_to_grid
---> 20         if this_energy_in_battery > battery_size:
     21             this_energy_excess_to_grid = this_energy_in_battery - battery_size
     22             this_energy_in_battery = battery_size
     23         # If battery is empty, copy excess to this_energy_excess_to_grid

[~/Library/jupyterlab-desktop/jlab_server/lib/python3.12/site-packages/pandas/core/generic.py](http://localhost:57644/~/Library/jupyterlab-desktop/jlab_server/lib/python3.12/site-packages/pandas/core/generic.py) in ?(self)
   1575     @final
   1576     def __nonzero__(self) -> NoReturn:
-> 1577         raise ValueError(
   1578             f"The truth value of a {type(self).__name__} is ambiguous. "
   1579             "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
   1580         )

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Maybe, a newer Pandas version is the issue.

I noticed that your data has a time resolution of 5 minutes so sharp power peaks might be averaged out. But I think that the script should still give you meaningful results.

Yes, the resolution is not as fine as yours. But I agree that the results should still be meaningful.

[PV-Soft]

I've loaded your sample file using following code:

all_data = pd.read_csv('Detailed_Daily_Report_2024_05_11.csv', skiprows= [1])
all_data["timestamp"] = all_data["Datum und Uhrzeit"]
all_data["power"] = all_data["Wirkleistung L1 Einspeisepunkt | PowerMeter"] + all_data["Wirkleistung L2 Einspeisepunkt | PowerMeter"] + all_data["Wirkleistung L3 Einspeisepunkt | PowerMeter"]
all_data = all_data[["timestamp", "power"]]
all_data.head()

In the following cell ("Calculate Power and Energy usable by Battery") I had to remove the argument format='ISO8601 for the conversion to datetime format. Also in the very last cell I removed the argument. Afterwards, the script was running as expected.

Can you reproduce this?

[kiryph]

@PV-Soft the argument for the csv reader skiprows= [1] fixed it for me. Now it runs. Very nice. Thanks for sharing your notebook and helping me out.

Now I have the data I was looking for.

Total energy used from battery instead of grid (in kWh):
total_energy_provided_by_battery_2.56       97.55983
total_energy_provided_by_battery_5.12     135.418436
total_energy_provided_by_battery_7.68     156.898914
total_energy_provided_by_battery_10.24    175.242445
Name: 62738, dtype: object

The Fronius Simulator has returned 130kWh for the 5.12kWh size. But I think I have to reduce the sizes by 5% which should be the usable sizes and then matches very closely the result from Fronius (probably the same logic is used):

Update Here the table with the battery sizes reduced by 5%:

Total energy used from battery instead of grid (in kWh):
total_energy_provided_by_battery_2.56      94.494901
total_energy_provided_by_battery_5.12     132.858436
total_energy_provided_by_battery_7.68     153.826914
total_energy_provided_by_battery_10.24    171.658445
Name: 62738, dtype: object

For reference, I have also looked at https://github.com/hif2k1/battery_sim for Home Assistant but I am not sure if this can use historical data provided by a csv file at all.

[PV-Soft]

@kiryph I'm happy that the script helped you. The reason for the error was that your csv file contains two header rows. The second one is excluded with skiprows= [1] to avoid that this is interpreted as data and crashes the script.

Unfortunately, I'm not familiar with the home assistant battery simulator that you mentioned, so I don't know if you can feed it with csv data.