search

openego / eTraGo

Optimization of flexibility options for transmission grids based on PyPSA
GNU Affero General Public License v3.0
33 stars 13 forks source link

eTraGo exceptions related to k-means and network clustering #137

Closed nesnoj closed 6 years ago

nesnoj commented 6 years ago

Hey folks,

I'm trying to run eTraGo but get different exceptions - can you help me out?

Some basic info: Ubuntu 17.04, python 3.5.3, venv with eTraGo version from current dev incl. dependencies according to setup.py (pyPSA 0.11 etc..., no further packages/adjustments).

This problem seems to be related to k-means and network clustering since when I disable both, it works..

1. k-means clustering (20) only Args:

args = {# Setup and Configuration:
        'db': 'oedb_remote', # db session
        'gridversion': 'v0.3.0', # None for model_draft or Version number (e.g. v0.2.11) for grid schema
        'method': 'lopf', # lopf or pf
        'pf_post_lopf': False, # state whether you want to perform a pf after a lopf simulation
        'start_snapshot': 1, 
        'end_snapshot' : 2,
        'scn_name': 'Status Quo', # state which scenario you want to run: Status Quo, NEP 2035, eGo100
        'solver': 'gurobi', # glpk, cplex or gurobi
        # Export options:
        'lpfile': False, # state if and where you want to save pyomo's lp file: False or /path/tofolder
        'results': 'results.csv', # state if and where you want to save results as csv: False or /path/tofolder
        'export': False, # state if you want to export the results back to the database
        # Settings:        
        'storage_extendable':False, # state if you want storages to be installed at each node if necessary.
        'generator_noise':False, # state if you want to apply a small generator noise
        'reproduce_noise': False, # state if you want to use a predefined set of random noise for the given scenario. if so, provide path, e.g. 'noise_values.csv'
        'minimize_loading':False,
        # Clustering:
        'k_mean_clustering': 20, # state if you want to perform a k-means clustering on the given network. State False or the value k (e.g. 20).
        'network_clustering': False, # state if you want to perform a clustering of HV buses to EHV buses.
        # Simplifications:
        'parallelisation':False, # state if you want to run snapshots parallely.
        'skip_snapshots':False,
        'line_grouping': False, # state if you want to group lines running between the same buses.
        'branch_capacity_factor': 0.7, # globally extend or lower branch capacities
        'load_shedding':False, # meet the demand at very high cost; for debugging purposes.
        'comments':None }

Log:

...
No data for StoragePqSet in column p_set.
Series p_set of component StorageUnit could not be imported
start k-mean clustering
Traceback (most recent call last):
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 2178, in agg_series
    return self._aggregate_series_fast(obj, func)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 2198, in _aggregate_series_fast
    result, counts = grouper.get_result()
  File "pandas/_libs/src/reduce.pyx", line 407, in pandas._libs.lib.SeriesGrouper.get_result (pandas/_libs/lib.c:39105)
  File "pandas/_libs/src/reduce.pyx", line 391, in pandas._libs.lib.SeriesGrouper.get_result (pandas/_libs/lib.c:38888)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 842, in <lambda>
    f = lambda x: func(x, *args, **kwargs)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pypsa/networkclustering.py", line 53, in consense
    .format(component, attr, x.name, x)
AssertionError: In Generator cluster p_nom_min the values of attribute p_nom_min do not agree:
200017    NaN
24816     0.0
24985     0.0
...
Name: p_nom_min, Length: 92, dtype: float64

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jonathan/git-repos/openego/eTraGo/etrago/appl.py", line 339, in <module>
    network = etrago(args)
  File "/home/jonathan/git-repos/openego/eTraGo/etrago/appl.py", line 277, in etrago
    network = kmean_clustering(network, n_clusters=args['k_mean_clustering'])
  File "/home/jonathan/git-repos/openego/eTraGo/etrago/cluster/networkclustering.py", line 424, in kmean_clustering
    clustering = get_clustering_from_busmap(network, busmap, aggregate_generators_weighted=True, aggregate_one_ports=aggregate_one_ports)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pypsa/networkclustering.py", line 227, in get_clustering_from_busmap
    generators, generators_pnl = aggregategenerators(network, busmap, with_time=with_time)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pypsa/networkclustering.py", line 74, in aggregategenerators
    new_df = generators.groupby(grouper, axis=0).agg(strategies)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 4036, in aggregate
    return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 3468, in aggregate
    result, how = self._aggregate(arg, _level=_level, *args, **kwargs)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/base.py", line 573, in _aggregate
    result = _agg(arg, _agg_1dim)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/base.py", line 524, in _agg
    result[fname] = func(fname, agg_how)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/base.py", line 507, in _agg_1dim
    return colg.aggregate(how, _level=(_level or 0) + 1)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 2880, in aggregate
    return self._python_agg_general(func_or_funcs, *args, **kwargs)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 848, in _python_agg_general
    result, counts = self.grouper.agg_series(obj, f)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 2180, in agg_series
    return self._aggregate_series_pure_python(obj, func)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 2211, in _aggregate_series_pure_python
    res = func(group)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 842, in <lambda>
    f = lambda x: func(x, *args, **kwargs)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pypsa/networkclustering.py", line 53, in consense
    .format(component, attr, x.name, x)
AssertionError: In Generator cluster p_nom_min the values of attribute p_nom_min do not agree:
200017    NaN
24816     0.0
24985     0.0
...

(same problem when I set gridversion = None (model_draft)

2. k-means clustering (500) only Args: As in 1 but with k_mean_clustering = 500

Log:

...
No data for StoragePqSet in column p_set.
Series p_set of component StorageUnit could not be imported
start k-mean clustering
Traceback (most recent call last):
  File "/home/jonathan/git-repos/openego/eTraGo/etrago/appl.py", line 339, in <module>
    network = etrago(args)
  File "/home/jonathan/git-repos/openego/eTraGo/etrago/appl.py", line 277, in etrago
    network = kmean_clustering(network, n_clusters=args['k_mean_clustering'])
  File "/home/jonathan/git-repos/openego/eTraGo/etrago/cluster/networkclustering.py", line 424, in kmean_clustering
    clustering = get_clustering_from_busmap(network, busmap, aggregate_generators_weighted=True, aggregate_one_ports=aggregate_one_ports)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pypsa/networkclustering.py", line 227, in get_clustering_from_busmap
    generators, generators_pnl = aggregategenerators(network, busmap, with_time=with_time)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pypsa/networkclustering.py", line 75, in aggregategenerators
    new_df.index = _flatten_multiindex(new_df.index).rename("name")
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pypsa/networkclustering.py", line 46, in _flatten_multiindex
    return reduce(lambda x, y: x+join+y, levels, next(levels))
NameError: name 'reduce' is not defined

3. network clustering only Args: As in 1 but with k_mean_clustering = False and network_clustering = True

Log:

...
No data for StoragePqSet in column p_set.
Series p_set of component StorageUnit could not be imported
WARNING:pypsa.io:The following Line have buses which are not defined:
Index(['6933'], dtype='object')
Traceback (most recent call last):
  File "/home/jonathan/git-repos/openego/eTraGo/etrago/appl.py", line 339, in <module>
    network = etrago(args)
  File "/home/jonathan/git-repos/openego/eTraGo/etrago/appl.py", line 273, in etrago
    network = cluster_on_extra_high_voltage(network, busmap, with_time=True)
  File "/home/jonathan/git-repos/openego/eTraGo/etrago/cluster/networkclustering.py", line 96, in cluster_on_extra_high_voltage
    new_df, new_pnl = aggregategenerators(network, busmap, with_time)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pypsa/networkclustering.py", line 69, in aggregategenerators
    weighting = generators.weight.groupby(grouper, axis=0).transform(lambda x: (x/x.sum()).fillna(1.))
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/groupby.py", line 3063, in transform
    result.index = self._selected_obj.index
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/generic.py", line 3094, in __setattr__
    return object.__setattr__(self, name, value)
  File "pandas/_libs/src/properties.pyx", line 65, in pandas._libs.lib.AxisProperty.__set__ (pandas/_libs/lib.c:45255)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/series.py", line 308, in _set_axis
    self._data.set_axis(axis, labels)
  File "/home/jonathan/virtualenvs/openego/etrago/lib/python3.5/site-packages/pandas/core/internals.py", line 2836, in set_axis
    (old_len, new_len))
ValueError: Length mismatch: Expected axis has 12345 elements, new values have 12352 elements

4. no clustering at all Args: As in 1 but with k_mean_clustering = False and network_clustering = False

Works!

wolfbunke commented 6 years ago

Do you use the PyPSA fork from https://github.com/openego/PyPSA/tree/dev ?

ClaraBuettner commented 6 years ago

Which pandas version is installed? Due to a bug in pypsa 0.11 it doesn't work with pandas > 20.3. And it seems as you use another db, is it exactly the same as the oedb? Than I could try to reproduce the error. If it's not the same (exactly same buses) you need to create a new busmap for network-clustering. In addition, there is at least one generator with p_nom_min = NaN which can cause the problem. Can you set this to zero and try again?

nesnoj commented 6 years ago

Thanks for your hints!

@wolfbunke According to the dependency link in setup.py the dev from ego fork should be used as install candidate for pyPSA. But according to the installed pyPSA Files, it was not! (you may check what's the problem here..) So I uninstalled pyPSA, cloned your fork@dev and reinstalled manually using pip3 -e. The import error in 2. (could not find reduce) seems to be fixed now so k-means doesn't throw an exception anymore. This is quite strange - it seems that before (installation of eTraGo+dependencies of setup.py) reduce could not be imported. Now with the manual install of PyPSA fork, it's gone.. :question:

So the only remaining bug concerns network clustering..

@ClaraBuettner

wolfbunke commented 6 years ago

Yes I think the problem is the dependency_links of pip which is not installing the dependency any longer. I will test thepip install -e eTraGo --process-dependency-links --allow-all-external see

wolfbunke commented 6 years ago

see #138

ClaraBuettner commented 6 years ago

The empty coulmn message has always been there and shouldn't cause a problem. Did you try to create a new busmap? Some weeks ago I fixed eHV bugs manually and I'm not sure if the busmap includes them. I can check this soon.

nesnoj commented 6 years ago

Did you try to create a new busmap? Some weeks ago I fixed eHV bugs manually and I'm not sure if the busmap includes them.

No I didn't modify the input data at all..

I can check this soon.

Thank you :+1:

ClaraBuettner commented 6 years ago

I tested network-clustering for scenario Status Quo in the model draft and it worked well. Currently, it is only working in model_draft because the eHV-subnetwork errors are not in fixed in grid schema but share a busmap in model_draft. So until the next dp run you can not use network-clustering in grid schema.

nesnoj commented 6 years ago

It works now, cheers!

Network clustering only (2 timesteps): etrago1_networkclustering

Interestingly, when I disable clustering the all line loadings are almost zero :astonished: : etrago1_noclustering

nesnoj commented 6 years ago

The 2nd one seems to be correct. I chose a different timestep which results in several line loadings >> 0:

etrago2_noclustering

Apparently, there's a large difference in line loading between the full 110kV-grid and the aggregated one.. seems quite odd to me as the relative loading of (non-)aggregated lines should be similar, right?

BartelsJ commented 6 years ago

Indeed, this is interesting. Both color scales have the same values in your graphics, haven't they?

nesnoj commented 6 years ago

I close this on in favor of a new issue #139 as the original problem is solved.