eosram
commented
7 years ago Thanks for reporting! I will look into it next week! I have yet no clue what is happening here.
Closed lukasol closed 6 years ago
eosram
commented
7 years ago Thanks for reporting! I will look into it next week! I have yet no clue what is happening here.
ulfmueller
commented
7 years ago I just reproduced the error. I do not have a clue why it happens. The busmap in model_draft exists but remains empty. I called the lopf on data from the versioned grid schema (because the model_draft pf schema are empty right now). Could that be a problem? Am I right that the busmap is put in the model_draft anyhow? @s3pp Have you had a look into it already?
eosram
commented
7 years ago Sorry for answering late. I successfully created the busmap without writing it to the database.
It seems like
217 df.to_sql('ego_grid_pf_hv_busmap', if_exists='append',
218 con=session.bind, schema='model_draft')
in etrago/cluster/networkclustering.py executes without throwing an exception, but does not work anymore, that's why the table remains empty.
eosram
commented
7 years ago I made a minor change regarding the export of the busmap, only affected the busmap functionality, that's why I merged it already. Now ego_pf_hv_busmap is populated again and does not remain empty as reported by @ulfmueller . Unfortunately this was not causing the original error. I had to address clustering strategies in PyPSA to make it work again. PyPSA groups some components like storages on bus and carrier, if within these groups a value, e.g. max_hours, varies a strategy for aggregating has to be in place. I added np.mean as strategy for all those cases. I guess, this should better be reviewed.
ulfmueller
commented
7 years ago now it works! thx @s3pp ! But PyPSA has to be checked out in that feature branch you created. Do you think we can merge it into dev or do the changes interfere with other functionalities? Could the changes affect the k-means clustering? If not, I would like to merge it into dev.
ulfmueller
commented
7 years ago Generally from my understanding of the method it should be fine to merge it to dev. Two thoughts on this:
lukasol
commented
7 years ago I would move your first point to a nice-to-have list with a lower prio, okay?
I cannot really comment on your second point, but yes it seems odd.
Apart from that, the networkclustering works for me for the Status Quo scenario, the NEP 2035 threw an error in my first attempt. I will post here if it reproduces.
lukasol
commented
7 years ago Okay I get this error whenever running the network clustering with the NEP 2035 scenario. Looks like something data-related. Anyone else running into this?
File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 466, in do_executemany cursor.executemany(statement, parameters) OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected [SQL: 'INSERT INTO model_draft.ego_grid_pf_hv_busmap (scn_name, bus0, bus1, path_length) VALUES (%(scn_name)s, %(bus0)s, %(bus1)s, %(path_length)s)'] [parameters: ({'path_length': 1.253197847916541, 'bus1': '24608', 'scn_name': 'NEP 2035', 'bus0': '1'}, {'path_length': 94.98702798105607, 'bus1': '27451', 'scn_name': 'NEP 2035', 'bus0': '100'}, {'path_length': 4.286684696577136, 'bus1': '24391', 'scn_name': 'NEP 2035', 'bus0': '10001'}, {'path_length': 13.643555016725788, 'bus1': '26277', 'scn_name': 'NEP 2035', 'bus0': '10010'}, {'path_length': 32.79698712737845, 'bus1': '26277', 'scn_name': 'NEP 2035', 'bus0': '10013'}, {'path_length': 0.734603068412609, 'bus1': '26277', 'scn_name': 'NEP 2035', 'bus0': '10017'}, {'path_length': 3.1952010638114894, 'bus1': '26277', 'scn_name': 'NEP 2035', 'bus0': '10018'}, {'path_length': 23.92986525409579, 'bus1': '26277', 'scn_name': 'NEP 2035', 'bus0': '10019'} ... displaying 10 of 11309 total bound parameter sets ... {'path_length': 0, 'bus1': '28428', 'scn_name': 'NEP 2035', 'bus0': '28428'}, {'path_length': 0, 'bus1': '28431', 'scn_name': 'NEP 2035', 'bus0': '28431'})]
wolfbunke
commented
7 years ago I was checking the message it looks like a known Postgresql Error: https://stackoverflow.com/questions/20201711/pg-internalerror-ssl-syscall-error-eof-detected. I will run the NEP scenario with clustering to see what is happing at my system.
wolfbunke
commented
7 years ago At my system network clustering works. I am working on /openego/PyPSA (dev) newest version and eTraGo with my last commit 08ad797dfac80e97ac636ed13cdbc8853922a466 from 8. September.
My args setting was:
{'network_clustering':True,
'db': 'oedb', # db session
'gridversion':'v0.2.11', #None for model_draft or Version number (e.g. v0.2.10) for grid schema
'method': 'lopf', # lopf or pf
'pf_post_lopf': True, #state whether you want to perform a pf after a lopf simulation
'start_snapshot': 2320,
'end_snapshot' : 2321,
'scn_name': 'SH NEP 2035',
'lpfile': False, # state if and where you want to save pyomo's lp file: False or '/path/tofolder'
'results': False , # state if and where you want to save results as csv: False or '/path/tofolder'
'export': False, # state if you want to export the results back to the database
'solver': 'gurobi', #glpk, cplex or gurobi
'branch_capacity_factor': 1, #to globally extend or lower branch capacities
'storage_extendable':True,
'load_shedding':True,
'generator_noise':True,
'minimize_loading':False,
'k_mean_clustering': False,
'parallelisation':False,
'comments': None}The result was:
lukasol
commented
7 years ago That looks like SH, that works for me too. I'm talking about the full German dataset.
eosram
commented
7 years ago Hm, maybe the connection to the database was lost while creating the busmap. Did it run a substantial time, like an hour, before the error was thrown?
lukasol
commented
7 years ago Yes, several hours (I thought busmap was created once?!).
My problem is that apart from the schema grid the function seems to rely on the schema model_draft, which makes running on a local dump more difficult.
eosram
commented
7 years ago Yes, several hours (I thought busmap was created once?!).
If a busmap is present via the session for a specific scenario it should not be created again. I see, if it has timed-out I think it would be necessary to check if there is still a valid connection to the database before writing the busmap back. I could implement that if that's the real issue here because maybe running the application locally the database is missing model_draft.ego_grid_pf_hv_busmap and that's why the error is thrown. As a workaround you could execute https://github.com/openego/data_processing/blob/dev/dataprocessing/sql_snippets/ego_dp_powerflow_hv_setup.sql#L237-L243 locally.
My problem is that apart from the schema grid the function seems to rely on the schema model_draft, which makes running on a local dump more difficult.
The busmap is not part of the grid schema, because it is potentially (different scenario name) updated each time running the application. That was our initial thought here and that's why it is kept in the model_draft. But maybe this is not correct anymore and I should change that? I'm not sure about that.
wolfbunke
commented
7 years ago Ok, a new calculation is started with the "NEP 2035" scenarion. It is running but I got few warnings:
WARNING:pypsa.components:Attribute network.now is not in newly-defined snapshots. (network.now is only relevant if you call e.g. network.pf() without specifying snapshots.)
WARNING:pypsa.io:Components Index(['20034', '20295', '20488', '20615', '20754', '20841', '21311', '21414',
'21549', '21599', '21763', '22020', '22543', '22750', '22886', '23005',
'23145', '23340', '23601', '23802', '23832', '24272', '24422', '24706',
'24922'],
dtype='object') for attribute p_set of Generator are not in main components dataframe generators
WARNING:pypsa.io:Components Index(['20034', '20295', '20488', '20615', '20754', '20841', '21311', '21414',
'21549', '21599', '21763', '22020', '22543', '22750', '22886', '23005',
'23145', '23340', '23601', '23802', '23832', '24272', '24422', '24706',
'24922'],
dtype='object') for attribute p_max_pu of Generator are not in main components dataframe generators
No data for StoragePqSet in column p_set.
Series p_set of component StorageUnit could not be imported
Busmap does not exist and will be created.These warnings shouldn't be related to the network clustering, but to the network data. There are some generators in grid.ego_pf_hv_generator_pq_set, that are not in grid.ego_pf_hv_generator.
wolfbunke
commented
7 years ago ok, thanks!
wolfbunke
commented
7 years ago After less then 12 h I run into gurobi licence issues :-) I will update gurobi and start it again. Maybe the report has some interesting parts so far:
WARNING:pypsa.components:Attribute network.now is not in newly-defined snapshots. (network.now is only relevant if you call e.g. network.pf() without specifying snapshots.)
WARNING:pypsa.io:Components Index(['20034', '20295', '20488', '20615', '20754', '20841', '21311', '21414',
'21549', '21599', '21763', '22020', '22543', '22750', '22886', '23005',
'23145', '23340', '23601', '23802', '23832', '24272', '24422', '24706',
'24922'],
dtype='object') for attribute p_set of Generator are not in main components dataframe generators
WARNING:pypsa.io:Components Index(['20034', '20295', '20488', '20615', '20754', '20841', '21311', '21414',
'21549', '21599', '21763', '22020', '22543', '22750', '22886', '23005',
'23145', '23340', '23601', '23802', '23832', '24272', '24422', '24706',
'24922'],
dtype='object') for attribute p_max_pu of Generator are not in main components dataframe generators
No data for StoragePqSet in column p_set.
Series p_set of component StorageUnit could not be imported
Busmap does not exist and will be created.
cpu_cores (default 4): 4
WARNING:pypsa.components:Attribute network.now is not in newly-defined snapshots. (network.now is only relevant if you call e.g. network.pf() without specifying snapshots.)
INFO:pypsa.pf:Slack bus for sub-network 0 is 10
INFO:pypsa.pf:Slack bus for sub-network 1 is 1127
INFO:pypsa.pf:Slack bus for sub-network 2 is 23561
INFO:pypsa.pf:Slack bus for sub-network 3 is 23795
INFO:pypsa.pf:Slack bus for sub-network 4 is 24504
INFO:pypsa.pf:Slack bus for sub-network 5 is 25533
INFO:pypsa.pf:Slack bus for sub-network 6 is 25562
INFO:pypsa.pf:Slack bus for sub-network 7 is 26299
INFO:pypsa.opf:Performed preliminary steps
INFO:pypsa.opf:Building pyomo model using `angles` formulation
INFO:pypsa.opf:Solving model using gurobi
ERROR: "[base]/site-packages/pyomo/opt/base/solvers.py", 600, solve
Solver (gurobi) returned non-zero return code (1)
ERROR:pyomo.opt:Solver (gurobi) returned non-zero return code (1)
ERROR: "[base]/site-packages/pyomo/opt/base/solvers.py", 605, solve
Solver log:
Traceback (most recent call last):
File "<stdin>", line 5, in <module>
File "/home/dozeumbuw/eTraGo/lib/python3.5/site-packages/pyomo/solvers/plugins/solvers/GUROBI_RUN.py", line 46, in gurobi_run
...
gurobipy.GurobiError: HostID mismatch (licensed to xx, hostid is yx) - license file '/home/dozeumbuw/gurobi.lic'
Traceback (most recent call last):
File "appl.py", line 170, in <module>
network = etrago(args)
File "appl.py", line 133, in etrago
network.lopf(scenario.timeindex, solver_name=args['solver'], extra_functionality=extra_functionality)
File "/home/dozeumbuw/eTraGo/src/pypsa/pypsa/opf.py", line 1099, in network_lopf
keepfiles=keep_files, options=solver_options)
File "/home/dozeumbuw/eTraGo/lib/python3.5/site-packages/pyomo/opt/base/solvers.py", line 607, in solve
"Solver (%s) did not exit normally" % self.name)
pyutilib.common._exceptions.ApplicationError: Solver (gurobi) did not exit normally
Well, looks good for the busmap. I just don't get that far. Could you please state your args? And are you running with the latest versions of eTraGo etc.?
wolfbunke
commented
7 years ago Yes I will update eTraGo. With my actual setting I got my first - unfortunately error- result:
WARNING:pypsa.components:Attribute network.now is not in newly-defined snapshots. (network.now is only relevant if you call e.g. network.pf() without specifying snapshots.)
WARNING:pypsa.io:Components Index(['20034', '20295', '20488', '20615', '20754', '20841', '21311', '21414',
'21549', '21599', '21763', '22020', '22543', '22750', '22886', '23005',
'23145', '23340', '23601', '23802', '23832', '24272', '24422', '24706',
'24922'],
dtype='object') for attribute p_set of Generator are not in main components dataframe generators
WARNING:pypsa.io:Components Index(['20034', '20295', '20488', '20615', '20754', '20841', '21311', '21414',
'21549', '21599', '21763', '22020', '22543', '22750', '22886', '23005',
'23145', '23340', '23601', '23802', '23832', '24272', '24422', '24706',
'24922'],
dtype='object') for attribute p_max_pu of Generator are not in main components dataframe generators
No data for StoragePqSet in column p_set.
Series p_set of component StorageUnit could not be imported
WARNING:pypsa.components:Attribute network.now is not in newly-defined snapshots. (network.now is only relevant if you call e.g. network.pf() without specifying snapshots.)
INFO:pypsa.pf:Slack bus for sub-network 0 is 10
INFO:pypsa.pf:Slack bus for sub-network 1 is 1127
INFO:pypsa.pf:Slack bus for sub-network 2 is 23561
INFO:pypsa.pf:Slack bus for sub-network 3 is 23795
INFO:pypsa.pf:Slack bus for sub-network 4 is 24504
INFO:pypsa.pf:Slack bus for sub-network 5 is 25533
INFO:pypsa.pf:Slack bus for sub-network 6 is 25562
INFO:pypsa.pf:Slack bus for sub-network 7 is 26299
INFO:pypsa.opf:Performed preliminary steps
INFO:pypsa.opf:Building pyomo model using `angles` formulation
INFO:pypsa.opf:Solving model using gurobi
INFO:pypsa.opf:Optimization successful
INFO:pypsa.pf:Slack bus for sub-network 0 is 10
INFO:pypsa.pf:Performing non-linear load-flow on AC sub-network SubNetwork 0 for snapshots DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 01:00:00',
'2011-01-01 02:00:00', '2011-01-01 03:00:00',
'2011-01-01 04:00:00', '2011-01-01 05:00:00',
'2011-01-01 06:00:00', '2011-01-01 07:00:00',
'2011-01-01 08:00:00', '2011-01-01 09:00:00',
'2011-01-01 10:00:00', '2011-01-01 11:00:00',
'2011-01-01 12:00:00', '2011-01-01 13:00:00',
'2011-01-01 14:00:00', '2011-01-01 15:00:00',
'2011-01-01 16:00:00', '2011-01-01 17:00:00',
'2011-01-01 18:00:00', '2011-01-01 19:00:00'],
dtype='datetime64[ns]', freq='H')
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003563 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003478 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.006021 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003527 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.005708 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003496 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003476 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003514 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003622 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003513 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003484 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.005627 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003671 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003471 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003533 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.005051 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003518 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003456 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.003371 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of nan in 0.004528 seconds
INFO:pypsa.pf:Slack bus for sub-network 1 is 1127
INFO:pypsa.pf:Performing non-linear load-flow on AC sub-network SubNetwork 1 for snapshots DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 01:00:00',
'2011-01-01 02:00:00', '2011-01-01 03:00:00',
'2011-01-01 04:00:00', '2011-01-01 05:00:00',
'2011-01-01 06:00:00', '2011-01-01 07:00:00',
'2011-01-01 08:00:00', '2011-01-01 09:00:00',
'2011-01-01 10:00:00', '2011-01-01 11:00:00',
'2011-01-01 12:00:00', '2011-01-01 13:00:00',
'2011-01-01 14:00:00', '2011-01-01 15:00:00',
'2011-01-01 16:00:00', '2011-01-01 17:00:00',
'2011-01-01 18:00:00', '2011-01-01 19:00:00'],
dtype='datetime64[ns]', freq='H')
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020286 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000001 in 0.011079 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000001 in 0.010733 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019300 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.021678 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.021547 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018313 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.022499 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020675 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019228 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010398 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019580 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018120 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020177 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010404 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010639 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010346 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020497 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020442 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018315 seconds
INFO:pypsa.pf:Slack bus for sub-network 2 is 23561
INFO:pypsa.pf:Performing non-linear load-flow on AC sub-network SubNetwork 2 for snapshots DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 01:00:00',
'2011-01-01 02:00:00', '2011-01-01 03:00:00',
'2011-01-01 04:00:00', '2011-01-01 05:00:00',
'2011-01-01 06:00:00', '2011-01-01 07:00:00',
'2011-01-01 08:00:00', '2011-01-01 09:00:00',
'2011-01-01 10:00:00', '2011-01-01 11:00:00',
'2011-01-01 12:00:00', '2011-01-01 13:00:00',
'2011-01-01 14:00:00', '2011-01-01 15:00:00',
'2011-01-01 16:00:00', '2011-01-01 17:00:00',
'2011-01-01 18:00:00', '2011-01-01 19:00:00'],
dtype='datetime64[ns]', freq='H')
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018545 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019625 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020088 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020428 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018126 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019618 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020371 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019100 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018481 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019650 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020124 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018272 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020633 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019780 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018078 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.021362 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020939 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018448 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018270 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019973 seconds
INFO:pypsa.pf:Slack bus for sub-network 3 is 23795
INFO:pypsa.pf:Performing non-linear load-flow on AC sub-network SubNetwork 3 for snapshots DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 01:00:00',
'2011-01-01 02:00:00', '2011-01-01 03:00:00',
'2011-01-01 04:00:00', '2011-01-01 05:00:00',
'2011-01-01 06:00:00', '2011-01-01 07:00:00',
'2011-01-01 08:00:00', '2011-01-01 09:00:00',
'2011-01-01 10:00:00', '2011-01-01 11:00:00',
'2011-01-01 12:00:00', '2011-01-01 13:00:00',
'2011-01-01 14:00:00', '2011-01-01 15:00:00',
'2011-01-01 16:00:00', '2011-01-01 17:00:00',
'2011-01-01 18:00:00', '2011-01-01 19:00:00'],
dtype='datetime64[ns]', freq='H')
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018340 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019709 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.021015 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020627 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018647 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019073 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020517 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020108 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018206 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020487 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019939 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018064 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020651 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020577 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018233 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019024 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020453 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020018 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018193 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019985 seconds
INFO:pypsa.pf:Slack bus for sub-network 4 is 24504
INFO:pypsa.pf:Performing non-linear load-flow on AC sub-network SubNetwork 4 for snapshots DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 01:00:00',
'2011-01-01 02:00:00', '2011-01-01 03:00:00',
'2011-01-01 04:00:00', '2011-01-01 05:00:00',
'2011-01-01 06:00:00', '2011-01-01 07:00:00',
'2011-01-01 08:00:00', '2011-01-01 09:00:00',
'2011-01-01 10:00:00', '2011-01-01 11:00:00',
'2011-01-01 12:00:00', '2011-01-01 13:00:00',
'2011-01-01 14:00:00', '2011-01-01 15:00:00',
'2011-01-01 16:00:00', '2011-01-01 17:00:00',
'2011-01-01 18:00:00', '2011-01-01 19:00:00'],
dtype='datetime64[ns]', freq='H')
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002481 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020850 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002390 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002454 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002401 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019871 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002433 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020343 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002375 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002432 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002397 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002374 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.019603 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002373 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020293 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002381 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020036 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020085 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002463 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002393 seconds
INFO:pypsa.pf:Slack bus for sub-network 5 is 25533
INFO:pypsa.pf:Performing non-linear load-flow on AC sub-network SubNetwork 5 for snapshots DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 01:00:00',
'2011-01-01 02:00:00', '2011-01-01 03:00:00',
'2011-01-01 04:00:00', '2011-01-01 05:00:00',
'2011-01-01 06:00:00', '2011-01-01 07:00:00',
'2011-01-01 08:00:00', '2011-01-01 09:00:00',
'2011-01-01 10:00:00', '2011-01-01 11:00:00',
'2011-01-01 12:00:00', '2011-01-01 13:00:00',
'2011-01-01 14:00:00', '2011-01-01 15:00:00',
'2011-01-01 16:00:00', '2011-01-01 17:00:00',
'2011-01-01 18:00:00', '2011-01-01 19:00:00'],
dtype='datetime64[ns]', freq='H')
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010282 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.011499 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010874 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010533 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010586 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.012792 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010183 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.012800 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000001 in 0.010296 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.023329 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020056 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018198 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020959 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.021870 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000001 in 0.010327 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000001 in 0.013473 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000001 in 0.010491 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000001 in 0.012468 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.018145 seconds
INFO:pypsa.pf:Newton-Raphson solved in 2 iterations with error of 0.000000 in 0.020455 seconds
INFO:pypsa.pf:Slack bus for sub-network 6 is 25562
INFO:pypsa.pf:Performing non-linear load-flow on AC sub-network SubNetwork 6 for snapshots DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 01:00:00',
'2011-01-01 02:00:00', '2011-01-01 03:00:00',
'2011-01-01 04:00:00', '2011-01-01 05:00:00',
'2011-01-01 06:00:00', '2011-01-01 07:00:00',
'2011-01-01 08:00:00', '2011-01-01 09:00:00',
'2011-01-01 10:00:00', '2011-01-01 11:00:00',
'2011-01-01 12:00:00', '2011-01-01 13:00:00',
'2011-01-01 14:00:00', '2011-01-01 15:00:00',
'2011-01-01 16:00:00', '2011-01-01 17:00:00',
'2011-01-01 18:00:00', '2011-01-01 19:00:00'],
dtype='datetime64[ns]', freq='H')
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010225 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.011065 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002747 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.011868 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002443 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010231 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002393 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.011508 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010208 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002994 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010320 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010366 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.004160 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002593 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002502 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002392 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.011547 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002427 seconds
INFO:pypsa.pf:Newton-Raphson solved in 1 iterations with error of 0.000000 in 0.010391 seconds
INFO:pypsa.pf:Newton-Raphson solved in 0 iterations with error of 0.000000 in 0.002390 seconds
INFO:pypsa.pf:Slack bus for sub-network 7 is 26299
INFO:pypsa.pf:Performing non-linear load-flow on AC sub-network SubNetwork 7 for snapshots DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 01:00:00',
'2011-01-01 02:00:00', '2011-01-01 03:00:00',
'2011-01-01 04:00:00', '2011-01-01 05:00:00',
'2011-01-01 06:00:00', '2011-01-01 07:00:00',
'2011-01-01 08:00:00', '2011-01-01 09:00:00',
'2011-01-01 10:00:00', '2011-01-01 11:00:00',
'2011-01-01 12:00:00', '2011-01-01 13:00:00',
'2011-01-01 14:00:00', '2011-01-01 15:00:00',
'2011-01-01 16:00:00', '2011-01-01 17:00:00',
'2011-01-01 18:00:00', '2011-01-01 19:00:00'],
dtype='datetime64[ns]', freq='H')
Traceback (most recent call last):
File "appl.py", line 170, in <module>
network = etrago(args)
File "appl.py", line 142, in etrago
pf_post_lopf(network, scenario)
File "/home/dozeumbuw/eTraGo/src/etrago/etrago/extras/utilities.py", line 220, in pf_post_lopf
network_pf.pf(scenario.timeindex, use_seed=True)
File "/home/dozeumbuw/eTraGo/src/pypsa/pypsa/pf.py", line 132, in network_pf
_network_prepare_and_run_pf(network, snapshots, skip_pre, linear=False, x_tol=x_tol, use_seed=use_seed)
File "/home/dozeumbuw/eTraGo/src/pypsa/pypsa/pf.py", line 109, in _network_prepare_and_run_pf
sub_network_pf_fun(sub_network, snapshots=snapshots, skip_pre=True, **kwargs)
File "/home/dozeumbuw/eTraGo/src/pypsa/pypsa/pf.py", line 301, in sub_network_pf
roots[i], n_iter, diff = newton_raphson_sparse(f,guess,dfdx,x_tol=x_tol)
File "/home/dozeumbuw/eTraGo/src/pypsa/pypsa/pf.py", line 143, in newton_raphson_sparse
F = f(guess)
File "/home/dozeumbuw/eTraGo/src/pypsa/pypsa/pf.py", line 237, in f
mismatch = V*np.conj(sub_network.Y*V) - s
AttributeError: 'SubNetwork' object has no attribute 'Y'
(eTraGo) dozeumbuw@n00004755:~/eTraGo/src/etrago/etrago$
My args have been:
'db': 'oedb', # db session
'gridversion':'v0.2.11',
'method': 'lopf',
'pf_post_lopf': True,
'start_snapshot': 2320,
'end_snapshot' : 2321,
'scn_name': 'NEP 2035',
'lpfile': False,
'results': False ,
'export': False,
'solver': 'gurobi',
'branch_capacity_factor': 1,
'storage_extendable':True,
'load_shedding':True,
'generator_noise':True,
'minimize_loading':False,
'k_mean_clustering': False,
'parallelisation':False,
'comments': None}well, Wolf, the clustering worked - you just had the 'pf_post_lopf' function set to 'True'. The lopf was solved well just the pf did not work. That is another known problem which does not have anything to do with the ehv clustering.
wolfbunke
commented
7 years ago ok, I will change the setting and run it again.
ulfmueller
commented
7 years ago it seems that the busmap was created fine! :+1: The following runs should be solved at normal speed.
wolfbunke
commented
7 years ago ok
wolfbunke
commented
7 years ago software is up-to-date and calculation runs. let's see in few hours what will happen :smile:
lukasol
commented
7 years ago I'm still struggling with the networkclustering for the German dataset (SH works fine). I do this with Status Quo data version v0.2.11 of the grid schema. NEP 2035 works fine again. This is the error I receive:
Busmap does not exist and will be created. cpu_cores (default 4): 4 Traceback (most recent call last): File "
", line 1, in network = etrago(args) File " ", line 181, in etrago busmap = busmap_from_psql(network, session, scn_name=args['scn_name']) File "/srv/ES2050/open_eGo/AP3/github/eTraGo/etrago/cluster/networkclustering.py", line 246, in busmap_from_psql cpu_cores=int(cpu_cores)) File "/srv/ES2050/open_eGo/AP3/github/eTraGo/etrago/cluster/networkclustering.py", line 180, in busmap_by_shortest_path df = df.loc[mask, :] File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/indexing.py", line 1309, in getitem return self._getitem_tuple(key) File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/indexing.py", line 795, in _getitem_tuple return self._getitem_lowerdim(tup) File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/indexing.py", line 902, in _getitem_lowerdim return self._getitem_nested_tuple(tup) File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/indexing.py", line 977, in _getitem_nested_tuple obj = getattr(obj, self.name)._getitem_axis(key, axis=axis) File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/indexing.py", line 1453, in _getitem_axis elif is_bool_indexer(key): File "/home/LuWien/miniconda3/envs/lukas_ego/lib/python3.5/site-packages/pandas/core/common.py", line 201, in is_bool_indexer raise ValueError('cannot index with vector containing ' ValueError: cannot index with vector containing NA / NaN values
lukasoldi
commented
6 years ago I could now reproduce the error above on the openego server in a virtual environment. did any of you encounter something similar? @s3pp do you have any idea what could be the issue here?
lukasoldi
commented
6 years ago This is the error message (in short) that I receive when doing the NEP 2035 scenario for DE and applying networkclustering. As discussed with @BartelsJ, this looks like something data related and there's hope that we can resolve this with version 0.3 of data processing next week...?!
OperationalError Traceback (most recent call last) [150/955] ~/lukas_ego/lib/python3.5/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, *args) 1158 parameters, -> 1159 context) 1160 elif not parameters and context.no_parameters: ~/lukas_ego/lib/python3.5/site-packages/sqlalchemy/engine/default.py in do_executemany(self, cursor, statement, parameters, context) 465 def do_executemany(self, cursor, statement, parameters, context=None): --> 466 cursor.executemany(statement, parameters) 467 OperationalError: SSL SYSCALL error: EOF detected The above exception was the direct cause of the following exception: OperationalError Traceback (most recent call last) OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected [SQL: 'INSERT INTO model_draft.ego_grid_pf_hv_busmap (scn_name, bus0, bus1, path_length) VALUES (%(scn_name)s, %(bus0)s, %(bus1)s, %(path_length)s)'] [parameters: ({'bus1': '24608', $ path_length': 1.253197847916541, 'bus0': '1', 'scn_name': 'NEP 2035'}, {'bus1': '27451', 'path_length': 94.98702798105607, 'bus0': '100', 'scn_name': 'NEP 2035'}, {'bus1': '24391', 'p$ th_length': 4.286684696577136, 'bus0': '10001', 'scn_name': 'NEP 2035'}, {'bus1': '26277', 'path_length': 13.643555016725788, 'bus0': '10010', 'scn_name': 'NEP 2035'}, {'bus1': '26277$ , 'path_length': 32.79698712737845, 'bus0': '10013', 'scn_name': 'NEP 2035'}, {'bus1': '26277', 'path_length': 0.734603068412609, 'bus0': '10017', 'scn_name': 'NEP 2035'}, {'bus1': '2$ 277', 'path_length': 3.1952010638114894, 'bus0': '10018', 'scn_name': 'NEP 2035'}, {'bus1': '26277', 'path_length': 23.92986525409579, 'bus0': '10019', 'scn_name': 'NEP 2035'} ... di$ playing 10 of 11309 total bound parameter sets ... {'bus1': '28428', 'path_length': 0, 'bus0': '28428', 'scn_name': 'NEP 2035'}, {'bus1': '28431', 'path_length': 0, 'bus0': '28431', $ scn_name': 'NEP 2035'})]
lukasoldi
commented
6 years ago I found some generators with source = null and replaced them with 0 but without a positive result on the error posted above.
BartelsJ
commented
6 years ago I am still a bit confused about this issue. However, the SH scenarios are not running at my system at the moment (see below) - working on the dev branch with an actual pull. Maybe anyone is familiar with such a message?
Traceback (most recent call last): File "
", line 1, in runfile('/home/jbartels/projekte/open-ego/eTraGo/etrago/appl.py', wdir='/home/jbartels/projekte/open-ego/eTraGo') File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 880, in runfile execfile(filename, namespace) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile exec(compile(f.read(), filename, 'exec'), namespace) File "/home/jbartels/projekte/open-ego/eTraGo/etrago/appl.py", line 323, in network = etrago(args) File "/home/jbartels/projekte/open-ego/eTraGo/etrago/appl.py", line 267, in etrago network = cluster_on_extra_high_voltage(network, busmap, with_time=True) File "/home/jbartels/projekte/open-ego/eTraGo/etrago/cluster/networkclustering.py", line 105, in cluster_on_extra_high_voltage new_df, new_pnl = aggregateoneport(network, busmap, component=one_port, with_time=with_time) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pypsa/networkclustering.py", line 101, in aggregateoneport new_df = old_df.groupby(grouper).agg(strategies) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/groupby.py", line 4036, in aggregate return super(DataFrameGroupBy, self).aggregate(arg, *args, kwargs) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/groupby.py", line 3468, in aggregate result, how = self._aggregate(arg, _level=_level, *args, *kwargs) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/base.py", line 573, in _aggregate result = _agg(arg, _agg_1dim) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/base.py", line 524, in _agg result[fname] = func(fname, agg_how) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/base.py", line 507, in _agg_1dim return colg.aggregate(how, _level=(_level or 0) + 1) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/groupby.py", line 2880, in aggregate return self._python_agg_general(func_or_funcs, args, kwargs) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/groupby.py", line 848, in _python_agg_general result, counts = self.grouper.agg_series(obj, f) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/groupby.py", line 2180, in agg_series return self._aggregate_series_pure_python(obj, func) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/groupby.py", line 2211, in _aggregate_series_pure_python res = func(group) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pandas/core/groupby.py", line 842, in f = lambda x: func(x, *args, **kwargs) File "/home/jbartels/miniconda3/envs/julian/lib/python3.6/site-packages/pypsa/networkclustering.py", line 54, in consense .format(component, attr, x.name, x) AssertionError: In StorageUnit cluster max_hours the values of attribute max_hours do not agree: 28233 168.0 24211 6.0 Name: max_hours, dtype: float64
lukasoldi
commented
6 years ago Networkclustering seems to work fine with the latest updates of PyPSA. Pull dev branch and have fun.
Hi, I receive the following error when setting
networkclustering = truewith data from the versioned schema. It seems to be irrelevant which scenario or spatial dimension I use (SH or DE). Any clues?