Add basic grid visualization

[maurerle]

This adds a way to visualize a given grid at the end of the Grafana dashboard. This requires nodes to have a shape included in the input data.

It works by writing the grid structure data to the database at the start of the simulation.

It includes some further fixes too:

• It switches the "node_id" to "node" correctly.
• A check for the needed additional_fields in the marketdesign is enforced when using redispatch
• Make nodal_pricing also use the same grid function.
• let loader_csv load the csv files and let the market_mechanism handle pandas Dataframes

What is still needed is to include writing the current load at each bus (and generator?) with the market result. Then we can extend the query with a join of this data and have nice visualizations of the grid usage at each time.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 16.96970% with 137 lines in your changes are missing coverage. Please review.

Project coverage is 77.45%. Comparing base (af02045) to head (c74ed2c).

Files	Patch %	Lines
assume/common/outputs.py	8.95%	61 Missing :warning:
assume/markets/clearing_algorithms/redispatch.py	2.77%	35 Missing :warning:
assume/common/grid_utils.py	0.00%	27 Missing :warning:
...ssume/markets/clearing_algorithms/nodal_pricing.py	23.07%	10 Missing :warning:
assume/markets/clearing_algorithms/__init__.py	40.00%	3 Missing :warning:
assume/markets/base_market.py	75.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #305 +/- ## ========================================== - Coverage 78.53% 77.45% -1.09% ========================================== Files 42 43 +1 Lines 4916 5016 +100 ========================================== + Hits 3861 3885 +24 - Misses 1055 1131 +76 ``` | [Flag](https://app.codecov.io/gh/assume-framework/assume/pull/305/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=assume-framework) | Coverage Δ | | |---|---|---| | [pytest](https://app.codecov.io/gh/assume-framework/assume/pull/305/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=assume-framework) | `77.45% <16.96%> (-1.09%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=assume-framework#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[nick-harder]

@maurerle another comment is that maybe we need a seperate dashboard for this functionality? Ramiz told me that for him it didn't work very well and was extremeley laggy for large networks with many nodes and lines. Putting it into a seperate we at least insure that the main dashboard is accesable, before we can actually check how it performs with large networks

[maurerle]

I addressed the review changes:

• use a separate grid dashboard - this is also the place to add utilization visualizations in the future
• use bus names everywhere and only need to load locations from bus locations
• do not require geopandas, but allow to use its geometry, if the dependency is installed
• use wkt_srid_4326 internally as text representation as pypsa does too

[nick-harder]

@maurerle I am getting the following error when trying to test it. I am testing it using local_db. I see you have a try except method implemented, but the whole error output is still being printed.

`ERROR:assume.common.outputs:tried writing grid data to non postGIS database Traceback (most recent call last): File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context self.dialect.do_execute( File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: near "EXTENSION": syntax error

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Users/nick/GitHub/assume_dev/assume/common/outputs.py", line 275, in store_grid db.execute(text("CREATE EXTENSION IF NOT EXISTS postgis;")) File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute return meth( ^^^^^ File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 517, in _execute_on_connection return connection._execute_clauseelement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context return self._exec_single_context( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context self._handle_dbapi_exception( File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2344, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context self.dialect.do_execute( File "/Users/nick/miniconda3/envs/assume-framework/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "EXTENSION": syntax error [SQL: CREATE EXTENSION IF NOT EXISTS postgis;] (Background on this error at: https://sqlalche.me/e/20/e3q8)`

[nick-harder]

@maurerle I have also tried to test the new dashboard, but I am getting the following error in the map view (grid information). I ran the redispatch example, which has all the info required, I believe

db query error: pq: relation "lines_geo" does not exist

[maurerle]

@nick-harder the db query error: pq: relation "lines_geo" does not exist should not happen if you have a database with postgis support - if not - you can not use the grid visualization feature

[nick-harder]

@maurerle with the new dev distribution I am getting an error again. I have deleted all containers, and DB files, and ran docker compose from scratch. Here is the error I am getting:

sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'Point' [SQL: INSERT INTO buses_geo (name, v_nom, carrier, x, y, wkt_srid_4326, simulation, geometry) VALUES (%(name0)s, %(v_nom0)s, %(carrier0)s, %(x0)s, %(y0)s, %(wkt_srid_43260)s, %(simulation0)s, %(geometry0)s), (%(name1)s, %(v_nom1)s, %(carrier1)s, %(x1)s, %(y1)s, %(wkt_srid_43261)s, %(simulation1)s, %(geometry1)s)] [parameters: {'name0': 'north', 'simulation0': 'example_01d_base', 'geometry0': <POINT (9.598 53.558)>, 'carrier0': 'AC', 'y0': 53.5585, 'wkt_srid_43260': 'SRID=4326;POINT (9.598 53.5585)', 'v_nom0': 380.0, 'x0': 9.598, 'name1': 'south', 'simulation1': 'example_01d_base', 'geometry1': <POINT (13.607 51.077)>, 'carrier1': 'AC', 'y1': 51.0769, 'wkt_srid_43261': 'SRID=4326;POINT (13.607 51.0769)', 'v_nom1': 380.0, 'x1': 13.607}] (Background on this error at: https://sqlalche.me/e/20/f405)

[maurerle]

@nick-harder I did not get this running this container. If you are sure that you have this error using the timescaledb-dev image, it is an upstream issue and will be fixed magically in the future..?

[nick-harder]

@maurerle I found the mistake, tha package geoalchemy2 was missing, but I saw this error only very deep inside the error message. Should we add it as a requirement for the network distribution?

[maurerle]

@nick-harder I thought to have it written to fail gracefully. I added a dumb geoalchemy2 import so that it runs in the import error and falls back to normal to_sql - which should work too