search

DLR-TS / tsc

coupling TAPAS and SUMO
Eclipse Public License 2.0
10 stars 3 forks source link

Out-Of-Memory crash during database upload #51

Open MHeinrichs opened 1 year ago

MHeinrichs commented 1 year ago

Committing a large simulation (100% Berlin) results in the following error on a 256GB-RAM machine:

Traceback (most recent call last): File "/usr/lib64/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib64/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "tsc_main.py", line 454, in simulation_request s2t_miv.upload_trip_results(conn, sim_key, params, final_routes) File "/usr/share/sumo/tools/sumolib/miscutils.py", line 41, in benchmark_wrapper result = func(args, **kwargs) File "/mnt/SUMO/tsc2023/tsc/s2t_miv.py", line 154, in upload_trip_results cursor.execute(insertQuery % schema_table + ','.join(values)) psycopg2.errors.ProgramLimitExceeded: FEHLER: Speicher aufgebraucht DETAIL: Kann Zeichenkettenpuffer mit 0 Bytes nicht um 1220878031 Bytes vergrößern.

suggested fix (txt instead of py) in lines 153-175: s2t_pt.txt

behrisch commented 1 year ago

The uploaded file contains the original (unchanged) source code

behrisch commented 1 year ago

We already had a try on using a dedicated postgres function here https://www.psycopg.org/docs/extras.html#fast-execution-helpers. I tried to re-enable this code, maybe it already helps