search

architecture-building-systems / CityEnergyAnalyst

The City Energy Analyst (CEA)
https://www.cityenergyanalyst.com/
MIT License
191 stars 61 forks source link

Decentralized optimization fails but stays "Running" #3214

Open martin-mosteiro opened 1 year ago

martin-mosteiro commented 1 year ago

Describe the bug I ran decentralized optimization and noticed even after running for a number of hours no results were being generated. On the dashboard, I still get a status: Running... message, but when I clicked on "More info" I got the following:

City Energy Analyst version 3.31.0
Running `cea optimization` with the following parameters:
- general:debug = False
  (default: False)
[...]
- optimization:crossover-method-continuous = Uniform
  (default: Uniform)
Missing thermal network simulation results. Consider running thermal network simulation script first.

i.e., the optimization tool has failed (no thermal network simulation results were found) but the script is still running.

To Reproduce Steps to reproduce the behavior:

  1. create a scenario
  2. run solar radiation
  3. run the schedule maker
  4. run the demand tool
  5. run all the energy potential tools
  6. run decentralized optimization
  7. run centralized optimization

Expected behavior If the tool has failed to run, the process should be cancelled.

Screenshots Screen Shot 2022-08-04 at 13 13 29

Hardware (please complete the following information):

martin-mosteiro commented 1 year ago

Assigning the optimizer (@MatNif) and the dashboarder extraordinaire (@reyery) since they seem the most suitable candidates to solve this.

lguilhermers commented 1 year ago

I got the same, it also happens if it misses other scripts (the ones from energy potentials, for example)

martin-mosteiro commented 1 year ago

Just an update after starting the optimization script as discussed in #3204: It's been 2+ hours, no new files have been created and the dashboard showed no news. The "More info" pane shows the same information since 2+ hours ago:

[...]
PRE-PROCESSING
PRE-PROCESSING 0/4: initialize directory
PRE-PROCESSING 1/4: weather features
PRE-PROCESSING 2/4: conversion systems database
PRE-PROCESSING 3/4: feedstocks systems database
PRE-PROCESSING 4/4: network features
2 seconds process time for Network summary for configuration 111111111111111
SUPPLY SYSTEMS OPTIMIZATION

So I checked the Terminal and saw the following:

[...]
/server/streams/write/<16>: PRE-PROCESSING 4/4: network features
cea-worker: 2cea-worker:  cea-worker: seconds process time for Network summary for configurationcea-worker:  cea-worker: 111111111111111cea-worker: 
/server/streams/write/<16>: 2 seconds process time for Network summary for configuration 111111111111111
cea-worker: SUPPLY SYSTEMS OPTIMIZATIONcea-worker: 
/server/streams/write/<16>: SUPPLY SYSTEMS OPTIMIZATION
Process SpawnPoolWorker-3:
Process SpawnPoolWorker-1:
Process SpawnPoolWorker-4:
Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-5:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-6:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-8:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-7:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-10:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-11:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-12:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-9:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-13:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Process SpawnPoolWorker-15:
Process SpawnPoolWorker-14:
Process SpawnPoolWorker-16:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
killing child processes of 4 (87091)
/server/streams/write/<16>: SUPPLY SYSTEMS OPTIMIZATION
Process SpawnPoolWorker-3:
Process SpawnPoolWorker-1:
Process SpawnPoolWorker-4:
Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/site-packages/deap/creator.py'>
Traceback (most recent call last):
  File "/Users/martin/opt/miniconda3/envs/cea/lib/python3.8/mul

i.e., it seems like a bunch of processes failed and were killed? Not sure if anything is running anymore, but there's no warning message or anything on the dashboard, either.