search

djgroen / flee

flee agent-based modelling code
BSD 3-Clause "New" or "Revised" License
6 stars 7 forks source link

Validate Flee 3.0 with autovalidator #59

Closed djgroen closed 1 year ago

djgroen commented 1 year ago

fabsim localhost validate_flee

Check how it compares with previous versions, and what kind of issues arise.

djgroen commented 1 year ago

Hi all. Perhaps as a first test, try to run either

fabsim localhost validate_flee or fabsim archer2 validate_flee:cores=16 and let me know what happens. If any of these take longer than 4 hours, feel free to kill it and let us know here how far it got for you.

ynxue commented 1 year ago

fabsim localhost validate_flee

roughly 40 minutes

djgroen commented 1 year ago

@cspgdds How do these compare with errors from Flee 2.0? Are you able to extract those from deliverables?

djgroen commented 1 year ago

Actually, I found these results from our Alphabet talk: image

djgroen commented 1 year ago

According to Yani's results we have for 3.0: Burundi: 0.543 CAR: 0.315 Mali: 0.279 South Sudan: 0.472

Results are slightly better for Mali and South Sudan, somewhat worse for CAR and much worse for Burundi. Is anyone perhaps willing to look into Burundi in detail?

ynxue commented 1 year ago

I got the following error when I run fabsim archer2 validate_flee:cores=16 or fabsim archer2 sflee:mali2012,simulation_period=300

Lmod has detected the following error: The following module(s) are unknown:
"python/3.7.3"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "python/3.7.3"

Also make sure that all modulefiles written in TCL start with the string
#%Module

Lmod has detected the following error: The following module(s) are unknown:
"openmpi/4.0.0_gcc620"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "openmpi/4.0.0_gcc620"

Also make sure that all modulefiles written in TCL start with the string
#%Module

/opt/cray/pe/python/3.8.5.0/bin/python3: can't open file '/work/e723/e723/yanixue/flee/runscripts/run_par.py': [Errno 2] No such file or directory
srun: error: nid001020: task 0: Exited with exit code 2
srun: launch/slurm: _step_signal: Terminating StepId=3545249.0
djgroen commented 1 year ago

Okay, after several code modifications I'm getting the following:

Validation Burundi: 0.43215342295775255

This is a substantial improvement imho, even if it isn't as low as before.

djgroen commented 1 year ago

Hi all.

I set a single validation settings file for all runs now, and populated it with defaults. This is what I got:

djgroen commented 1 year ago 
╭─ New/Updated environment variables from FabFlee plugin ─╮
│ env :                                                   │
│   --- local_system_time : 1688128875                    │
│   +++ local_system_time : 1688130125                    │
╰─────────────────────────────────────────────────────────╯
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016/validation_results.yml
Validation nigeria2016: 0.16797332004059856
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012/validation_results.yml
Validation mali2012: 0.25379243927291645
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013/validation_results.yml
Validation syria2013: 0.2807021717498154
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013/validation_results.yml
Validation ssudan2013: 0.34431084559711483
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015/validation_results.yml
Validation burundi2015: 0.4732512432711139
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016/validation_results.yml
Validation ssudan2016: 0.638695145256957
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020/validation_results.yml
Validation ethiopia2020: 0.4121094869569241
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013/validation_results.yml
Validation car2013: 0.3006704052206259
Mean score: 0.35893813217075826
csstddg@LIN-LAPTOP-DG:~/FabSim3/plugins/FabFlee/config_files/validation$ cat simsetting.yml 
log_levels:
  agent: 0
  camp: 0
  init: 0
spawn_rules:
  take_from_population: False
  conflict_spawn_decay: [1.0,1.0,1.0,0.5,0.1]
move_rules:
  start_on_foot: True
  avoid_short_stints: True
  awareness_level: 1
optimisations:
djgroen commented 1 year ago 
calling task validate_flee_output from plugin FabFlee
╭─ New/Updated environment variables from FabFlee plugin ─╮
│ env :                                                   │
│   --- local_system_time : 1688136775                    │
│   +++ local_system_time : 1688138096                    │
╰─────────────────────────────────────────────────────────╯
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016/validation_results.yml
Validation nigeria2016: 0.24806127517596688
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012/validation_results.yml
Validation mali2012: 0.6016316087528508
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013/validation_results.yml
Validation syria2013: 0.2804665284298677
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013/validation_results.yml
Validation ssudan2013: 0.5387999032251021
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015/validation_results.yml
Validation burundi2015: 0.454311146051518
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016/validation_results.yml
Validation ssudan2016: 0.6413571437674297
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020/validation_results.yml
Validation ethiopia2020: 0.4155992678058352
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013/validation_results.yml
Validation car2013: 0.49962555148632
Mean score: 0.45998155308686134
csstddg@LIN-LAPTOP-DG:~/FabSim3/plugins/FabFlee/config_files/validation$ cat simsetting.yml 
log_levels:
  agent: 0
  camp: 0
  init: 0
spawn_rules:
  take_from_population: False
  conflict_spawn_decay: [1.0,1.0,1.0,0.5,0.1]
move_rules:
  start_on_foot: True
  avoid_short_stints: True
  awareness_level: 2
optimisations:
djgroen commented 1 year ago

TakeFromPopulation seems to have virtually no effect on overall accuracy.

calling task validate_flee_output from plugin FabFlee
╭─ New/Updated environment variables from FabFlee plugin ─╮
│ env :                                                   │
│   --- local_system_time : 1688141064                    │
│   +++ local_system_time : 1688142315                    │
╰─────────────────────────────────────────────────────────╯
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016/validation_results.yml
Validation nigeria2016: 0.16806186709388385
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012/validation_results.yml
Validation mali2012: 0.2565580535684459
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013/validation_results.yml
Validation syria2013: 0.28163247080459797
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013/validation_results.yml
Validation ssudan2013: 0.33165842633274645
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015/validation_results.yml
Validation burundi2015: 0.47998599488107757
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016/validation_results.yml
Validation ssudan2016: 0.6382935065332085
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020/validation_results.yml
Validation ethiopia2020: 0.41466661987432524
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013/validation_results.yml
Validation car2013: 0.3009028337547685
Mean score: 0.3589699716053818
csstddg@LIN-LAPTOP-DG:~/FabSim3/plugins/FabFlee/config_files/validation$ cat simsetting.yml 
log_levels:
  agent: 0
  camp: 0
  init: 0
spawn_rules:
  take_from_population: True
  conflict_spawn_decay: [1.0,1.0,1.0,0.5,0.1]
move_rules:
  start_on_foot: True
  avoid_short_stints: True
  awareness_level: 1
optimisations:
djgroen commented 1 year ago

This is the last validation check I am doing before tagging the release. We may be able to debug this later on, but this is where we are right now :).

[Executing task] validate_flee_output
calling task validate_flee_output from plugin FabFlee
╭─ New/Updated environment variables from FabFlee plugin ─╮
│ env :                                                   │
│   --- local_system_time : 1689345389                    │
│   +++ local_system_time : 1689346640                    │
╰─────────────────────────────────────────────────────────╯
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/nigeria2016/validation_results.yml
Validation nigeria2016: 0.16762030001324854
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/mali2012/validation_results.yml
Validation mali2012: 0.2573675869680532
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/syria2013/validation_results.yml
Validation syria2013: 0.28194738209841097
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2013/validation_results.yml
Validation ssudan2013: 0.33050711533277566
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/burundi2015/validation_results.yml
Validation burundi2015: 0.4766032607013991
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ssudan2016/validation_results.yml
Validation ssudan2016: 0.6375645853483395
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/ethiopia2020/validation_results.yml
Validation ethiopia2020: 0.41222488116946476
/home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013
[local] export PYTHONPATH=/home/csstddg/flee:${PYTHONPATH}; export FLEE_TYPE_CHECK=False; python3 /home/csstddg/flee/flee/postprocessing/extract-validation-results.py /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013 > /home/csstddg/FabSim3/results/validation_archer2_24/RUNS/car2013/validation_results.yml
Validation car2013: 0.28183407096096236
Mean score: 0.3557086478240818