Does Python/R model automatically create outputs directory by default?

[Sparrow0hawk]

Description:

Running the python/R model without a devon_data/outputs directory leads to an error.

Traceback:

(ramp-ua) $ python microsim/main.py 
Reading parameters file: ./model_parameters/default.yml. Any other model-related command-line arguments are being ignored
Running model with the following parameters:
        Parameters file: ./model_parameters/default.yml
        Scenario directory: BaselineScenario
        Initialise (and then exit?): False
        Number of iterations: 80
        Data dir: devon_data
        Using quant data? yes: QUANT_RAMP
        Outputting results?: True
        Outputting results at every iteration?: False
        Debug mode?: False
        Number of repetitions: 1
        Lockdown file: google_mobility_lockdown_daily_14_day_moving_average.csv
        Use cache?: True
        Use OpenCL version?: False
        Use OpenCL GUI?: False
        Use OpenCL GPU for processing?: False
        Calibration parameters: {'hazard_individual_multipliers': {'presymptomatic': 1.0, 'asymptomatic': 0.25, 'symptomatic': 1.0}, 'hazard_location_multipliers': {'Retail': 1.0, 'PrimarySchool': 1.0, 'SecondarySchool': 1.0, 'Home': 1.0, 'Work': 1.0}, 'risk_multiplier': 1.0}
        Disease parameters: {'current_risk_beta': 0.00555, 'risk_cap': 5, 'seed_days': 5, 'exposed_dist': 'weibull', 'exposed_mean': 2.56, 'exposed_sd': 0.72, 'presymp_dist': 'weibull', 'presymp_mean': 2.3, 'presymp_sd': 0.35, 'infection_dist': 'lognormal', 'infection_mean': 17, 'infection_sd': 1.3, 'output_switch': True, 'rank_assign': False, 'local_outbreak_timestep': 7, 'local_outbreak': False, 'msoa_infect': 'E02004161', 'number_people_local': 100, 'local_prob_increase': 1.0, 'overweight_sympt_mplier': 1, 'obesity_40': 1.48, 'obesity_35': 1.48, 'obesity_30': 1.48, 'overweight': 1, 'cvd': 1, 'diabetes': 1, 'bloodpressure': 1, 'improve_health': False, 'set_seed': False}

Using QUANT data
/home/medacola/Code/forks/RAMP-UA-fork/devon_data/QUANT_RAMP
Loading data from previous cache
Implementing a lockdown with time activities from google_mobility_lockdown_daily_14_day_moving_average.csv
Reading time activity multiplier data from /home/medacola/Code/forks/RAMP-UA-fork/devon_data/google_mobility_lockdown_daily_14_day_moving_average.csv...

Running Python / R model
Traceback (most recent call last):
  File "microsim/main.py", line 326, in <module>
    main()
  File "/home/medacola/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/medacola/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/medacola/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/medacola/anaconda3/envs/ramp-ua/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "microsim/main.py", line 225, in main
    repetitions, parameters_file)
  File "microsim/main.py", line 265, in run_python_model
    m = Microsim(individuals_df, activity_locations_df, time_activity_multiplier, **msim_args)
  File "/home/medacola/Code/forks/RAMP-UA/microsim/microsim_model.py", line 80, in __init__
    self.SCEN_DIR = self._find_new_directory(os.path.join(self.DATA_DIR, "output"), self.SCEN_DIR)
  File "/home/medacola/Code/forks/RAMP-UA/microsim/microsim_model.py", line 584, in _find_new_directory
    os.mkdir(results_subdir)
FileNotFoundError: [Errno 2] No such file or directory: '/home/medacola/Code/forks/RAMP-UA-fork/devon_data/output/BaselineScenario'

Diagnosis:

This suggests to me that the Python/R model doesn't create the outputs directory by default (I note the OpenCL model does, and that the downloaded devon_data erroneously contains an outputs directory which masks this problem).

I think this arises from using os.mkdir here https://github.com/Urban-Analytics/RAMP-UA/blob/9c2491a89929487d41f04166e813ac9fe1767e40/microsim/microsim_model.py#L584 Which doesn't create a directory and parent directories (need os.mkdirs for that).

Steps to reproduce:

Running the model from the master branch if I do:

$ cd RAMP-UA

# remove or move the outputs folder
# to delete: rm -r devon_data/outputs
# to move: mv devon_data/outputs devon_data/outputs-mvd

$ python microsim/main.py

@nickmalleson can you attempt to reproduce this?