search

Urban-Analytics / RAMP-UA

https://urban-analytics.github.io/RAMP-UA/
MIT License
15 stars 10 forks source link

Microsim model run error on Windows #73

Open Sparrow0hawk opened 4 years ago

Sparrow0hawk commented 4 years ago

Problem and stacktrace

Testing out the microsim/microsim_model.py from master branch (commit d2d00afdc81de430b63bb2d4eedeba99100145cc) on Windows leads to a rpy2 error.

$ python microsim/microsim_model.py Traceback (most recent call last): File "microsim/microsim_model.py", line 15, in from r_interface import RInterface File "C:\Users\medacola\Code\forks\RAMP-UA\microsim\r_interface.py", line 3, in import rpy2.rinterface File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\rinterface.py", line 6, in from rpy2.rinterface_lib import openrlib File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\rinterface_lib\openrlib.py", line 44, in rlib = _dlopen_rlib(R_HOME) File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\rinterface_lib\openrlib.py", line 37, in _dlopen_rlib rlib = ffi.dlopen(lib_path) OSError: cannot load library 'C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\R\bin\x64\R.dll': error 0x7e

This look like it can't see R installed within the conda environment.

Steps to reproduce

On windows 10 via git-bash:

conda activate ramp-ua
cd RAMP-UA
python microsim/microsim_model.py

I've previously created the conda environment using the environment.yml and pulled in data using git lfs checkout.

Sparrow0hawk commented 4 years ago

Replicated within github actions in above commit.

nnelissen commented 4 years ago

@Sparrow0hawk I had the same error on Windows 10 (the reason I switched to Linux). os.path.exists(lib_path) is true so python sees the path/file, but can't read it?

Sparrow0hawk commented 4 years ago

OK so reverting to rpy2=2.9.4 resolves the OSError.

Tests start to run but now fail on test_random. Something i'll create a new issue for.

Sparrow0hawk commented 4 years ago

@nnelissen so looks like you can resolve this issue by rolling back the version of rpy2 installed. My fix has updated the environment.yml which once merged into master you can rebase into your current local branch. Otherwise you can do:

conda activate ramp-ua
conda install rpy2=2.9.4

To get the working version. Let me know if it then works for you!

nnelissen commented 4 years ago

@Sparrow0hawk I tried and it gives another error now, after (seemingly normal) initialisation, around the time it should switch to the R scripts. It looks like it's loops of (the same?) errors which crash the terminal, I'll try to attach a screenshot in case anything looks familiar error

Sparrow0hawk commented 4 years ago

@nnelissen Hmm this looks tied into the multiprocessing library. Can you copy and paste in the full stacktrace?

I'm unable to replicate but get another error, which looks similar to our initial problems:


> $ python microsim/microsim_model.py
> C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a
>  future version.  The public classes are available in the top-level namespace.
>   from pandas.core.index import Index as PandasIndex
> Reading parameters file: ./model_parameters/default.yml. Any other command-line arguments are being ignored
> Running model with the following parameters:
>         Parameters file: ./model_parameters/default.yml
>         Number of iterations: 10
>         Data dir: devon_data
>         Outputting results?: True
>         Debug mode?: False
>         Number of repetitions: 1
>         Lockdown from file? : True
>         Calibration parameters: {'hazard_multiplier_presymptomatic': 1.0, 'hazard_multiplier_asymptomatic': 1.0, 'hazard_multiplier_symptomatic': 1.0, 'risk_multiplier': 1.0}
> 
> Reading time use and health data ...
> microsim/microsim_model.py:440: UserWarning: There are 6 / 695308 non-unique individuals.
>   warnings.warn(f"There are {len(tuh)-len(set(unique_individuals))} / {len(tuh)} non-unique individuals.")
> microsim/microsim_model.py:489: UserWarning: 32 people > 18y/o go to school, but they are not being assigned to a primary or secondary school (so their schooling is ignored at the momen
> t
>   warnings.warn(f"{len(adults_in_school)} people > 18y/o go to school, but they are not being assigned to a "
> microsim/microsim_model.py:500: UserWarning: There are 0 households with more than 10 people in them. This covers 0 people. These households are being removed.
>   warnings.warn(f"There are {len(large_house_idx)} households with more than 10 people in them. This covers "
> ... finished reading TU&H data. There are 695308 individuals in 346343 houses over 107 MSOAs
> Reading retail flow data for Devon...
> ... read 106990.0 flows from 107 areas.
> Assigning individual flows for Retail: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [00:05<00:00, 18.92it/s]
> Reading school flow data for Devon...
> ... finished reading school flows.
> Assigning individual flows for PrimarySchool: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [00:06<00:00, 16.94it/s]
> Assigning individual flows for SecondarySchool: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [00:05<00:00, 18.53it/s]
> Assigning work venues ...
> C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a
>  future version.  The public classes are available in the top-level namespace.
>   from pandas.core.index import Index as PandasIndex
> C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a
>  future version.  The public classes are available in the top-level namespace.
>   from pandas.core.index import Index as PandasIndex
> C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a
>  future version.  The public classes are available in the top-level namespace.
>   from pandas.core.index import Index as PandasIndex
> C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a
>  future version.  The public classes are available in the top-level namespace.
>   from pandas.core.index import Index as PandasIndex
> Rounding all durations: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:03<00:00,  1.34it/s]
> Reading time activity multiplier data...
>  ... finished initialisation.
> Saving initial models for analysis ...
> Initialising R interface. Loading R scripts in C:\Users\medacola\Code\forks\RAMP-UA\R\py_int.
> package is up-to-date devel version
> 
> Attaching package: 'dplyr'
> 
> The following objects are masked from 'package:stats':
> 
>     filter, lag
> 
> The following objects are masked from 'package:base':
> 
>     intersect, setdiff, setequal, union
> 
> Registered S3 methods overwritten by 'ffbase':
>   method   from
>   [.ff     ff
>   [.ffdf   ff
>   [<-.ff   ff
>   [<-.ffdf ff
> 
> Iteration: 1
> 
>         Applying regular (google mobility) multiplier 0.9305173110479441
>         Updating danger associated with visiting each venue
>                 Home activity
>                 Retail activity
>                 PrimarySchool activity
>                 SecondarySchool activity
>                 Work activity
>         Calculating new disease status...[1] "R timestep: 1"
> Error in file(file, ifelse(append, "a", "w")) :
>   cannot open the connection
> In addition: Warning messages:
> 1: In dir.create(tmp.dir, recursive = TRUE) :
>   cannot create dir 'C:\Users\medacola\Code\forks\RAMP-UA\R\py_int\output\2020-08-03 13:45:34', reason 'Invalid argument'
> 2: In file(file, ifelse(append, "a", "w")) :
>   cannot open file 'C:/Users/medacola/Code/forks/RAMP-UA/R/py_int/output/2020-08-03 13:45:34/daily_1.csv': Invalid argument
> Traceback (most recent call last):
>   File "microsim/microsim_model.py", line 1619, in <module>
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 829, in __call__
>     return self.main(*args, **kwargs)
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 782, in main
>     rv = self.invoke(ctx)
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 1066, in invoke
>     return ctx.invoke(self.callback, **ctx.params)
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 610, in invoke
>     return callback(*args, **kwargs)
>   File "microsim/microsim_model.py", line 1595, in run_script
>   File "microsim/microsim_model.py", line 1452, in run
>   File "microsim/microsim_model.py", line 1433, in step
>   File "microsim/microsim_model.py", line 1306, in calculate_new_disease_status
>   File "C:\Users\medacola\Code\forks\RAMP-UA\microsim\r_interface.py", line 60, in calculate_disease_status
>     r_df = self.R.run_status(individuals_reduced, iteration)  # This gets converted to a pandas dataframe implicitly
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\functions.py", line 178, in __call__
>     return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\functions.py", line 106, in __call__
>     res = super(Function, self).__call__(*new_args, **new_kwargs)
> rpy2.rinterface.RRuntimeError: Error in file(file, ifelse(append, "a", "w")) :
>   cannot open the connection
jabrams23 commented 4 years ago

Hey all,

This is an error caused by some fo the R packages that we load. These packages were updated a few days ago and since then we have been getting an error. Fiona solved it temporarily on another branch by removing a few functions that were used to process the mobility data. But we would like to have those functions as part of the R package. Maybe we can solve it by installing older versions of the packages?

Fiona and I are busy all week with a “Grand Challenge” with Gavin and the CDT so I don’t think we will find time to work on this.

On 3 Aug 2020, at 13:58, Alex Coleman notifications@github.com wrote:

Hmm this looks tied into the multiprocessing library. Can you copy and paste in the full stacktrace?

I'm unable to replicate but get another error: @nnelissen https://github.com/nnelissen I think I replicate this:

$ python microsim/microsim_model.py C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace. from pandas.core.index import Index as PandasIndex Reading parameters file: ./model_parameters/default.yml. Any other command-line arguments are being ignored Running model with the following parameters: Parameters file: ./model_parameters/default.yml Number of iterations: 10 Data dir: devon_data Outputting results?: True Debug mode?: False Number of repetitions: 1 Lockdown from file? : True Calibration parameters: {'hazard_multiplier_presymptomatic': 1.0, 'hazard_multiplier_asymptomatic': 1.0, 'hazard_multiplier_symptomatic': 1.0, 'risk_multiplier': 1.0}

Reading time use and health data ... microsim/microsim_model.py:440: UserWarning: There are 6 / 695308 non-unique individuals. warnings.warn(f"There are {len(tuh)-len(set(unique_individuals))} / {len(tuh)} non-unique individuals.") microsim/microsim_model.py:489: UserWarning: 32 people > 18y/o go to school, but they are not being assigned to a primary or secondary school (so their schooling is ignored at the momen t warnings.warn(f"{len(adults_in_school)} people > 18y/o go to school, but they are not being assigned to a " microsim/microsim_model.py:500: UserWarning: There are 0 households with more than 10 people in them. This covers 0 people. These households are being removed. warnings.warn(f"There are {len(large_house_idx)} households with more than 10 people in them. This covers " ... finished reading TU&H data. There are 695308 individuals in 346343 houses over 107 MSOAs Reading retail flow data for Devon... ... read 106990.0 flows from 107 areas. Assigning individual flows for Retail: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [00:05<00:00, 18.92it/s] Reading school flow data for Devon... ... finished reading school flows. Assigning individual flows for PrimarySchool: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [00:06<00:00, 16.94it/s] Assigning individual flows for SecondarySchool: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [00:05<00:00, 18.53it/s] Assigning work venues ... C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace. from pandas.core.index import Index as PandasIndex C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace. from pandas.core.index import Index as PandasIndex C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace. from pandas.core.index import Index as PandasIndex C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\pandas2ri.py:17: FutureWarning: pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace. from pandas.core.index import Index as PandasIndex Rounding all durations: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:03<00:00, 1.34it/s] Reading time activity multiplier data... ... finished initialisation. Saving initial models for analysis ... Initialising R interface. Loading R scripts in C:\Users\medacola\Code\forks\RAMP-UA\R\py_int. package is up-to-date devel version

Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

filter, lag

The following objects are masked from 'package:base':

intersect, setdiff, setequal, union

Registered S3 methods overwritten by 'ffbase': method from [.ff ff [.ffdf ff [<-.ff ff [<-.ffdf ff

Iteration: 1

    Applying regular (google mobility) multiplier 0.9305173110479441
    Updating danger associated with visiting each venue
            Home activity
            Retail activity
            PrimarySchool activity
            SecondarySchool activity
            Work activity
    Calculating new disease status...[1] "R timestep: 1"

Error in file(file, ifelse(append, "a", "w")) : cannot open the connection In addition: Warning messages: 1: In dir.create(tmp.dir, recursive = TRUE) : cannot create dir 'C:\Users\medacola\Code\forks\RAMP-UA\R\py_int\output\2020-08-03 13:45:34', reason 'Invalid argument' 2: In file(file, ifelse(append, "a", "w")) : cannot open file 'C:/Users/medacola/Code/forks/RAMP-UA/R/py_int/output/2020-08-03 13:45:34/daily_1.csv': Invalid argument Traceback (most recent call last): File "microsim/microsim_model.py", line 1619, in File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 829, in call return self.main(args, kwargs) File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 782, in main rv = self.invoke(ctx) File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 610, in invoke return callback(args, kwargs) File "microsim/microsim_model.py", line 1595, in run_script File "microsim/microsim_model.py", line 1452, in run File "microsim/microsim_model.py", line 1433, in step File "microsim/microsim_model.py", line 1306, in calculate_new_disease_status File "C:\Users\medacola\Code\forks\RAMP-UA\microsim\r_interface.py", line 60, in calculate_disease_status r_df = self.R.run_status(individuals_reduced, iteration) # This gets converted to a pandas dataframe implicitly File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\functions.py", line 178, in call return super(SignatureTranslatedFunction, self).call(*args, *kwargs) File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\functions.py", line 106, in call res = super(Function, self).call(new_args, new_kwargs) rpy2.rinterface.RRuntimeError: Error in file(file, ifelse(append, "a", "w")) : cannot open the connection

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Urban-Analytics/RAMP-UA/issues/73#issuecomment-668007944, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEUDMAJNZCBBOMA3ECF3VEDR62X65ANCNFSM4PN2WAJQ.

Sparrow0hawk commented 4 years ago

@jabrams23

No problem, i'll dedicate my time on thursday to taking a look at this.

spoonerf commented 4 years ago

Hi All,

I think this error may be something different - I think a read.csv line I may have left in accidentally.

Going to do a pull request which should be a temporary fix for the package problems and for this issue. Essentially just using the R package from a branch which doesn't have the problematic packages.

Cheers,

Fiona

Sparrow0hawk commented 4 years ago

Error when running locally on Windows with #77 


> Iteration: 1
> 
>         Applying regular (google mobility) multiplier 0.9305173110479441
>         Updating danger associated with visiting each venue
>                 Home activity
>                 Retail activity
>                 PrimarySchool activity
>                 SecondarySchool activity
>                 Work activity
>         Calculating new disease status...[1] "R timestep: 1"
> Error in file(file, ifelse(append, "a", "w")) :
>   cannot open the connection
> In addition: Warning messages:
> 1: In dir.create(tmp.dir, recursive = TRUE) :
>   cannot create dir 'C:\Users\medacola\Code\forks\RAMP-UA\R\py_int\output\2020-08-03 15:51:31', reason 'Invalid argument'
> 2: In file(file, ifelse(append, "a", "w")) :
>   cannot open file 'C:/Users/medacola/Code/forks/RAMP-UA/R/py_int/output/2020-08-03 15:51:31/daily_1.csv': Invalid argument
> Traceback (most recent call last):
>   File "microsim/microsim_model.py", line 1619, in <module>
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 829, in __ca
> ll__
>     return self.main(*args, **kwargs)
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 782, in main
>     rv = self.invoke(ctx)
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 1066, in inv
> oke
>     return ctx.invoke(self.callback, **ctx.params)
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\click\core.py", line 610, in invo
> ke
>     return callback(*args, **kwargs)
>   File "microsim/microsim_model.py", line 1595, in run_script
>   File "microsim/microsim_model.py", line 1452, in run
>   File "microsim/microsim_model.py", line 1433, in step
>   File "microsim/microsim_model.py", line 1306, in calculate_new_disease_status
>   File "C:\Users\medacola\Code\forks\RAMP-UA\microsim\r_interface.py", line 60, in calculate_disease_status
>     r_df = self.R.run_status(individuals_reduced, iteration)  # This gets converted to a pandas dataframe implicitly
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\functions.py", line
>  178, in __call__
>     return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
>   File "C:\Users\medacola\AppData\Local\Continuum\anaconda3\envs\ramp-ua\lib\site-packages\rpy2\robjects\functions.py", line
>  106, in __call__
>     res = super(Function, self).__call__(*new_args, **new_kwargs)
> rpy2.rinterface.RRuntimeError: Error in file(file, ifelse(append, "a", "w")) :
>   cannot open the connection
Sparrow0hawk commented 4 years ago

This is an error with R code within covid_run.R, specifically within the creation of output directories

The crux of this problem is Sys.time() returns a timestamp that includes colons which is an invalid character in Windows file paths (can only be used for specifying partitions i.e. C:\).

I've resolved this in some upcoming commits.

Sparrow0hawk commented 4 years ago

Reopening this issue given problems identified in #79 and reversion to newer rpy2 version.

I'll look into a fix but at this stage I'd advise people to not run the model on Windows given this issue.