updating data generation pipeline to python 3.11?

Leeds-MRG / Minos

SIPHER Microsimulation for estimating the effect on Income policy on mental health.

MIT License

4 stars 3 forks source link

updating data generation pipeline to python 3.11? #392

Open RobertClay opened 7 months ago

RobertClay commented 7 months ago

as we're hoping on being able to run R3.4 any day now it might be nice to try and upgrade the python version to a more faster modern version. will require some debugging of old pandas mainly. should just be an afternoon of painful debugging.

paddy-r commented 7 months ago

@RobertClay also see #159 from yonks ago...

RobertClay commented 7 months ago

did you get very far? can do this if you really want..

paddy-r commented 7 months ago

No, never did it, was just sticking a pin in it, so might be best to close that one, as this supersedes it.

RobertClay commented 6 months ago

raw data

just the latin1 warning. think this is dodgy stata file from UKHLS?

locf imputation

replacing .replace() method. we want to use fillna because its fastest. however this also gives deprecated warnings so going to use .ffill() method for now.
also now dont replace NA each time in the groupby apply function. replace all NA values at the start of imputation and put back any values that aren't filled in at the end. seems to be faster.
missingness description function requires axis in row sum.

composites

housing quality and neighbourhood safety and marital status all default to -9 (integer). making it a string "-9" instead stops dtype warning.
casting phealth to float rather than int. theres a division between float and int throwing a warning so this just seems more consistent.

final us

supress settingwithcopywarning as we've done with composites.
wave copy uses different default depending on dtype.

RobertClay commented 6 months ago

actually mananged to get a stable env working with python 3.11 and R4.3.3. runs all the way through to transition models with an extra step to get lme4 installed.

create env activate env R install.packages("lme4", type="source") library(lme4)#check its install and run a little test model. should see some output. lmer(Reaction ~ Days + (1|Subject), sleepstudy) Linear mixed model fit by REML ['lmerMod'] Formula: Reaction ~ Days + (1 | Subject) Data: sleepstudy REML criterion at convergence: 1786.465 Random effects: Groups Name Std.Dev. Subject (Intercept) 37.12
Residual 30.99
Number of obs: 180, groups: Subject, 18 Fixed Effects: (Intercept) Days
251.41 10.47

make transitions_default

RobertClay commented 6 months ago

anyway, shelving this for now. getting new vivarium working seems like a decent sized job for another issue.

RobertClay commented 6 months ago

https://github.com/lme4/lme4/issues/763