search

simon-smart88 / disagapp

R shiny app for disaggregation regression
https://simon-smart88.github.io/disagapp/
0 stars 0 forks source link

Fitting large models crashes Rstudio #23

Open simon-smart88 opened 5 months ago

simon-smart88 commented 5 months ago

E.g. Fitting a model for Nigeria with 1.5 million pixels

Works fine on the same data run outside the app, and works fine if the app is launched from command line

simon-smart88 commented 5 months ago

Decreased the resolution and it still happened, so maybe it is related to the shapefile, which is also causing issues for cov_landuse

simon-smart88 commented 3 months ago

@timcdlucas - I've encountered this again today and figured out that it's seemingly caused by running out of RAM so we might want to see if there is anything we can do to stop disaggregation::disgag_model being so greedy (I have 16 GB but only about 5GB was unused when I went to fit)

timcdlucas commented 3 months ago

We can have a look but I doubt we'll be able to decrease the RAM much. The models just are big. Anita spent a decent chunk of time look at how we could reduce the size of models and didn't really get anywhere. It's just the nature of the beast really.

I'm not sure if we can trycatch to make it fail more gracefully, or use a few simple heuristics to guess the final model size. Or use sensible documentation to guide people.

Two easy things that will probably drastically change model size are fitting without a field and (possible) without iid terms. So guidance to fit without first might be useful for example.

simon-smart88 commented 3 months ago

Yes I think it's more trying to fail more gracefully that would be the main help as crashing R is a bit extreme. I'll have a look into potential ways to deal with it.