search

stan-dev / loo

loo R package for approximate leave-one-out cross-validation (LOO-CV) and Pareto smoothed importance sampling (PSIS)
https://mc-stan.org/loo
Other
150 stars 34 forks source link

Error when running loo with moment_match = TRUE on rstan model from previous session #183

Open jonathanicholas opened 3 years ago

jonathanicholas commented 3 years ago

Hi,

I'm trying to run loo with moment matching on an rstan model fit in a previous session. The stanfit object was saved using:

fit <- stan(...)
saveRDS(fit,...)

And then loaded and loo (from https://github.com/stan-dev/rstan/blob/develop/rstan/rstan/R/loo.R) was run with:

fit <- loadRDS(...)
loo(fit, moment_match = TRUE)

which throws the following error:

Error in .local(object, ...) : 
  the model object is not created or not valid
Calls: loo ... apply -> FUN -> <Anonymous> -> <Anonymous> -> .local
Execution halted

Fitting and loading are both happening on the same system (university HPC cluster). I have no problems loading the stanfit object and then running other functions on this system (running extract(fit) for example) or loading the stanfit object on my local system (Mac OS 11.2.1). This also works fine if the model is fit and then loo is run within the same session (i.e. not from a .rds file).

HPC Cluster specs:

cat /etc/*-release

Cluster Manager v9.1
slave
NAME="Red Hat Enterprise Linux"
VERSION="8.2 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.2"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.2 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.2"
Red Hat Enterprise Linux release 8.2 (Ootpa)
Red Hat Enterprise Linux release 8.2 (Ootpa)

R environment specs:

R version 4.1.0 (2021-05-18) -- "Camp Pontanezen"
rstan (version 2.21.2)

Thanks for your help and developing such an awesome package! If this is an rstan and not a loo problem I'm happy to move the issue there.

jonaudet commented 3 years ago

Hi, I seem to be having a similar issue (Ubuntu 20.04, R 4.1.1, RStan 2.21.2). Compiling a model, using it, and then calling loo with moment_match = T is fine. But saving the results to RDS and reading them in a new session fails. Recompiling the model under the same object name as before (assuming the sampling was done using sampling rather than stan) also fails.

This also means that moment matching can't be used on the results of cmdstan. ETA: loo doesn't work directly on cmdstanfit objects, but the moment matching fails after transforming to a stanfit object.

topipa commented 3 years ago

We are aware of this issue and it is a general problem with rstan that is not directly related to the loo package. There are certain methods that rstan uses for moment matching that don’t work after reloading a model in a new session. Unfortunately there are no straightforward workarounds at the moment, so moment matching for rstan fits needs to be done in the same R session as sampling.

jonaudet commented 3 years ago

Ok, thanks for the clarification. I assume that is also the reason moment_matching can't currently work with cmdstan.

Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Topi Paananen @.> Sent: Friday, August 27, 2021 9:36:39 AM To: stan-dev/loo @.> Cc: jonaudet @.>; Manual @.> Subject: Re: [stan-dev/loo] Error when running loo with moment_match = TRUE on rstan model from previous session (#183)

We are aware of this issue and it is a general problem with rstan that is not directly related to the loo package. There are certain methods that rstan uses for moment matching that don’t work after reloading a model in a new session. Unfortunately there are no straightforward workarounds at the moment, so moment matching for rstan fits needs to be done in the same R session as sampling.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fstan-dev%2Floo%2Fissues%2F183%23issuecomment-907250717&data=04%7C01%7C%7C4704a2172c664a49643d08d9696814e8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637656718009288422%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rn726eT99BZplEAHeV%2BRTPkZ32Iqb%2F3ryFTmFg05CDI%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACINOUVZHI2VJBAN2FAE5Q3T66PHPANCNFSM5BLIJ4UA&data=04%7C01%7C%7C4704a2172c664a49643d08d9696814e8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637656718009298412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NRaZxRPY4bVvIEP5su9vh9obgOrgKrcLBRQbr5aHBbY%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7C%7C4704a2172c664a49643d08d9696814e8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637656718009298412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QZZmt2P0ww%2B5%2F7I5qgrCo4ZCLWjlt9nUmxnk1OfrZRE%3D&reserved=0 or Androidhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7C%7C4704a2172c664a49643d08d9696814e8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637656718009308409%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=10wYigrizvCvc7740DnbZd%2BupoRx%2FcI5wzJhuyUdVuU%3D&reserved=0.