chjackson / msm

The msm R package for continuous-time multi-state modelling of panel data
https://chjackson.github.io/msm/
57 stars 16 forks source link

Too many paths causing errors? #101

Open 26pan opened 2 months ago

26pan commented 2 months ago

Dear Professor: First of all, thank you so much for always answering our questions on the forums, it's much appreciated. I have been using "msm" for almost half a year now and I am currently experiencing a problem. The polymorphic model I built before was 5 states and 8 paths, at first I also encountered the problem that the results could not be fitted, but after adjusting some parameters in the code, it works fine. Currently I have built a new model which contains 9 states and 20 paths, the flowchart and sample size are as follows. At first I used my own laptop to do the calculations, but the computer crashed and showed that there were 356mb of content that could not be processed. I then switched to a more capable desktop computer to run it, but after about 7/8 hours, there were still no results, and R popped up with a message saying that it had encountered some errors and needed to restart the device. So I have some questions, is it because there are too many paths and the computation is too large for the computer to handle, or is it because there are too many paths and not enough samples for the model to fit? I hope the professor can give me some suggestions to solve the problem, thank you very much for your reply.

Sincerely, Pan. ![Uploading 图片1.png…]()

chjackson commented 2 months ago

The time to convergence of an msm model depends on the number of likelihood evaluations needed and the speed of a single likelihood evaluation.

The time of a likelihood evaluation strongly depends on the number of states (it's proportional to the square of the number of states).

The number of likelihood evalutions needed depends on the number of parameters in the model, but also on the data. If there are too many parameters for the data then the likelihood will be near-flat and many more evaluations will be needed.

I don't know what to suggest here other than the usual advice to get something simple working first, then build it up!

26pan commented 2 months ago

Thank you very much!

---Original--- From: "Chris @.> Date: Thu, Jun 27, 2024 00:16 AM To: @.>; Cc: @.**@.>; Subject: Re: [chjackson/msm] Too many paths causing errors? (Issue #101)

The time to convergence of an msm model depends on the number of likelihood evaluations needed and the speed of a single likelihood evaluation.

The time of a likelihood evaluation strongly depends on the number of states (it's proportional to the square of the number of states).

The number of likelihood evalutions needed depends on the number of parameters in the model, but also on the data. If there are too many parameters for the data then the likelihood will be near-flat and many more evaluations will be needed.

I don't know what to suggest here other than the usual advice to get something simple working first, then build it up!

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>