NathanielF commented 1 month ago

CFA and SEM with PyMC

Related to this issue here I'm adding a PR to demonstrate the functionality of CFA and SEM models using PyMC https://github.com/pymc-devs/pymc-examples/issues/695

[X] Notebook follows style guide https://docs.pymc.io/en/latest/contributing/jupyter_style.html
[X] PR description contains a link to the relevant issue:
- a tracker one for existing notebooks (tracker issues have the "tracker id" label)
- or a proposal one for new notebooks
[x] Check the notebook is not excluded from any pre-commit check: https://github.com/pymc-devs/pymc-examples/blob/main/.pre-commit-config.yaml

Helpful links

https://github.com/pymc-devs/pymc-examples/blob/main/CONTRIBUTING.md

📚 Documentation preview 📚: https://pymc-examples--698.org.readthedocs.build/en/698/

review-notebook-app[bot] commented 1 month ago

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

NathanielF commented 3 weeks ago

Think this is ready for review @drbenvincent , @cluhmann , @AlexAndorra . Any feedback/pushback welcome.

NathanielF commented 2 weeks ago

Giving this another nudge!

Thanks!

drbenvincent commented 2 weeks ago

FYI: I'm totally overloaded at the moment, so going to struggle with a review. Will see if things ease up a bit next week.

NathanielF commented 2 weeks ago

No worries.

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:46Z ----------------------------------------------------------------

Perhaps add reference to psychometrics in the title to make it more specific, and aid in discoverability.

NathanielF commented on 2024-09-25T11:38:08Z ----------------------------------------------------------------

Good idea.

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:47Z ----------------------------------------------------------------

nitpick: data set should not have a hyphen; data set or dataset, no?

NathanielF commented on 2024-09-25T11:38:41Z ----------------------------------------------------------------

changed.

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:47Z ----------------------------------------------------------------

Maybe hyperlink SEM and CFA to some introductory materials on the topics (even a Wikipedia link would be good)

NathanielF commented on 2024-09-25T11:38:54Z ----------------------------------------------------------------

Added links to wikipedia.

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:48Z ----------------------------------------------------------------

There are lots of UserWarnings in the notebook. Consider filtering them out.

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:49Z ----------------------------------------------------------------

The last line of the equation set contains a typo -- should be \psi_n rather than \psi_3

NathanielF commented on 2024-09-25T11:39:29Z ----------------------------------------------------------------

Fixed. Thanks, good catch!

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:50Z ----------------------------------------------------------------

Line #29.        chol, _, _ = pm.LKJCholeskyCov("chol_cov", n=2, eta=2, sd_dist=sd_dist, compute_corr=True)

Why do you set compute_corr=True and discard the correlations?

_NathanielF commented on 2024-09-25T11:40:37Z_ ----------------------------------------------------------------

It's not used here in this model, but useful to have computed for analysing the correlations between the constructs post model fit. I demonstrate this below for the full measurement model....

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:50Z ----------------------------------------------------------------

Cool plots!

NathanielF commented on 2024-09-25T11:45:43Z ----------------------------------------------------------------

Thanks!

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:51Z ----------------------------------------------------------------

Line #81.            draws=10000,

Why so many draws? Bad autocorrelation?

_NathanielF commented on 2024-09-25T11:42:40Z_ ----------------------------------------------------------------

Yes, it seemed to help generally boost the ESS. The model samples pretty fast so it was easy enough to just boost the samples.

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:52Z ----------------------------------------------------------------

typo: reflected

NathanielF commented on 2024-09-25T11:42:50Z ----------------------------------------------------------------

Fixed.

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:53Z ----------------------------------------------------------------

typo in first sentence: individual

NathanielF commented on 2024-09-25T11:42:59Z ----------------------------------------------------------------

Fixed.

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:53Z ----------------------------------------------------------------

Typo: each

NathanielF commented on 2024-09-25T11:43:11Z ----------------------------------------------------------------

Fixed

review-notebook-app[bot] commented 1 week ago

View / edit / reply to this conversation on ReviewNB

fonnesbeck commented on 2024-09-25T03:08:54Z ----------------------------------------------------------------

What about measures of relative fit among the models? What do the LOO/WAIC values look like, and are they helpful in selecting the appropriate level of complexity?

NathanielF commented on 2024-09-25T11:45:26Z ----------------------------------------------------------------

Added the global fit comparison for the SEM and full measurement models. The LOO metrics are quite close, but the full measurement model "wins" on this score. I added a note to say we need to consider the value of the trade-off between expressive power of the SEMs and the questions we can answer in the context of simple global fit comparisons.