search

stan-dev / cmdstanpy

CmdStanPy is a lightweight interface to Stan for Python users which provides the necessary objects and functions to compile a Stan program and fit the model to data using CmdStan.
BSD 3-Clause "New" or "Revised" License
151 stars 69 forks source link

cmdstanpy yields an error when sampling is completed "CmdStan error: terminated by signal 6 Unknown error -6" but from_csv reconstructs the fit #754

Open Garren-H opened 4 months ago

Garren-H commented 4 months ago

Description:

Hi all, I have ran into another issue (on a different cluster) when running my stan code. The sampling executes as expected but afterwards an error is displayed

The stan .txt output file yields: ``` method = sample (Default) sample num_samples = 1000 (Default) num_warmup = 5000 save_warmup = 0 (Default) thin = 1 (Default) adapt engaged = 1 (Default) gamma = 0.050000 (Default) delta = 0.800000 (Default) kappa = 0.750000 (Default) t0 = 10.000000 (Default) init_buffer = 75 (Default) term_buffer = 50 (Default) window = 25 (Default) save_metric = 0 (Default) algorithm = hmc (Default) hmc engine = nuts (Default) nuts max_depth = 5 metric = dense_e metric_file = (Default) stepsize = 1.000000 (Default) stepsize_jitter = 0.000000 (Default) num_chains = 8 id = 1 (Default) data file = Subsets/Alkane_Primary alcohol/Include_clusters_False/Variance_known_True/Variance_MC_known_True/rank_2/data.json init = 2 (Default) random seed = 24898 output file = /home/22796002/Pure RK PMF/Subsets/Alkane_Primary alcohol/Include_clusters_False/Variance_known_True/Variance_MC_known_True/rank_2/Step1/Pure_PMF-20240522025118.csv diagnostic_file = (Default) refresh = 100 (Default) sig_figs = 18 profile_file = /home/22796002/Pure RK PMF/Subsets/Alkane_Primary alcohol/Include_clusters_False/Variance_known_True/Variance_MC_known_True/rank_2/Step1/Pure_PMF-20240522025118-profile.csv save_cmdstan_config = 0 (Default) num_threads = 32 (Default) Gradient evaluation took 0.021501 seconds 1000 transitions using 10 leapfrog steps per transition would take 215.01 seconds. Adjust your expectations accordingly! Gradient evaluation took 0.017354 seconds 1000 transitions using 10 leapfrog steps per transition would take 173.54 seconds. Adjust your expectations accordingly! Gradient evaluation took 0.015889 seconds 1000 transitions using 10 leapfrog steps per transition would take 158.89 seconds. Adjust your expectations accordingly! Gradient evaluation took 0.016126 seconds 1000 transitions using 10 leapfrog steps per transition would take 161.26 seconds. Adjust your expectations accordingly! Gradient evaluation took 0.027377 seconds 1000 transitions using 10 leapfrog steps per transition would take 273.77 seconds. Adjust your expectations accordingly! Gradient evaluation took 0.015305 seconds 1000 transitions using 10 leapfrog steps per transition would take 153.05 seconds. Adjust your expectations accordingly! Gradient evaluation took 0.014925 seconds 1000 transitions using 10 leapfrog steps per transition would take 149.25 seconds. Adjust your expectations accordingly! Gradient evaluation took 0.013077 seconds 1000 transitions using 10 leapfrog steps per transition would take 130.77 seconds. Adjust your expectations accordingly! Chain [1] Iteration: 1 / 6000 [ 0%] (Warmup) Chain [3] Iteration: 1 / 6000 [ 0%] (Warmup) Chain [4] Iteration: 1 / 6000 [ 0%] (Warmup) Chain [2] Iteration: 1 / 6000 [ 0%] (Warmup) Chain [5] Iteration: 1 / 6000 [ 0%] (Warmup) Chain [6] Iteration: 1 / 6000 [ 0%] (Warmup) Chain [8] Iteration: 1 / 6000 [ 0%] (Warmup) Informational Message: The current Metropolis proposal is about to be rejected because of the following issue: Exception: Exception: multi_normal_lpdf: Location parameter[1] is -nan, but must be finite! (in 'Pure_PMF.stan', line 62, column 20 to column 112) (in 'Pure_PMF.stan', line 192, column 12 to line 195, column 39) If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine, but if this warning occurs often then your model may be either severely ill-conditioned or misspecified. Chain [7] Iteration: 1 / 6000 [ 0%] (Warmup) Chain [4] Iteration: 100 / 6000 [ 1%] (Warmup) Chain [7] Iteration: 100 / 6000 [ 1%] (Warmup) Chain [1] Iteration: 100 / 6000 [ 1%] (Warmup) Chain [3] Iteration: 100 / 6000 [ 1%] (Warmup) Chain [5] Iteration: 100 / 6000 [ 1%] (Warmup) Chain [6] Iteration: 100 / 6000 [ 1%] (Warmup) Chain [8] Iteration: 100 / 6000 [ 1%] (Warmup) Chain [2] Iteration: 100 / 6000 [ 1%] (Warmup) Chain [5] Iteration: 200 / 6000 [ 3%] (Warmup) Chain [4] Iteration: 200 / 6000 [ 3%] (Warmup) Chain [6] Iteration: 200 / 6000 [ 3%] (Warmup) Chain [8] Iteration: 200 / 6000 [ 3%] (Warmup) Chain [7] Iteration: 200 / 6000 [ 3%] (Warmup) Chain [2] Iteration: 200 / 6000 [ 3%] (Warmup) Chain [1] Iteration: 200 / 6000 [ 3%] (Warmup) Chain [3] Iteration: 200 / 6000 [ 3%] (Warmup) Chain [5] Iteration: 300 / 6000 [ 5%] (Warmup) Chain [4] Iteration: 300 / 6000 [ 5%] (Warmup) Chain [6] Iteration: 300 / 6000 [ 5%] (Warmup) Chain [2] Iteration: 300 / 6000 [ 5%] (Warmup) Chain [8] Iteration: 300 / 6000 [ 5%] (Warmup) Chain [3] Iteration: 300 / 6000 [ 5%] (Warmup) Chain [7] Iteration: 300 / 6000 [ 5%] (Warmup) Chain [1] Iteration: 300 / 6000 [ 5%] (Warmup) Chain [5] Iteration: 400 / 6000 [ 6%] (Warmup) Chain [4] Iteration: 400 / 6000 [ 6%] (Warmup) Chain [6] Iteration: 400 / 6000 [ 6%] (Warmup) Chain [3] Iteration: 400 / 6000 [ 6%] (Warmup) Chain [2] Iteration: 400 / 6000 [ 6%] (Warmup) Chain [8] Iteration: 400 / 6000 [ 6%] (Warmup) Chain [1] Iteration: 400 / 6000 [ 6%] (Warmup) Chain [7] Iteration: 400 / 6000 [ 6%] (Warmup) Chain [5] Iteration: 500 / 6000 [ 8%] (Warmup) Chain [4] Iteration: 500 / 6000 [ 8%] (Warmup) Chain [2] Iteration: 500 / 6000 [ 8%] (Warmup) Chain [6] Iteration: 500 / 6000 [ 8%] (Warmup) Chain [8] Iteration: 500 / 6000 [ 8%] (Warmup) Chain [1] Iteration: 500 / 6000 [ 8%] (Warmup) Chain [3] Iteration: 500 / 6000 [ 8%] (Warmup) Chain [7] Iteration: 500 / 6000 [ 8%] (Warmup) Chain [5] Iteration: 600 / 6000 [ 10%] (Warmup) Chain [4] Iteration: 600 / 6000 [ 10%] (Warmup) Chain [2] Iteration: 600 / 6000 [ 10%] (Warmup) Chain [6] Iteration: 600 / 6000 [ 10%] (Warmup) Chain [1] Iteration: 600 / 6000 [ 10%] (Warmup) Chain [8] Iteration: 600 / 6000 [ 10%] (Warmup) Chain [3] Iteration: 600 / 6000 [ 10%] (Warmup) Chain [7] Iteration: 600 / 6000 [ 10%] (Warmup) Chain [4] Iteration: 700 / 6000 [ 11%] (Warmup) Chain [5] Iteration: 700 / 6000 [ 11%] (Warmup) Chain [2] Iteration: 700 / 6000 [ 11%] (Warmup) Chain [6] Iteration: 700 / 6000 [ 11%] (Warmup) Chain [3] Iteration: 700 / 6000 [ 11%] (Warmup) Chain [1] Iteration: 700 / 6000 [ 11%] (Warmup) Chain [8] Iteration: 700 / 6000 [ 11%] (Warmup) Chain [7] Iteration: 700 / 6000 [ 11%] (Warmup) Chain [5] Iteration: 800 / 6000 [ 13%] (Warmup) Chain [4] Iteration: 800 / 6000 [ 13%] (Warmup) Chain [2] Iteration: 800 / 6000 [ 13%] (Warmup) Chain [6] Iteration: 800 / 6000 [ 13%] (Warmup) Chain [3] Iteration: 800 / 6000 [ 13%] (Warmup) Chain [1] Iteration: 800 / 6000 [ 13%] (Warmup) Chain [8] Iteration: 800 / 6000 [ 13%] (Warmup) Chain [7] Iteration: 800 / 6000 [ 13%] (Warmup) Chain [4] Iteration: 900 / 6000 [ 15%] (Warmup) Chain [5] Iteration: 900 / 6000 [ 15%] (Warmup) Chain [2] Iteration: 900 / 6000 [ 15%] (Warmup) Chain [6] Iteration: 900 / 6000 [ 15%] (Warmup) Chain [3] Iteration: 900 / 6000 [ 15%] (Warmup) Chain [8] Iteration: 900 / 6000 [ 15%] (Warmup) Chain [1] Iteration: 900 / 6000 [ 15%] (Warmup) Chain [7] Iteration: 900 / 6000 [ 15%] (Warmup) Chain [4] Iteration: 1000 / 6000 [ 16%] (Warmup) Chain [5] Iteration: 1000 / 6000 [ 16%] (Warmup) Chain [2] Iteration: 1000 / 6000 [ 16%] (Warmup) Chain [8] Iteration: 1000 / 6000 [ 16%] (Warmup) Chain [6] Iteration: 1000 / 6000 [ 16%] (Warmup) Chain [3] Iteration: 1000 / 6000 [ 16%] (Warmup) Chain [7] Iteration: 1000 / 6000 [ 16%] (Warmup) Chain [1] Iteration: 1000 / 6000 [ 16%] (Warmup) Chain [4] Iteration: 1100 / 6000 [ 18%] (Warmup) Chain [5] Iteration: 1100 / 6000 [ 18%] (Warmup) Chain [2] Iteration: 1100 / 6000 [ 18%] (Warmup) Chain [8] Iteration: 1100 / 6000 [ 18%] (Warmup) Chain [3] Iteration: 1100 / 6000 [ 18%] (Warmup) Chain [7] Iteration: 1100 / 6000 [ 18%] (Warmup) Chain [6] Iteration: 1100 / 6000 [ 18%] (Warmup) Chain [1] Iteration: 1100 / 6000 [ 18%] (Warmup) Chain [2] Iteration: 1200 / 6000 [ 20%] (Warmup) Chain [4] Iteration: 1200 / 6000 [ 20%] (Warmup) Chain [5] Iteration: 1200 / 6000 [ 20%] (Warmup) Chain [8] Iteration: 1200 / 6000 [ 20%] (Warmup) Chain [7] Iteration: 1200 / 6000 [ 20%] (Warmup) Chain [3] Iteration: 1200 / 6000 [ 20%] (Warmup) Chain [1] Iteration: 1200 / 6000 [ 20%] (Warmup) Chain [6] Iteration: 1200 / 6000 [ 20%] (Warmup) Chain [4] Iteration: 1300 / 6000 [ 21%] (Warmup) Chain [2] Iteration: 1300 / 6000 [ 21%] (Warmup) Chain [5] Iteration: 1300 / 6000 [ 21%] (Warmup) Chain [8] Iteration: 1300 / 6000 [ 21%] (Warmup) Chain [7] Iteration: 1300 / 6000 [ 21%] (Warmup) Chain [3] Iteration: 1300 / 6000 [ 21%] (Warmup) Chain [6] Iteration: 1300 / 6000 [ 21%] (Warmup) Chain [1] Iteration: 1300 / 6000 [ 21%] (Warmup) Chain [4] Iteration: 1400 / 6000 [ 23%] (Warmup) Chain [2] Iteration: 1400 / 6000 [ 23%] (Warmup) Chain [5] Iteration: 1400 / 6000 [ 23%] (Warmup) Chain [8] Iteration: 1400 / 6000 [ 23%] (Warmup) Chain [7] Iteration: 1400 / 6000 [ 23%] (Warmup) Chain [3] Iteration: 1400 / 6000 [ 23%] (Warmup) Chain [6] Iteration: 1400 / 6000 [ 23%] (Warmup) Chain [1] Iteration: 1400 / 6000 [ 23%] (Warmup) Chain [4] Iteration: 1500 / 6000 [ 25%] (Warmup) Chain [2] Iteration: 1500 / 6000 [ 25%] (Warmup) Chain [5] Iteration: 1500 / 6000 [ 25%] (Warmup) Chain [8] Iteration: 1500 / 6000 [ 25%] (Warmup) Chain [7] Iteration: 1500 / 6000 [ 25%] (Warmup) Chain [3] Iteration: 1500 / 6000 [ 25%] (Warmup) Chain [6] Iteration: 1500 / 6000 [ 25%] (Warmup) Chain [1] Iteration: 1500 / 6000 [ 25%] (Warmup) Chain [4] Iteration: 1600 / 6000 [ 26%] (Warmup) Chain [2] Iteration: 1600 / 6000 [ 26%] (Warmup) Chain [5] Iteration: 1600 / 6000 [ 26%] (Warmup) Chain [8] Iteration: 1600 / 6000 [ 26%] (Warmup) Chain [7] Iteration: 1600 / 6000 [ 26%] (Warmup) Chain [3] Iteration: 1600 / 6000 [ 26%] (Warmup) Chain [6] Iteration: 1600 / 6000 [ 26%] (Warmup) Chain [1] Iteration: 1600 / 6000 [ 26%] (Warmup) Chain [4] Iteration: 1700 / 6000 [ 28%] (Warmup) Chain [2] Iteration: 1700 / 6000 [ 28%] (Warmup) Chain [5] Iteration: 1700 / 6000 [ 28%] (Warmup) Chain [8] Iteration: 1700 / 6000 [ 28%] (Warmup) Chain [7] Iteration: 1700 / 6000 [ 28%] (Warmup) Chain [3] Iteration: 1700 / 6000 [ 28%] (Warmup) Chain [6] Iteration: 1700 / 6000 [ 28%] (Warmup) Chain [1] Iteration: 1700 / 6000 [ 28%] (Warmup) Chain [4] Iteration: 1800 / 6000 [ 30%] (Warmup) Chain [2] Iteration: 1800 / 6000 [ 30%] (Warmup) Chain [5] Iteration: 1800 / 6000 [ 30%] (Warmup) Chain [8] Iteration: 1800 / 6000 [ 30%] (Warmup) Chain [7] Iteration: 1800 / 6000 [ 30%] (Warmup) Chain [3] Iteration: 1800 / 6000 [ 30%] (Warmup) Chain [6] Iteration: 1800 / 6000 [ 30%] (Warmup) Chain [1] Iteration: 1800 / 6000 [ 30%] (Warmup) Chain [4] Iteration: 1900 / 6000 [ 31%] (Warmup) Chain [2] Iteration: 1900 / 6000 [ 31%] (Warmup) Chain [5] Iteration: 1900 / 6000 [ 31%] (Warmup) Chain [8] Iteration: 1900 / 6000 [ 31%] (Warmup) Chain [7] Iteration: 1900 / 6000 [ 31%] (Warmup) Chain [3] Iteration: 1900 / 6000 [ 31%] (Warmup) Chain [6] Iteration: 1900 / 6000 [ 31%] (Warmup) Chain [1] Iteration: 1900 / 6000 [ 31%] (Warmup) Chain [4] Iteration: 2000 / 6000 [ 33%] (Warmup) Chain [2] Iteration: 2000 / 6000 [ 33%] (Warmup) Chain [5] Iteration: 2000 / 6000 [ 33%] (Warmup) Chain [8] Iteration: 2000 / 6000 [ 33%] (Warmup) Chain [7] Iteration: 2000 / 6000 [ 33%] (Warmup) Chain [3] Iteration: 2000 / 6000 [ 33%] (Warmup) Chain [6] Iteration: 2000 / 6000 [ 33%] (Warmup) Chain [1] Iteration: 2000 / 6000 [ 33%] (Warmup) Chain [4] Iteration: 2100 / 6000 [ 35%] (Warmup) Chain [2] Iteration: 2100 / 6000 [ 35%] (Warmup) Chain [5] Iteration: 2100 / 6000 [ 35%] (Warmup) Chain [7] Iteration: 2100 / 6000 [ 35%] (Warmup) Chain [8] Iteration: 2100 / 6000 [ 35%] (Warmup) Chain [3] Iteration: 2100 / 6000 [ 35%] (Warmup) Chain [6] Iteration: 2100 / 6000 [ 35%] (Warmup) Chain [1] Iteration: 2100 / 6000 [ 35%] (Warmup) Chain [4] Iteration: 2200 / 6000 [ 36%] (Warmup) Chain [2] Iteration: 2200 / 6000 [ 36%] (Warmup) Chain [5] Iteration: 2200 / 6000 [ 36%] (Warmup) Chain [7] Iteration: 2200 / 6000 [ 36%] (Warmup) Chain [8] Iteration: 2200 / 6000 [ 36%] (Warmup) Chain [3] Iteration: 2200 / 6000 [ 36%] (Warmup) Chain [6] Iteration: 2200 / 6000 [ 36%] (Warmup) Chain [1] Iteration: 2200 / 6000 [ 36%] (Warmup) Chain [4] Iteration: 2300 / 6000 [ 38%] (Warmup) Chain [2] Iteration: 2300 / 6000 [ 38%] (Warmup) Chain [5] Iteration: 2300 / 6000 [ 38%] (Warmup) Chain [7] Iteration: 2300 / 6000 [ 38%] (Warmup) Chain [8] Iteration: 2300 / 6000 [ 38%] (Warmup) Chain [3] Iteration: 2300 / 6000 [ 38%] (Warmup) Chain [6] Iteration: 2300 / 6000 [ 38%] (Warmup) Chain [1] Iteration: 2300 / 6000 [ 38%] (Warmup) Chain [4] Iteration: 2400 / 6000 [ 40%] (Warmup) Chain [2] Iteration: 2400 / 6000 [ 40%] (Warmup) Chain [5] Iteration: 2400 / 6000 [ 40%] (Warmup) Chain [7] Iteration: 2400 / 6000 [ 40%] (Warmup) Chain [8] Iteration: 2400 / 6000 [ 40%] (Warmup) Chain [3] Iteration: 2400 / 6000 [ 40%] (Warmup) Chain [6] Iteration: 2400 / 6000 [ 40%] (Warmup) Chain [1] Iteration: 2400 / 6000 [ 40%] (Warmup) Chain [4] Iteration: 2500 / 6000 [ 41%] (Warmup) Chain [2] Iteration: 2500 / 6000 [ 41%] (Warmup) Chain [5] Iteration: 2500 / 6000 [ 41%] (Warmup) Chain [7] Iteration: 2500 / 6000 [ 41%] (Warmup) Chain [8] Iteration: 2500 / 6000 [ 41%] (Warmup) Chain [3] Iteration: 2500 / 6000 [ 41%] (Warmup) Chain [6] Iteration: 2500 / 6000 [ 41%] (Warmup) Chain [1] Iteration: 2500 / 6000 [ 41%] (Warmup) Chain [4] Iteration: 2600 / 6000 [ 43%] (Warmup) Chain [2] Iteration: 2600 / 6000 [ 43%] (Warmup) Chain [5] Iteration: 2600 / 6000 [ 43%] (Warmup) Chain [7] Iteration: 2600 / 6000 [ 43%] (Warmup) Chain [8] Iteration: 2600 / 6000 [ 43%] (Warmup) Chain [3] Iteration: 2600 / 6000 [ 43%] (Warmup) Chain [6] Iteration: 2600 / 6000 [ 43%] (Warmup) Chain [1] Iteration: 2600 / 6000 [ 43%] (Warmup) Chain [4] Iteration: 2700 / 6000 [ 45%] (Warmup) Chain [2] Iteration: 2700 / 6000 [ 45%] (Warmup) Chain [5] Iteration: 2700 / 6000 [ 45%] (Warmup) Chain [7] Iteration: 2700 / 6000 [ 45%] (Warmup) Chain [8] Iteration: 2700 / 6000 [ 45%] (Warmup) Chain [3] Iteration: 2700 / 6000 [ 45%] (Warmup) Chain [6] Iteration: 2700 / 6000 [ 45%] (Warmup) Chain [1] Iteration: 2700 / 6000 [ 45%] (Warmup) Chain [4] Iteration: 2800 / 6000 [ 46%] (Warmup) Chain [2] Iteration: 2800 / 6000 [ 46%] (Warmup) Chain [5] Iteration: 2800 / 6000 [ 46%] (Warmup) Chain [7] Iteration: 2800 / 6000 [ 46%] (Warmup) Chain [8] Iteration: 2800 / 6000 [ 46%] (Warmup) Chain [3] Iteration: 2800 / 6000 [ 46%] (Warmup) Chain [6] Iteration: 2800 / 6000 [ 46%] (Warmup) Chain [1] Iteration: 2800 / 6000 [ 46%] (Warmup) Chain [4] Iteration: 2900 / 6000 [ 48%] (Warmup) Chain [2] Iteration: 2900 / 6000 [ 48%] (Warmup) Chain [5] Iteration: 2900 / 6000 [ 48%] (Warmup) Chain [7] Iteration: 2900 / 6000 [ 48%] (Warmup) Chain [8] Iteration: 2900 / 6000 [ 48%] (Warmup) Chain [3] Iteration: 2900 / 6000 [ 48%] (Warmup) Chain [6] Iteration: 2900 / 6000 [ 48%] (Warmup) Chain [1] Iteration: 2900 / 6000 [ 48%] (Warmup) Chain [4] Iteration: 3000 / 6000 [ 50%] (Warmup) Chain [2] Iteration: 3000 / 6000 [ 50%] (Warmup) Chain [5] Iteration: 3000 / 6000 [ 50%] (Warmup) Chain [7] Iteration: 3000 / 6000 [ 50%] (Warmup) Chain [8] Iteration: 3000 / 6000 [ 50%] (Warmup) Chain [3] Iteration: 3000 / 6000 [ 50%] (Warmup) Chain [6] Iteration: 3000 / 6000 [ 50%] (Warmup) Chain [1] Iteration: 3000 / 6000 [ 50%] (Warmup) Chain [4] Iteration: 3100 / 6000 [ 51%] (Warmup) Chain [2] Iteration: 3100 / 6000 [ 51%] (Warmup) Chain [5] Iteration: 3100 / 6000 [ 51%] (Warmup) Chain [7] Iteration: 3100 / 6000 [ 51%] (Warmup) Chain [8] Iteration: 3100 / 6000 [ 51%] (Warmup) Chain [3] Iteration: 3100 / 6000 [ 51%] (Warmup) Chain [6] Iteration: 3100 / 6000 [ 51%] (Warmup) Chain [1] Iteration: 3100 / 6000 [ 51%] (Warmup) Chain [4] Iteration: 3200 / 6000 [ 53%] (Warmup) Chain [2] Iteration: 3200 / 6000 [ 53%] (Warmup) Chain [5] Iteration: 3200 / 6000 [ 53%] (Warmup) Chain [7] Iteration: 3200 / 6000 [ 53%] (Warmup) Chain [8] Iteration: 3200 / 6000 [ 53%] (Warmup) Chain [3] Iteration: 3200 / 6000 [ 53%] (Warmup) Chain [6] Iteration: 3200 / 6000 [ 53%] (Warmup) Chain [1] Iteration: 3200 / 6000 [ 53%] (Warmup) Chain [4] Iteration: 3300 / 6000 [ 55%] (Warmup) Chain [2] Iteration: 3300 / 6000 [ 55%] (Warmup) Chain [5] Iteration: 3300 / 6000 [ 55%] (Warmup) Chain [8] Iteration: 3300 / 6000 [ 55%] (Warmup) Chain [7] Iteration: 3300 / 6000 [ 55%] (Warmup) Chain [3] Iteration: 3300 / 6000 [ 55%] (Warmup) Chain [6] Iteration: 3300 / 6000 [ 55%] (Warmup) Chain [1] Iteration: 3300 / 6000 [ 55%] (Warmup) Chain [4] Iteration: 3400 / 6000 [ 56%] (Warmup) Chain [2] Iteration: 3400 / 6000 [ 56%] (Warmup) Chain [5] Iteration: 3400 / 6000 [ 56%] (Warmup) Chain [8] Iteration: 3400 / 6000 [ 56%] (Warmup) Chain [7] Iteration: 3400 / 6000 [ 56%] (Warmup) Chain [3] Iteration: 3400 / 6000 [ 56%] (Warmup) Chain [6] Iteration: 3400 / 6000 [ 56%] (Warmup) Chain [1] Iteration: 3400 / 6000 [ 56%] (Warmup) Chain [4] Iteration: 3500 / 6000 [ 58%] (Warmup) Chain [2] Iteration: 3500 / 6000 [ 58%] (Warmup) Chain [5] Iteration: 3500 / 6000 [ 58%] (Warmup) Chain [8] Iteration: 3500 / 6000 [ 58%] (Warmup) Chain [7] Iteration: 3500 / 6000 [ 58%] (Warmup) Chain [3] Iteration: 3500 / 6000 [ 58%] (Warmup) Chain [6] Iteration: 3500 / 6000 [ 58%] (Warmup) Chain [1] Iteration: 3500 / 6000 [ 58%] (Warmup) Chain [4] Iteration: 3600 / 6000 [ 60%] (Warmup) Chain [2] Iteration: 3600 / 6000 [ 60%] (Warmup) Chain [5] Iteration: 3600 / 6000 [ 60%] (Warmup) Chain [8] Iteration: 3600 / 6000 [ 60%] (Warmup) Chain [7] Iteration: 3600 / 6000 [ 60%] (Warmup) Chain [3] Iteration: 3600 / 6000 [ 60%] (Warmup) Chain [6] Iteration: 3600 / 6000 [ 60%] (Warmup) Chain [1] Iteration: 3600 / 6000 [ 60%] (Warmup) Chain [4] Iteration: 3700 / 6000 [ 61%] (Warmup) Chain [2] Iteration: 3700 / 6000 [ 61%] (Warmup) Chain [5] Iteration: 3700 / 6000 [ 61%] (Warmup) Chain [8] Iteration: 3700 / 6000 [ 61%] (Warmup) Chain [7] Iteration: 3700 / 6000 [ 61%] (Warmup) Chain [3] Iteration: 3700 / 6000 [ 61%] (Warmup) Chain [6] Iteration: 3700 / 6000 [ 61%] (Warmup) Chain [1] Iteration: 3700 / 6000 [ 61%] (Warmup) Chain [4] Iteration: 3800 / 6000 [ 63%] (Warmup) Chain [2] Iteration: 3800 / 6000 [ 63%] (Warmup) Chain [5] Iteration: 3800 / 6000 [ 63%] (Warmup) Chain [8] Iteration: 3800 / 6000 [ 63%] (Warmup) Chain [7] Iteration: 3800 / 6000 [ 63%] (Warmup) Chain [3] Iteration: 3800 / 6000 [ 63%] (Warmup) Chain [6] Iteration: 3800 / 6000 [ 63%] (Warmup) Chain [1] Iteration: 3800 / 6000 [ 63%] (Warmup) Chain [4] Iteration: 3900 / 6000 [ 65%] (Warmup) Chain [2] Iteration: 3900 / 6000 [ 65%] (Warmup) Chain [5] Iteration: 3900 / 6000 [ 65%] (Warmup) Chain [8] Iteration: 3900 / 6000 [ 65%] (Warmup) Chain [7] Iteration: 3900 / 6000 [ 65%] (Warmup) Chain [3] Iteration: 3900 / 6000 [ 65%] (Warmup) Chain [6] Iteration: 3900 / 6000 [ 65%] (Warmup) Chain [1] Iteration: 3900 / 6000 [ 65%] (Warmup) Chain [4] Iteration: 4000 / 6000 [ 66%] (Warmup) Chain [5] Iteration: 4000 / 6000 [ 66%] (Warmup) Chain [2] Iteration: 4000 / 6000 [ 66%] (Warmup) Chain [8] Iteration: 4000 / 6000 [ 66%] (Warmup) Chain [7] Iteration: 4000 / 6000 [ 66%] (Warmup) Chain [3] Iteration: 4000 / 6000 [ 66%] (Warmup) Chain [6] Iteration: 4000 / 6000 [ 66%] (Warmup) Chain [1] Iteration: 4000 / 6000 [ 66%] (Warmup) Chain [4] Iteration: 4100 / 6000 [ 68%] (Warmup) Chain [5] Iteration: 4100 / 6000 [ 68%] (Warmup) Chain [2] Iteration: 4100 / 6000 [ 68%] (Warmup) Chain [8] Iteration: 4100 / 6000 [ 68%] (Warmup) Chain [7] Iteration: 4100 / 6000 [ 68%] (Warmup) Chain [3] Iteration: 4100 / 6000 [ 68%] (Warmup) Chain [6] Iteration: 4100 / 6000 [ 68%] (Warmup) Chain [1] Iteration: 4100 / 6000 [ 68%] (Warmup) Chain [4] Iteration: 4200 / 6000 [ 70%] (Warmup) Chain [5] Iteration: 4200 / 6000 [ 70%] (Warmup) Chain [2] Iteration: 4200 / 6000 [ 70%] (Warmup) Chain [8] Iteration: 4200 / 6000 [ 70%] (Warmup) Chain [7] Iteration: 4200 / 6000 [ 70%] (Warmup) Chain [3] Iteration: 4200 / 6000 [ 70%] (Warmup) Chain [6] Iteration: 4200 / 6000 [ 70%] (Warmup) Chain [1] Iteration: 4200 / 6000 [ 70%] (Warmup) Chain [4] Iteration: 4300 / 6000 [ 71%] (Warmup) Chain [2] Iteration: 4300 / 6000 [ 71%] (Warmup) Chain [5] Iteration: 4300 / 6000 [ 71%] (Warmup) Chain [8] Iteration: 4300 / 6000 [ 71%] (Warmup) Chain [7] Iteration: 4300 / 6000 [ 71%] (Warmup) Chain [3] Iteration: 4300 / 6000 [ 71%] (Warmup) Chain [6] Iteration: 4300 / 6000 [ 71%] (Warmup) Chain [1] Iteration: 4300 / 6000 [ 71%] (Warmup) Chain [4] Iteration: 4400 / 6000 [ 73%] (Warmup) Chain [8] Iteration: 4400 / 6000 [ 73%] (Warmup) Chain [5] Iteration: 4400 / 6000 [ 73%] (Warmup) Chain [2] Iteration: 4400 / 6000 [ 73%] (Warmup) Chain [7] Iteration: 4400 / 6000 [ 73%] (Warmup) Chain [3] Iteration: 4400 / 6000 [ 73%] (Warmup) Chain [6] Iteration: 4400 / 6000 [ 73%] (Warmup) Chain [1] Iteration: 4400 / 6000 [ 73%] (Warmup) Chain [4] Iteration: 4500 / 6000 [ 75%] (Warmup) Chain [8] Iteration: 4500 / 6000 [ 75%] (Warmup) Chain [2] Iteration: 4500 / 6000 [ 75%] (Warmup) Chain [5] Iteration: 4500 / 6000 [ 75%] (Warmup) Chain [7] Iteration: 4500 / 6000 [ 75%] (Warmup) Chain [3] Iteration: 4500 / 6000 [ 75%] (Warmup) Chain [6] Iteration: 4500 / 6000 [ 75%] (Warmup) Chain [1] Iteration: 4500 / 6000 [ 75%] (Warmup) Chain [4] Iteration: 4600 / 6000 [ 76%] (Warmup) Chain [8] Iteration: 4600 / 6000 [ 76%] (Warmup) Chain [5] Iteration: 4600 / 6000 [ 76%] (Warmup) Chain [2] Iteration: 4600 / 6000 [ 76%] (Warmup) Chain [7] Iteration: 4600 / 6000 [ 76%] (Warmup) Chain [3] Iteration: 4600 / 6000 [ 76%] (Warmup) Chain [6] Iteration: 4600 / 6000 [ 76%] (Warmup) Chain [1] Iteration: 4600 / 6000 [ 76%] (Warmup) Chain [4] Iteration: 4700 / 6000 [ 78%] (Warmup) Chain [8] Iteration: 4700 / 6000 [ 78%] (Warmup) Chain [5] Iteration: 4700 / 6000 [ 78%] (Warmup) Chain [2] Iteration: 4700 / 6000 [ 78%] (Warmup) Chain [7] Iteration: 4700 / 6000 [ 78%] (Warmup) Chain [3] Iteration: 4700 / 6000 [ 78%] (Warmup) Chain [6] Iteration: 4700 / 6000 [ 78%] (Warmup) Chain [1] Iteration: 4700 / 6000 [ 78%] (Warmup) Chain [4] Iteration: 4800 / 6000 [ 80%] (Warmup) Chain [8] Iteration: 4800 / 6000 [ 80%] (Warmup) Chain [5] Iteration: 4800 / 6000 [ 80%] (Warmup) Chain [2] Iteration: 4800 / 6000 [ 80%] (Warmup) Chain [7] Iteration: 4800 / 6000 [ 80%] (Warmup) Chain [3] Iteration: 4800 / 6000 [ 80%] (Warmup) Chain [6] Iteration: 4800 / 6000 [ 80%] (Warmup) Chain [1] Iteration: 4800 / 6000 [ 80%] (Warmup) Chain [4] Iteration: 4900 / 6000 [ 81%] (Warmup) Chain [8] Iteration: 4900 / 6000 [ 81%] (Warmup) Chain [2] Iteration: 4900 / 6000 [ 81%] (Warmup) Chain [5] Iteration: 4900 / 6000 [ 81%] (Warmup) Chain [7] Iteration: 4900 / 6000 [ 81%] (Warmup) Chain [3] Iteration: 4900 / 6000 [ 81%] (Warmup) Chain [6] Iteration: 4900 / 6000 [ 81%] (Warmup) Chain [1] Iteration: 4900 / 6000 [ 81%] (Warmup) Chain [4] Iteration: 5000 / 6000 [ 83%] (Warmup) Chain [4] Iteration: 5001 / 6000 [ 83%] (Sampling) Chain [2] Iteration: 5000 / 6000 [ 83%] (Warmup) Chain [2] Iteration: 5001 / 6000 [ 83%] (Sampling) Chain [5] Iteration: 5000 / 6000 [ 83%] (Warmup) Chain [8] Iteration: 5000 / 6000 [ 83%] (Warmup) Chain [5] Iteration: 5001 / 6000 [ 83%] (Sampling) Chain [8] Iteration: 5001 / 6000 [ 83%] (Sampling) Chain [7] Iteration: 5000 / 6000 [ 83%] (Warmup) Chain [3] Iteration: 5000 / 6000 [ 83%] (Warmup) Chain [7] Iteration: 5001 / 6000 [ 83%] (Sampling) Chain [3] Iteration: 5001 / 6000 [ 83%] (Sampling) Chain [6] Iteration: 5000 / 6000 [ 83%] (Warmup) Chain [6] Iteration: 5001 / 6000 [ 83%] (Sampling) Chain [1] Iteration: 5000 / 6000 [ 83%] (Warmup) Chain [1] Iteration: 5001 / 6000 [ 83%] (Sampling) Chain [4] Iteration: 5100 / 6000 [ 85%] (Sampling) Chain [2] Iteration: 5100 / 6000 [ 85%] (Sampling) Chain [5] Iteration: 5100 / 6000 [ 85%] (Sampling) Chain [8] Iteration: 5100 / 6000 [ 85%] (Sampling) Chain [7] Iteration: 5100 / 6000 [ 85%] (Sampling) Chain [3] Iteration: 5100 / 6000 [ 85%] (Sampling) Chain [6] Iteration: 5100 / 6000 [ 85%] (Sampling) Chain [1] Iteration: 5100 / 6000 [ 85%] (Sampling) Chain [4] Iteration: 5200 / 6000 [ 86%] (Sampling) Chain [2] Iteration: 5200 / 6000 [ 86%] (Sampling) Chain [5] Iteration: 5200 / 6000 [ 86%] (Sampling) Chain [8] Iteration: 5200 / 6000 [ 86%] (Sampling) Chain [7] Iteration: 5200 / 6000 [ 86%] (Sampling) Chain [3] Iteration: 5200 / 6000 [ 86%] (Sampling) Chain [6] Iteration: 5200 / 6000 [ 86%] (Sampling) Chain [1] Iteration: 5200 / 6000 [ 86%] (Sampling) Chain [4] Iteration: 5300 / 6000 [ 88%] (Sampling) Chain [2] Iteration: 5300 / 6000 [ 88%] (Sampling) Chain [8] Iteration: 5300 / 6000 [ 88%] (Sampling) Chain [5] Iteration: 5300 / 6000 [ 88%] (Sampling) Chain [7] Iteration: 5300 / 6000 [ 88%] (Sampling) Chain [3] Iteration: 5300 / 6000 [ 88%] (Sampling) Chain [6] Iteration: 5300 / 6000 [ 88%] (Sampling) Chain [1] Iteration: 5300 / 6000 [ 88%] (Sampling) Chain [4] Iteration: 5400 / 6000 [ 90%] (Sampling) Chain [2] Iteration: 5400 / 6000 [ 90%] (Sampling) Chain [8] Iteration: 5400 / 6000 [ 90%] (Sampling) Chain [5] Iteration: 5400 / 6000 [ 90%] (Sampling) Chain [7] Iteration: 5400 / 6000 [ 90%] (Sampling) Chain [3] Iteration: 5400 / 6000 [ 90%] (Sampling) Chain [6] Iteration: 5400 / 6000 [ 90%] (Sampling) Chain [1] Iteration: 5400 / 6000 [ 90%] (Sampling) Chain [4] Iteration: 5500 / 6000 [ 91%] (Sampling) Chain [2] Iteration: 5500 / 6000 [ 91%] (Sampling) Chain [8] Iteration: 5500 / 6000 [ 91%] (Sampling) Chain [5] Iteration: 5500 / 6000 [ 91%] (Sampling) Chain [7] Iteration: 5500 / 6000 [ 91%] (Sampling) Chain [3] Iteration: 5500 / 6000 [ 91%] (Sampling) Chain [6] Iteration: 5500 / 6000 [ 91%] (Sampling) Chain [1] Iteration: 5500 / 6000 [ 91%] (Sampling) Chain [4] Iteration: 5600 / 6000 [ 93%] (Sampling) Chain [2] Iteration: 5600 / 6000 [ 93%] (Sampling) Chain [5] Iteration: 5600 / 6000 [ 93%] (Sampling) Chain [8] Iteration: 5600 / 6000 [ 93%] (Sampling) Chain [7] Iteration: 5600 / 6000 [ 93%] (Sampling) Chain [3] Iteration: 5600 / 6000 [ 93%] (Sampling) Chain [6] Iteration: 5600 / 6000 [ 93%] (Sampling) Chain [1] Iteration: 5600 / 6000 [ 93%] (Sampling) Chain [4] Iteration: 5700 / 6000 [ 95%] (Sampling) Chain [2] Iteration: 5700 / 6000 [ 95%] (Sampling) Chain [5] Iteration: 5700 / 6000 [ 95%] (Sampling) Chain [8] Iteration: 5700 / 6000 [ 95%] (Sampling) Chain [7] Iteration: 5700 / 6000 [ 95%] (Sampling) Chain [3] Iteration: 5700 / 6000 [ 95%] (Sampling) Chain [6] Iteration: 5700 / 6000 [ 95%] (Sampling) Chain [1] Iteration: 5700 / 6000 [ 95%] (Sampling) Chain [4] Iteration: 5800 / 6000 [ 96%] (Sampling) Chain [2] Iteration: 5800 / 6000 [ 96%] (Sampling) Chain [5] Iteration: 5800 / 6000 [ 96%] (Sampling) Chain [8] Iteration: 5800 / 6000 [ 96%] (Sampling) Chain [7] Iteration: 5800 / 6000 [ 96%] (Sampling) Chain [3] Iteration: 5800 / 6000 [ 96%] (Sampling) Chain [6] Iteration: 5800 / 6000 [ 96%] (Sampling) Chain [1] Iteration: 5800 / 6000 [ 96%] (Sampling) Chain [4] Iteration: 5900 / 6000 [ 98%] (Sampling) Chain [2] Iteration: 5900 / 6000 [ 98%] (Sampling) Chain [5] Iteration: 5900 / 6000 [ 98%] (Sampling) Chain [8] Iteration: 5900 / 6000 [ 98%] (Sampling) Chain [7] Iteration: 5900 / 6000 [ 98%] (Sampling) Chain [3] Iteration: 5900 / 6000 [ 98%] (Sampling) Chain [6] Iteration: 5900 / 6000 [ 98%] (Sampling) Chain [1] Iteration: 5900 / 6000 [ 98%] (Sampling) Chain [4] Iteration: 6000 / 6000 [100%] (Sampling) Elapsed Time: 5448.98 seconds (Warm-up) 1134.54 seconds (Sampling) 6583.52 seconds (Total) Chain [2] Iteration: 6000 / 6000 [100%] (Sampling) Elapsed Time: 5453.06 seconds (Warm-up) 1134.51 seconds (Sampling) 6587.57 seconds (Total) Chain [5] Iteration: 6000 / 6000 [100%] (Sampling) Elapsed Time: 5458.26 seconds (Warm-up) 1132.12 seconds (Sampling) 6590.38 seconds (Total) Chain [8] Iteration: 6000 / 6000 [100%] (Sampling) Elapsed Time: 5458.49 seconds (Warm-up) 1133.37 seconds (Sampling) 6591.87 seconds (Total) Chain [7] Iteration: 6000 / 6000 [100%] (Sampling) Elapsed Time: 5463.18 seconds (Warm-up) 1131.45 seconds (Sampling) 6594.64 seconds (Total) Chain [3] Iteration: 6000 / 6000 [100%] (Sampling) Elapsed Time: 5466.54 seconds (Warm-up) 1131.53 seconds (Sampling) 6598.07 seconds (Total) Chain [6] Iteration: 6000 / 6000 [100%] (Sampling) Elapsed Time: 5475.86 seconds (Warm-up) 1127.31 seconds (Sampling) 6603.17 seconds (Total) Chain [1] Iteration: 6000 / 6000 [100%] (Sampling) Elapsed Time: 5485.22 seconds (Warm-up) 1123.69 seconds (Sampling) 6608.91 seconds (Total) double free or corruption (!prev) ```

The python error indicates (only relevant parts included)

fit = model.sample(data=f'{path}/data.json', output_dir=output_dir1,
                            refresh=100, iter_warmup=5000, 
                            iter_sampling=1000, chains=chains, parallel_chains=chains, 
                            threads_per_chain=threads_per_chain, max_treedepth=5,
                            metric='dense_e', save_profile=True, sig_figs=18)

CmdStan error: terminated by signal 6 Unknown error -6
RuntimeError: Error during sampling:
Exception: Exception: multi_normal_lpdf: Location parameter[1] is -nan, but must be finite! (in 'Pure_PMF.stan', line 62, column 20 to column 112) (in 'Pure_PMF.stan', line 192, column 12 to line 195, column 39)
Command and output files:
RunSet: chains=8, chain_ids=[1, 2, 3, 4, 5, 6, 7, 8], num_processes=1
 cmd (chain 1):
    ['/home/22796002/Pure RK PMF/Subsets/Alkane_Primary alcohol/Include_clusters_False/Variance_known_True/Variance_MC_known_True/rank_2/Pure_PMF', 'id=1', 'random', 'seed=24898', 'data', 'file=Subsets/Alkane_Primary alcohol/Include_clusters_False/Variance_known_True/Variance_MC_known_True/rank_2/data.json', 'output', 'file=/home/22796002/Pure RK PMF/Subsets/Alkane_Primary alcohol/Include_clusters_False/Variance_known_True/Variance_MC_known_True/rank_2/Step1/Pure_PMF-20240522025118.csv', 'profile_file=/home/22796002/Pure RK PMF/Subsets/Alkane_Primary alcohol/Include_clusters_False/Variance_known_True/Variance_MC_known_True/rank_2/Step1/Pure_PMF-20240522025118-profile.csv', 'refresh=100', 'sig_figs=18', 'method=sample', 'num_samples=1000', 'num_warmup=5000', 'algorithm=hmc', 'engine=nuts', 'max_depth=5', 'metric=dense_e', 'adapt', 'engaged=1', 'num_chains=8']
 retcodes=[-6]

Reconstructing the fit from .csv files does not produce any errors

Stan code for reference ``` functions { // kernel for composition matrix Kx(vector x1, vector x2, int order) { int N = rows(x1); int M = rows(x2); matrix[N, order+1] X1; matrix[M, order+1] X2; for (i in 1:order) { X1[:,i] = x1 .^(order+2-i) - x1; X2[:,i] = x2 .^(order+2-i) - x2; } X1[:,order+1] = 1e-1 * x1 .* sqrt(1-x1) .* exp(x1); X2[:,order+1] = 1e-1 * x2 .* sqrt(1-x2) .* exp(x2); return X1 * X2'; } // kernel for Temperature matrix KT(vector T1, vector T2) { int N = rows(T1); int M = rows(T2); matrix[N, 4] TT1 = append_col(append_col(append_col(rep_vector(1.0, N), T1), 1e-2* T1.^2), 1e-4* T1.^3); matrix[M, 4] TT2 = append_col(append_col(append_col(rep_vector(1.0, M), T2), 1e-2* T2.^2), 1e-4* T2.^3); return TT1 * TT2'; } // Combined kernel matrix Kernel(vector x1, vector x2, vector T1, vector T2, int order) { return Kx(x1, x2, order) .* KT(T1, T2); } // functions for the priors of the feature matrices real ps_feature_matrices(array[] int M_slice, int start, int end, array[] matrix U_raw, array[] matrix V_raw) { real all_target = 0; for (m in start:end) { all_target += std_normal_lpdf(to_vector(U_raw[m])); all_target += std_normal_lpdf(to_vector(V_raw[m])); } return all_target; } // function for the likelihood and prior on smoothed values real ps_like(array[] int N_slice, int start, int end, vector y, matrix cov_f2_f1, matrix mu_y_y_MC, matrix L_MC_inv, vector v, array[,] int Idx_all, int M, int N_T, int N_MC, int N_C, int N_known, array[] int N_points, vector v_ARD, array[,] matrix U_raw, array[,] matrix V_raw, real v_MC) { real all_target = 0; for (i in start:end) { vector[N_MC] y_MC_pred; for(t in 1:N_T) { for (m in 1:M-1) { y_MC_pred[m+N_C*(t-1)] = dot_product(U_raw[t,m,:,Idx_all[i,1]] .* v_ARD, V_raw[t,m,:,Idx_all[i,2]]); y_MC_pred[N_C-m+1+N_C*(t-1)] = dot_product(U_raw[t,m,:,Idx_all[i,2]] .* v_ARD, V_raw[t,m,:,Idx_all[i,1]]); } y_MC_pred[M+N_C*(t-1)] = dot_product(U_raw[t,M,:,Idx_all[i,1]] .* v_ARD, U_raw[t,M,:,Idx_all[i,2]]); } all_target += -0.5 * dot_self(L_MC_inv * y_MC_pred); // GP prior if (i <= N_known) { vector[N_points[i]] y_pred = mu_y_y_MC[sum(N_points[:i-1])+1:sum(N_points[:i]),:] * y_MC_pred; matrix[N_points[i], N_points[i]] cov_y = add_diag(cov_f2_f1[sum(N_points[:i-1])+1:sum(N_points[:i]), :N_points[i]], v[i]); all_target += multi_normal_lpdf(y[sum(N_points[:i-1])+1:sum(N_points[:i])] | y_pred, cov_y); } } return all_target; } } data { int N_known; // number of known mixtures int N_unknown; // number of unknown mixtures array[N_known] int N_points; // number of experimental points per known dataset int order; // order of the compositional polynomial vector[sum(N_points)] x1; // experimental composition vector[sum(N_points)] T1; // experimental temperatures vector[sum(N_points)] y1; // experimental excess enthalpy int N_T; // number of interpolated temperatures int N_C; // number of interpolated compositions vector[N_T] T2_int; // unique temperatures to interpolate vector[N_C] x2_int; // unique compositions to interpolate real v_MC; // error between of y_MC and (U,V) int grainsize; // number of grainsizes int N; // number of components int D; // rank of feature matrices array[N_known, 2] int Idx_known; // indices (row, column) of known datasets array[N_unknown, 2] int Idx_unknown;// indices (row, column) of unknown datasets real jitter; // jitter for stability of covariances vector[N_known] v; // known data-model variance } transformed data { real error = 0.01; // error in the data (fraction of experimental data) vector[sum(N_points)] var_data = square(error*y1); // variance of the data int M = (N_C + 1) %/% 2; // interger division to get the number of U matrices int N_MC = N_C*N_T; // overall number of interpolated datapoints per dataset array[M-1] int M_slice; // array of integers to be used as indices in parallel computations array[N_known+N_unknown] int N_slice; // array of integers to be used as indices in parallel computations array[N_known+N_unknown,2] int Idx_all = append_array(Idx_known, Idx_unknown); // indices of all datasets vector[N_MC] x2; // concatnated vector of x2_int vector[N_MC] T2; // concatenated vector of T2_int matrix[N_MC, N_MC] K_MC; // kernel for the interpolated data matrix[N_MC, N_MC] y_MC_prec; // inverse of kernel matrix[sum(N_points), N_MC] mu_y_y_MC; // Mapping from y_MC to y (predictive GP mean) matrix[sum(N_points), max(N_points)] cov_f2_f1; // covariance of y smmooth|y_MC matrix[N_MC, N_MC] L_y_MC_inv_cov; // Inverse of cholesky decomposition of K_MC matrix[sum(N_points), max(N_points)] cov_y_yMC; // conditional covariance matrix y|y_MC matrix[N_MC, N_MC] cov_y_MC; // conditional covariance of y_MC|(U,V) // Assign MC input vectors for temperature and composition for (i in 1:N_T) { x2[(i-1)*N_C+1:i*N_C] = x2_int; T2[(i-1)*N_C+1:i*N_C] = rep_vector(T2_int[i],N_C); } // Assign MC kernel K_MC = add_diag(Kernel(x2, x2, T2, T2, order), jitter); // Compute inverse using cholesky { L_y_MC_inv_cov = inverse(cholesky_decompose(add_diag(K_MC, v_MC))); y_MC_prec = crossprod(L_y_MC_inv_cov); // L_MC_inv' * L_MC_inv } { // Loop through data sets to compute the relevant matrix, mapping from y_MC to y and covariance for (i in 1:N_known) { // obtain kernel matrix for cross covariance matrix[N_points[i], N_MC] K_y_yMC = Kernel(x1[sum(N_points[:i-1])+1:sum(N_points[:i])], x2, T1[sum(N_points[:i-1])+1:sum(N_points[:i])], T2, order); // Stable implementation of A*B*A^T where B is a positive semi-definite matrix // This is decomposed into A*(L^-1)^T*L^-1*A^T where L is the cholesky factor of B // A*B*A^T = C*C^T where C = A*(L^-1)^T // Computation of above mentioned K12*K22^-1*K12^T matrix[N_points[i], N_MC] stable_inv = K_y_yMC * L_y_MC_inv_cov'; // obtain kernel for the data matrix[N_points[i], N_points[i]] K_y = Kernel(x1[sum(N_points[:i-1])+1:sum(N_points[:i])], x1[sum(N_points[:i-1])+1:sum(N_points[:i])], T1[sum(N_points[:i-1])+1:sum(N_points[:i])], T1[sum(N_points[:i-1])+1:sum(N_points[:i])], order); // Computation of K12*K22^-1*K12^T cov_f2_f1[sum(N_points[:i-1])+1:sum(N_points[:i]), :N_points[i]] = add_diag(K_y - stable_inv * stable_inv', var_data[sum(N_points[:i-1])+1:sum(N_points[:i])]); // Mapping from y_MC to y mu_y_y_MC[sum(N_points[:i-1])+1:sum(N_points[:i]), :] = K_y_yMC * y_MC_prec; // slice variable for parallel computations N_slice[i] = i; } } // Continue to assign the slice variable for unknown datasets for (i in 1:N_unknown) { N_slice[N_known+i] = N_known+i; } // Slice variables for feature matrices M_slice = N_slice[:M-1]; } parameters { array[N_T, M] matrix[D,N] U_raw; // feature matrices U array[N_T, M-1] matrix[D,N] V_raw; // feature matrices V real scale; // scale dictating the strenght of ARD effect vector[D] v_ARD; // ARD variances aranged in increasing order with lower bound zero } model { // Gamma prior for scale profile("Scale Prior"){ scale ~ gamma(1e-9, 1e-9); } // Exponential prior on ARD variances profile("ARD Prior"){ v_ARD ~ exponential(0.1); } // priors for feature matrices profile("Feature Matrices"){ for (t in 1:N_T) { target += reduce_sum(ps_feature_matrices, M_slice, grainsize, U_raw[t,:M-1], V_raw[t,:]); to_vector(U_raw[t,M,:,:]) ~ std_normal(); } } // Likelihood function profile("Likelihood"){ target += reduce_sum(ps_like, N_slice, grainsize, y1, cov_f2_f1, mu_y_y_MC, L_y_MC_inv_cov, v, Idx_all, M, N_T, N_MC, N_C, N_known, N_points, v_ARD, U_raw, V_raw, v_MC); } } ```

I do not know if the error might be related to the previous issue raised

Versions

cmdstan 2.34.0 hff4ab46_0 conda-forge cmdstanpy 1.2.1 pyhd8ed1ab_0 conda-forge

OS: Scientific Linux 7.9 (Nitrogen)

ahartikainen commented 4 months ago

I bet it is a problem with paths containing space char. Can you move your location and retest?

Weird thing is that there should be automatic fix for this.

Garren-H commented 4 months ago

I am running the same code by only changing the value of D in the data file. For the case when D=1 the code did not result in any errors, the above was for when D=2. Surely it would've given the same error for when D=1 if this were the case?

ahartikainen commented 4 months ago

yeah, true it would not make any sense

Have you tried add prints to debug what can be seen and when it crashes (or at what step)?

Garren-H commented 4 months ago

Surely it has to be post-processing as the fit can be reconstructed using the from_csv method from the csv files. The thing that bugs me is the final line in the .txt output from stan, which I have not previously encountered.

double free or corruption (!prev)

WardBrian commented 4 months ago

Can you run the model under gdb to get a backtrace from that double free?

I saw your previous issue saying it wasn’t recreatable on another machine, is this one?

Garren-H commented 4 months ago

I have no idea what gdb is?

I will see if I can reproduce the error on my personal device. I do however have less cores on my personal device, so it will take longer. I will report back once this has completed.

WardBrian commented 4 months ago

Do these issues still occur if you remove the use of profile from the model code? We recently fixed some issues with that functionality that could be related

Garren-H commented 4 months ago

I am running the model with and without profiling on the cluster (changed the stan code to remove profiles) to see if the error can be reproduced on the cluster. Will take a few hours to complete

Garren-H commented 4 months ago

I have ran these models and it seems that this error is not reproducible either. Both the model with and without profiling completed without errors this time around