pcangsd didn't converge!!

[NeethuCherukuri]

sbatch script that I'm using!! echo "Job started at: $(date)"

module load singularity/3.10.3

Define paths DATA_DIR="/work/seedpod/output/angsd_spal2/angsd_result.beagle.gz" OUT_PREFIX="/work/seedpod/output/pcangsd/pcangsd_spal" SINGULARITY_IMAGE="/work/seedpod/test/seedpod_latest.sif"

echo "Running pcangsd inside Singularity container..." singularity exec -B "/work:/work" $SINGULARITY_IMAGE /opt/miniconda/bin/pcangsd -b $DATA_DIR -o $OUT_PREFIX -t 32 -e 1 --maf 0.01 --iter 3000 --tole 1e-7

echo "Job finished at: $(date)"

Output file: Job started at: Mon Mar 4 13:00:36 EST 2024 Running pcangsd inside Singularity container... PCAngsd v1.21 Jonas Meisner and Anders Albrechtsen. Using 32 thread(s). Parsing Beagle file. Loaded 730243 sites and 374 individuals. Estimating minor allele frequencies. EM (MAF) converged at iteration: 213 Number of sites after MAF filtering (0.01): 730243 Estimating covariance matrix. Using 1 principal components (manually selected). Individual allele frequencies estimated (1). Individual allele frequencies estimated (2). RMSE=0.103376299 Individual allele frequencies estimated (3). RMSE=0.103394873 Individual allele frequencies estimated (4). RMSE=0.001676962 Individual allele frequencies estimated (5). RMSE=0.001336438 Individual allele frequencies estimated (6). RMSE=0.001089854 Individual allele frequencies estimated (7). RMSE=0.000904074 Individual allele frequencies estimated (8). RMSE=0.00076122 . . . Individual allele frequencies estimated (2996). RMSE=6.68e-07 Individual allele frequencies estimated (2997). RMSE=6.48e-07 Individual allele frequencies estimated (2998). RMSE=6.67e-07 Individual allele frequencies estimated (2999). RMSE=6.62e-07 Individual allele frequencies estimated (3000). RMSE=6.12e-07 Individual allele frequencies estimated (3001). RMSE=6.31e-07 PCAngsd did not converge! Saved covariance matrix as /work/seedpod/output/pcangsd/pcangsd_spal.cov

Total elapsed time: 74m53s Job finished at: Mon Mar 4 14:15:30 EST 2024 ood.discovery.neu.edu_pun_sys_dashboard_files_fs__work_seedpod_test_pcangsd_41265911.out.pdf ood.discovery.neu.edu_pun_sys_dashboard_files_fs__work_seedpod_test_pcangsd_41265911.out.pdf

[Rosemeis]

Hi!

You have set the tolerance to 1e-7 (--tole), which is very low. By the looks of it, PCAngsd should converge a long time before with the default parameter :-)

Best, Jonas

[NeethuCherukuri]

Hello Jonas, I have given all the default parameters and just gave --t 32 for my first run and still got the same error

[laneatmore]

Did you ever figure this out? I have the same issue and it persists with different species/datasets/parameters

[Rosemeis]

Can you send a detailed output like above along with parameters used? :-)

[laneatmore]

Sure. This has happened every time I use PCAngsd, so I've used many different parameters and datasets and always have the same error. The most recent time I tried it is here:

source /arc/project/st-cspeller-1/software/pcangsd/bin/activate PATH_TO_DATA=/scratch/st-cspeller-1/angsd/data PATH_TO_OUT=/scratch/st-cspeller-1/angsd/pcangsd

mkdir -p $PATH_TO_OUT

prefix=$1

pcangsd -b $PATH_TO_DATA/$prefix.beagle.gz -o $PATH_TO_OUT/$prefix -e 4 -t 16 --maf 0.01

---

run with: sbatch pcangsd.sh modern.1.MQ30.post0.95.maf0 (chr1, filtered on MQ>30, posterior SNP probability 0.95, no maf)

---

PCAngsd v1.21 Jonas Meisner and Anders Albrechtsen. Using 16 thread(s).

Parsing Beagle file. Loaded 83875698 sites and 21 individuals. Estimating minor allele frequencies. EM (MAF) converged at iteration: 8 Number of sites after MAF filtering (0.01): 400994

Estimating covariance matrix. Using 4 principal components (manually selected). Individual allele frequencies estimated (1). Individual allele frequencies estimated (2). RMSE=0.233268648 Individual allele frequencies estimated (3). RMSE=0.218888462 Individual allele frequencies estimated (4). RMSE=0.029780695 Individual allele frequencies estimated (5). RMSE=0.016666595 Individual allele frequencies estimated (6). RMSE=0.013546568 Individual allele frequencies estimated (7). RMSE=0.010963074 Individual allele frequencies estimated (8). RMSE=0.008840648 Individual allele frequencies estimated (9). RMSE=0.007934751 Individual allele frequencies estimated (10). RMSE=0.012474069 Individual allele frequencies estimated (11). RMSE=0.037707459 Individual allele frequencies estimated (12). RMSE=0.078537092 Individual allele frequencies estimated (13). RMSE=0.03695567 Individual allele frequencies estimated (14). RMSE=0.014033105 Individual allele frequencies estimated (15). RMSE=0.008964554 Individual allele frequencies estimated (16). RMSE=0.008011876 Individual allele frequencies estimated (17). RMSE=0.007773824 Individual allele frequencies estimated (18). RMSE=0.007561219 Individual allele frequencies estimated (19). RMSE=0.007362758 Individual allele frequencies estimated (20). RMSE=0.006856747 Individual allele frequencies estimated (21). RMSE=0.006291586 Individual allele frequencies estimated (22). RMSE=0.005901046 Individual allele frequencies estimated (23). RMSE=0.005466489 Individual allele frequencies estimated (24). RMSE=0.005244594 Individual allele frequencies estimated (25). RMSE=0.004654536 Individual allele frequencies estimated (26). RMSE=0.004474309 Individual allele frequencies estimated (27). RMSE=0.004179147 Individual allele frequencies estimated (28). RMSE=0.004139526 Individual allele frequencies estimated (29). RMSE=0.003786299 Individual allele frequencies estimated (30). RMSE=0.003697566 Individual allele frequencies estimated (31). RMSE=0.003514667 Individual allele frequencies estimated (32). RMSE=0.003429946 Individual allele frequencies estimated (33). RMSE=0.003331275 Individual allele frequencies estimated (34). RMSE=0.003146685 Individual allele frequencies estimated (35). RMSE=0.003019442 Individual allele frequencies estimated (36). RMSE=0.002939447 Individual allele frequencies estimated (37). RMSE=0.00286618 Individual allele frequencies estimated (38). RMSE=0.002675446 Individual allele frequencies estimated (39). RMSE=0.002660762 Individual allele frequencies estimated (40). RMSE=0.002607391 Individual allele frequencies estimated (41). RMSE=0.002551455 Individual allele frequencies estimated (42). RMSE=0.002408127 Individual allele frequencies estimated (43). RMSE=0.002332172 Individual allele frequencies estimated (44). RMSE=0.002339506 Individual allele frequencies estimated (45). RMSE=0.00232356 Individual allele frequencies estimated (46). RMSE=0.002273365 Individual allele frequencies estimated (47). RMSE=0.002232998 Individual allele frequencies estimated (48). RMSE=0.002179965 Individual allele frequencies estimated (49). RMSE=0.002109344 Individual allele frequencies estimated (50). RMSE=0.002080992 Individual allele frequencies estimated (51). RMSE=0.002086011 Individual allele frequencies estimated (52). RMSE=0.002108455 Individual allele frequencies estimated (53). RMSE=0.002238302 Individual allele frequencies estimated (54). RMSE=0.002356824 Individual allele frequencies estimated (55). RMSE=0.002338551 Individual allele frequencies estimated (56). RMSE=0.002371655 Individual allele frequencies estimated (57). RMSE=0.218219861 Individual allele frequencies estimated (58). RMSE=0.002903154 Individual allele frequencies estimated (59). RMSE=0.002795981 Individual allele frequencies estimated (60). RMSE=0.003145065 Individual allele frequencies estimated (61). RMSE=0.003389859 Individual allele frequencies estimated (62). RMSE=0.0035505 Individual allele frequencies estimated (63). RMSE=0.218218446 Individual allele frequencies estimated (64). RMSE=0.003875953 Individual allele frequencies estimated (65). RMSE=0.00407893 Individual allele frequencies estimated (66). RMSE=0.004160056 Individual allele frequencies estimated (67). RMSE=0.00421347 Individual allele frequencies estimated (68). RMSE=0.004304378 Individual allele frequencies estimated (69). RMSE=0.004278152 Individual allele frequencies estimated (70). RMSE=0.004199763 Individual allele frequencies estimated (71). RMSE=0.004001605 Individual allele frequencies estimated (72). RMSE=0.218217731 Individual allele frequencies estimated (73). RMSE=0.003837258 Individual allele frequencies estimated (74). RMSE=0.003597165 Individual allele frequencies estimated (75). RMSE=0.218218669 Individual allele frequencies estimated (76). RMSE=0.003377041 Individual allele frequencies estimated (77). RMSE=0.003222406 Individual allele frequencies estimated (78). RMSE=0.003081697 Individual allele frequencies estimated (79). RMSE=0.002967563 Individual allele frequencies estimated (80). RMSE=0.002817503 Individual allele frequencies estimated (81). RMSE=0.002813829 Individual allele frequencies estimated (82). RMSE=0.002592889 Individual allele frequencies estimated (83). RMSE=0.002490194 Individual allele frequencies estimated (84). RMSE=0.002221427 Individual allele frequencies estimated (85). RMSE=0.002062748 Individual allele frequencies estimated (86). RMSE=0.00199131 Individual allele frequencies estimated (87). RMSE=0.001960381 Individual allele frequencies estimated (88). RMSE=0.00185379 Individual allele frequencies estimated (89). RMSE=0.001732162 Individual allele frequencies estimated (90). RMSE=0.001607779 Individual allele frequencies estimated (91). RMSE=0.001431088 Individual allele frequencies estimated (92). RMSE=0.001300459 Individual allele frequencies estimated (93). RMSE=0.001236139 Individual allele frequencies estimated (94). RMSE=0.001242036 Individual allele frequencies estimated (95). RMSE=0.001214254 Individual allele frequencies estimated (96). RMSE=0.00120741 Individual allele frequencies estimated (97). RMSE=0.001132747 Individual allele frequencies estimated (98). RMSE=0.001019111 Individual allele frequencies estimated (99). RMSE=0.000910385 Individual allele frequencies estimated (100). RMSE=0.00091651 Individual allele frequencies estimated (101). RMSE=0.000924085 PCAngsd did not converge! Saved covariance matrix as /scratch/st-cspeller-1/angsd/pcangsd/modern.1.MQ30.post0.95.maf0.cov

Total elapsed time: 15m14s

[Rosemeis]

I believe PCAngsd have some trouble converging due to the small sample size! But I would simply try to increase the number of iterations (--iter) to perhaps 500.

Hope it helps! :-)

Best, Jonas

[laneatmore]

That did the trick, thanks so much :)