mfumagalli / ngsPopGen

Population genetics analyses from NGS data
GNU General Public License v3.0
25 stars 7 forks source link

ngs2dSFS and ngsFst giving "Possible error reading SFS, binary file might be broken..." even after using old SAF #11

Closed sagitaninta closed 11 months ago

sagitaninta commented 11 months ago

Hi Matteo,

I was trying ngsFst for getting some Fst estimates that was mentioned in this paper. My motivation trying this was I can't find any clean outlier from two populations that are supposed to be different when using other Fst estimators in ANGSD.

So what I did was this:

~/bin/ngsPopGen/ngsFST -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -priorfile wildPop.domePop.unfolded.2dsfs -outfile wildPop.domePop.ngsFST -nind 17 39 -nsites 100000 -verbose 1

The .oldsaf.gz files were coming from realSFS print -oldfile 1 as was said in the "Using NGStools/NGSpopgen" section of this page. The .2dsfs file was coming from realSFS of the saf.idx of the 2 populations used there. The wildPop has 17 diploid individuals and the domesticPop has 39 diploid individuals.

The message I get is this

        ->Dumping file: wildPop.domePop.ngsFST
        ->Using some of these args: -nind 56 -nind1 17 -nind2 39 -nsites 100000 -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -priorfiles (null) (null) -priorfile wildPop.domePop.unfolded.2dsfs -outfile wildPop.domePop.ngsFST -verbose 1 -firstbase 1 -block_size 20000

Reading 2D prior...
 num win 5 win0 is 0 19999
Block 0 out of 4 from 0 to 19999

        -> Possible error reading SFS, binary file might be broken...

I thought that was because the 2dSFS not right (unfolded 2DSFS made with ANGSD) so I change into folded 2DSFS made with ANGSD. The message I got was still the same.

Then I was trying to make a 2D SFS file using ngs2dsfs:

~/bin/ngsPopGen/ngs2dSFS -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -outfile wildPop.domePop.ngs2dSFS -nind 34 78 -nsites 100000 -relative 1

        -> Possible error reading SFS, binary file might be broken...

Still the same message.

Do you know how can I get ngsFst working?

mfumagalli commented 11 months ago

Hi Sabhrina,

you should use the latest implementation of FST in ANGSD which is based on this more recent paper https://academic.oup.com/gigascience/article/doi/10.1093/gigascience/giac032/6586813 It is more efficient and possibly precise. ngsPopGen is not actively maintained.

However, ngsPopGen and ANGSD are not expected to give very different estimates of FST. You should check the effect of priors in ANGSD (are you using the 2d-sfs?) to see if that changes your distribution of FST values.

Matteo


From: Sabhrina Aninta @.> Sent: 06 November 2023 3:47 PM To: mfumagalli/ngsPopGen @.> Cc: Subscribed @.***> Subject: [mfumagalli/ngsPopGen] ngs2dSFS and ngsFst giving "Possible error reading SFS, binary file might be broken..." even after using old SAF (Issue #11)

Hi Matteo,

I was trying ngsFst for getting some Fst estimates that was mentioned in this paperhttps://doi.org/10.1534/genetics.113.154740. My motivation trying this was I can't find any clean outlier from two populations that are supposed to be different when using other Fst estimators in ANGSD.

So what I did was this:

~/bin/ngsPopGen/ngsFST -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -priorfile wildPop.domePop.unfolded.2dsfs -outfile wildPop.domePop.ngsFST -nind 17 39 -nsites 100000 -verbose 1

The .oldsaf.gz files were coming from realSFS print -oldfile 1 as was said in the "Using NGStools/NGSpopgen" section of this pagehttp://www.popgen.dk/angsd/index.php/RealSFS. The .2dsfs file was coming from realSFS of the saf.idx of the 2 populations used there. The wildPop has 17 diploid individuals and the domesticPop has 39 diploid individuals.

The message I get is this

    ->Dumping file: wildPop.domePop.ngsFST
    ->Using some of these args: -nind 56 -nind1 17 -nind2 39 -nsites 100000 -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -priorfiles (null) (null) -priorfile wildPop.domePop.unfolded.2dsfs -outfile wildPop.domePop.ngsFST -verbose 1 -firstbase 1 -block_size 20000

Reading 2D prior... num win 5 win0 is 0 19999 Block 0 out of 4 from 0 to 19999

    -> Possible error reading SFS, binary file might be broken...

I thought that was because the 2dSFS not right (unfolded 2DSFS made with ANGSD) so I change into folded 2DSFS made with ANGSD. The message I got was still the same.

Then I was trying to make a 2D SFS file using ngs2dsfs:

~/bin/ngsPopGen/ngs2dSFS -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -outfile wildPop.domePop.ngs2dSFS -nind 34 78 -nsites 100000 -relative 1

    -> Possible error reading SFS, binary file might be broken...

Still the same message.

Do you know how can I get ngsFst working?

— Reply to this email directly, view it on GitHubhttps://github.com/mfumagalli/ngsPopGen/issues/11, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAQ26CP6BWKQRUN5TZPTM73YDEBCPAVCNFSM6AAAAAA67XHZ7GVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE3TSNBXGM4TEMY. You are receiving this because you are subscribed to this thread.Message ID: @.***>

sagitaninta commented 11 months ago

Hi Matteo,

Thanks for your response!

Yes, I was using the 2D SFS as a prior following the example usage.

Is it possible to use other kinds of priors? I can't think of any other possible way of doing this with only two populations.

On Mon, 6 Nov 2023, 21:41 Matteo Fumagalli, @.***> wrote:

Hi Sabhrina,

you should use the latest implementation of FST in ANGSD which is based on this more recent paper

https://academic.oup.com/gigascience/article/doi/10.1093/gigascience/giac032/6586813 It is more efficient and possibly precise. ngsPopGen is not actively maintained.

However, ngsPopGen and ANGSD are not expected to give very different estimates of FST. You should check the effect of priors in ANGSD (are you using the 2d-sfs?) to see if that changes your distribution of FST values.

Matteo


From: Sabhrina Aninta @.> Sent: 06 November 2023 3:47 PM To: mfumagalli/ngsPopGen @.> Cc: Subscribed @.***> Subject: [mfumagalli/ngsPopGen] ngs2dSFS and ngsFst giving "Possible error reading SFS, binary file might be broken..." even after using old SAF (Issue #11)

Hi Matteo,

I was trying ngsFst for getting some Fst estimates that was mentioned in this paperhttps://doi.org/10.1534/genetics.113.154740. My motivation trying this was I can't find any clean outlier from two populations that are supposed to be different when using other Fst estimators in ANGSD.

So what I did was this:

~/bin/ngsPopGen/ngsFST -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -priorfile wildPop.domePop.unfolded.2dsfs -outfile wildPop.domePop.ngsFST -nind 17 39 -nsites 100000 -verbose 1

The .oldsaf.gz files were coming from realSFS print -oldfile 1 as was said in the "Using NGStools/NGSpopgen" section of this page< http://www.popgen.dk/angsd/index.php/RealSFS>. The .2dsfs file was coming from realSFS of the saf.idx of the 2 populations used there. The wildPop has 17 diploid individuals and the domesticPop has 39 diploid individuals.

The message I get is this

->Dumping file: wildPop.domePop.ngsFST ->Using some of these args: -nind 56 -nind1 17 -nind2 39 -nsites 100000 -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -priorfiles (null) (null) -priorfile wildPop.domePop.unfolded.2dsfs -outfile wildPop.domePop.ngsFST -verbose 1 -firstbase 1 -block_size 20000

Reading 2D prior... num win 5 win0 is 0 19999 Block 0 out of 4 from 0 to 19999

-> Possible error reading SFS, binary file might be broken...

I thought that was because the 2dSFS not right (unfolded 2DSFS made with ANGSD) so I change into folded 2DSFS made with ANGSD. The message I got was still the same.

Then I was trying to make a 2D SFS file using ngs2dsfs:

~/bin/ngsPopGen/ngs2dSFS -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -outfile wildPop.domePop.ngs2dSFS -nind 34 78 -nsites 100000 -relative 1

-> Possible error reading SFS, binary file might be broken...

Still the same message.

Do you know how can I get ngsFst working?

— Reply to this email directly, view it on GitHub< https://github.com/mfumagalli/ngsPopGen/issues/11>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/AAQ26CP6BWKQRUN5TZPTM73YDEBCPAVCNFSM6AAAAAA67XHZ7GVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE3TSNBXGM4TEMY>.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/mfumagalli/ngsPopGen/issues/11#issuecomment-1796400747, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGYK74UPGRS2JDIYWCGFE43YDFDQ5AVCNFSM6AAAAAA67XHZ7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJWGQYDANZUG4 . You are receiving this because you authored the thread.Message ID: @.***>

mfumagalli commented 11 months ago

You can use different priors both for the genotypes and the 2D, but if you have low-cov data the "standard" approach you are using should be the best one. If questions, please feel free to contact me directly.


From: Sabhrina Aninta @.> Sent: 06 November 2023 9:18 PM To: mfumagalli/ngsPopGen @.> Cc: Matteo Fumagalli @.>; Comment @.> Subject: Re: [mfumagalli/ngsPopGen] ngs2dSFS and ngsFst giving "Possible error reading SFS, binary file might be broken..." even after using old SAF (Issue #11)

Hi Matteo,

Thanks for your response!

Yes, I was using the 2D SFS as a prior following the example usage.

Is it possible to use other kinds of priors? I can't think of any other possible way of doing this with only two populations.

On Mon, 6 Nov 2023, 21:41 Matteo Fumagalli, @.***> wrote:

Hi Sabhrina,

you should use the latest implementation of FST in ANGSD which is based on this more recent paper

https://academic.oup.com/gigascience/article/doi/10.1093/gigascience/giac032/6586813 It is more efficient and possibly precise. ngsPopGen is not actively maintained.

However, ngsPopGen and ANGSD are not expected to give very different estimates of FST. You should check the effect of priors in ANGSD (are you using the 2d-sfs?) to see if that changes your distribution of FST values.

Matteo


From: Sabhrina Aninta @.> Sent: 06 November 2023 3:47 PM To: mfumagalli/ngsPopGen @.> Cc: Subscribed @.***> Subject: [mfumagalli/ngsPopGen] ngs2dSFS and ngsFst giving "Possible error reading SFS, binary file might be broken..." even after using old SAF (Issue #11)

Hi Matteo,

I was trying ngsFst for getting some Fst estimates that was mentioned in this paperhttps://doi.org/10.1534/genetics.113.154740. My motivation trying this was I can't find any clean outlier from two populations that are supposed to be different when using other Fst estimators in ANGSD.

So what I did was this:

~/bin/ngsPopGen/ngsFST -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -priorfile wildPop.domePop.unfolded.2dsfs -outfile wildPop.domePop.ngsFST -nind 17 39 -nsites 100000 -verbose 1

The .oldsaf.gz files were coming from realSFS print -oldfile 1 as was said in the "Using NGStools/NGSpopgen" section of this page< http://www.popgen.dk/angsd/index.php/RealSFS>. The .2dsfs file was coming from realSFS of the saf.idx of the 2 populations used there. The wildPop has 17 diploid individuals and the domesticPop has 39 diploid individuals.

The message I get is this

->Dumping file: wildPop.domePop.ngsFST ->Using some of these args: -nind 56 -nind1 17 -nind2 39 -nsites 100000 -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -priorfiles (null) (null) -priorfile wildPop.domePop.unfolded.2dsfs -outfile wildPop.domePop.ngsFST -verbose 1 -firstbase 1 -block_size 20000

Reading 2D prior... num win 5 win0 is 0 19999 Block 0 out of 4 from 0 to 19999

-> Possible error reading SFS, binary file might be broken...

I thought that was because the 2dSFS not right (unfolded 2DSFS made with ANGSD) so I change into folded 2DSFS made with ANGSD. The message I got was still the same.

Then I was trying to make a 2D SFS file using ngs2dsfs:

~/bin/ngsPopGen/ngs2dSFS -postfiles wildPop.folded.oldsaf.gz domesticPop.folded.oldsaf.gz -outfile wildPop.domePop.ngs2dSFS -nind 34 78 -nsites 100000 -relative 1

-> Possible error reading SFS, binary file might be broken...

Still the same message.

Do you know how can I get ngsFst working?

— Reply to this email directly, view it on GitHub< https://github.com/mfumagalli/ngsPopGen/issues/11>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/AAQ26CP6BWKQRUN5TZPTM73YDEBCPAVCNFSM6AAAAAA67XHZ7GVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE3TSNBXGM4TEMY>.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/mfumagalli/ngsPopGen/issues/11#issuecomment-1796400747, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGYK74UPGRS2JDIYWCGFE43YDFDQ5AVCNFSM6AAAAAA67XHZ7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJWGQYDANZUG4 . You are receiving this because you authored the thread.Message ID: @.***>

— Reply to this email directly, view it on GitHubhttps://github.com/mfumagalli/ngsPopGen/issues/11#issuecomment-1796505492, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAQ26CLPLIPSW663DXVZM53YDFH3ZAVCNFSM6AAAAAA67XHZ7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJWGUYDKNBZGI. You are receiving this because you commented.Message ID: @.***>

sagitaninta commented 11 months ago

I see. Thanks Matteo! I am closing this then.