Using FUBAR cache to resume a stalled run?

[hayleyjaywilson]

Hello I am (new to) using FUBAR to determine selection in my alignment. I am working on a slurm system where my job runtime is limited to 12 hours. My FUBAR job runs for the 12 hours and then exits without completing. Am I right in thinking the .cache stores this information so that a run can be resumed from where it stopped rather than restarting? I have tried to resubmit this job but it seems like it just starts from scratch each time. My command is:

hyphy fubar ENV="USE_MEMORY_SAVING_DATA_STRUCTURES=1e8;" --alignment FINAL_for_fubar.fasta --tree FINAL_for_dnds_only.final_tree.tree

Am I missing something? Apologies if this is obvious. Thanks

[spond]

Dear @hayleyjaywilson,

How big is your alignment? FUBAR will save "milestone" checkpoints, so it doesn't have to redo them, but it does not store partial step information. In other words, if a step (e.g. fitting some model stage) takes > 12 hours, then it won't be cached.

I am always interested in testing interesting use cases, so if you don't mind sharing the input file with me, I'll be happy to run and report on what is going on.

Best, Sergei

[hayleyjaywilson]

Hi Sergei,

My alignment is 1200 isolates with a ~2Mbp genome. Ah so yes maybe we aren't managing to hit the first milestone.

Happy to email you over the alignment and tree, that would be very helpful. Which address is best to use?

Thanks

Hayley

---

From: Sergei Pond @.> Sent: 13 February 2023 14:22 To: veg/hyphy @.> Cc: Hayley Joanna Wilson @.>; Mention @.> Subject: Re: [veg/hyphy] Using FUBAR cache to resume a stalled run? (Issue #1569)

Dear @hayleyjaywilsonhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhayleyjaywilson&data=05%7C01%7Chjw58%40universityofcambridgecloud.onmicrosoft.com%7C06c4e4ebae6c48d71dae08db0dcdbe1a%7C49a50445bdfa4b79ade3547b4f3986e9%7C1%7C0%7C638118949524849374%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VcJ5HC0MtdW%2FZcvVgAgtiOu%2Bevqn%2FKLZrZ4SZR0xLFQ%3D&reserved=0,

How big is your alignment? FUBAR will save "milestone" checkpoints, so it doesn't have to redo them, but it does not store partial step information. In other words, if a step (e.g. fitting some model stage) takes > 12 hours, then it won't be cached.

I am always interested in testing interesting use cases, so if you don't mind sharing the input file with me, I'll be happy to run and report on what is going on.

Best, Sergei

— Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fveg%2Fhyphy%2Fissues%2F1569%23issuecomment-1428022811&data=05%7C01%7Chjw58%40universityofcambridgecloud.onmicrosoft.com%7C06c4e4ebae6c48d71dae08db0dcdbe1a%7C49a50445bdfa4b79ade3547b4f3986e9%7C1%7C0%7C638118949524849374%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FgrRAWPa7anp1sv4YLBnxO1S%2FvJC%2FXmOb7FZtMGqiUo%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATR6G7VVRR5SHS7N4PXH47TWXI7SJANCNFSM6AAAAAAUZ7LLMQ&data=05%7C01%7Chjw58%40universityofcambridgecloud.onmicrosoft.com%7C06c4e4ebae6c48d71dae08db0dcdbe1a%7C49a50445bdfa4b79ade3547b4f3986e9%7C1%7C0%7C638118949524849374%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=w0jocWvkfedSWeA81I%2BelFL7MnNIQqrb6l4pPYXNwbI%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

[spond]

Dear @hayleyjaywilson,

Yeah, that might do it. I am curious to benchmark this kind of alignment because I don't routinely come across something this long. As this is likely a big file, please use the following dropbox link to send it to me (thanks!). It'll stay open for a few days.

https://www.dropbox.com/request/8tvfcvj8R6yZLqbLN7Pq

Best, Sergei

[spond]

Dear @hayleyjaywilson,

Got your file, working through it. I don't think I've ever had such a large and LONG alignment to run HyPhy on, so it's exposing some interesting behavior. Thanks for sharing -- I'll post the results here when I have them, and I will probably also spend some time optimizing the code for the next release so such "large and long" alignments are processed more efficiently.

Stay tuned.

Best, Sergei

[hayleyjaywilson]

Hi Sergei,

Thanks so much. This is a great help to me then also as it is my first time using the program so I am probably not aware of some of the minutiae that might make a difference! I look forward to the outcome!

Hayley

---

From: Sergei Pond @.> Sent: 15 February 2023 13:58 To: veg/hyphy @.> Cc: Hayley Joanna Wilson @.>; Mention @.> Subject: Re: [veg/hyphy] Using FUBAR cache to resume a stalled run? (Issue #1569)

Dear @hayleyjaywilsonhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhayleyjaywilson&data=05%7C01%7Chjw58%40universityofcambridgecloud.onmicrosoft.com%7C7843ef89377540c825aa08db0f5cbae2%7C49a50445bdfa4b79ade3547b4f3986e9%7C1%7C0%7C638120663160737774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=L4lnjBtXzZnKcfC98IgCD%2F6Uzaa6YfPgYYbFv779XNw%3D&reserved=0,

Got your file, working through it. I don't think I've ever had such a large and LONG alignment to run HyPhy on, so it's exposing some interesting behavior. Thanks for sharing -- I'll post the results here when I have them, and I will probably also spend some time optimizing the code for the next release so such "large and long" alignments are processed more efficiently.

Stay tuned.

Best, Sergei

— Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fveg%2Fhyphy%2Fissues%2F1569%23issuecomment-1431412020&data=05%7C01%7Chjw58%40universityofcambridgecloud.onmicrosoft.com%7C7843ef89377540c825aa08db0f5cbae2%7C49a50445bdfa4b79ade3547b4f3986e9%7C1%7C0%7C638120663160737774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=uZVpJ8HVC%2Fx%2Fsny4X1fw1Nm89AWqaMevzFIGRLZ84hE%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATR6G7WO5NDCYIOKBYHABADWXTOIRANCNFSM6AAAAAAUZ7LLMQ&data=05%7C01%7Chjw58%40universityofcambridgecloud.onmicrosoft.com%7C7843ef89377540c825aa08db0f5cbae2%7C49a50445bdfa4b79ade3547b4f3986e9%7C1%7C0%7C638120663160737774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=joIoPS0VT%2FlmlHIi%2BBne%2FTvouqK0FetAWYusiQ5501I%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

[spond]

Dear @hayleyjaywilson,

In 999/1000 cases HyPhy is used for single gene alignments, so we are talking on the order of 1000s bp. Occasionally it has been used on smaller LONG alignments (~10s of sequences and >>1000s bp). I think this is the first case I know of when there are >1000 sequences and >1000000bp. As a software developer, I really appreciate the chance to stress test things and find ways to improve HyPhy performance.

Best, Sergei

[github-actions[bot]]

Stale issue message

[hayleyjaywilson]

Hi,

This has been marked as completed - did you get anywhere with stress testing?

Hayley

---

From: Sergei Pond @.> Sent: 15 February 2023 14:23 To: veg/hyphy @.> Cc: Hayley Joanna Wilson @.>; Mention @.> Subject: Re: [veg/hyphy] Using FUBAR cache to resume a stalled run? (Issue #1569)

Dear @hayleyjaywilsonhttps://github.com/hayleyjaywilson,

In 999/1000 cases HyPhy is used for single gene alignments, so we are talking on the order of 1000s bp. Occasionally it has been used on smaller LONG alignments (~10s of sequences and >>1000s bp). I think this is the first case I know of when there are >1000 sequences and >1000000bp. As a software developer, I really appreciate the chance to stress test things and find ways to improve HyPhy performance.

Best, Sergei

— Reply to this email directly, view it on GitHubhttps://github.com/veg/hyphy/issues/1569#issuecomment-1431451056, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATR6G7RNMHL6EMX24J6HKNLWXTRH3ANCNFSM6AAAAAAUZ7LLMQ. You are receiving this because you were mentioned.Message ID: @.***>

[spond]

Dear @hayleyjaywilson,

GitHub automation sometimes closes issues due to inactivity. I performed some diagnostics and realized that the current implementation of HyPhy does not scale for such long alignments. I started doing some optimizations to improve the performance but didn't quite finish.

Let me circle back to it. I have an idea for an algorithmic improvement for such large datasets (it's based on https://pubmed.ncbi.nlm.nih.gov/34734192/) which allows much faster approximate estimation on such long alignments).

Best, Sergei