search

darrenabbey / ymap

YMAP - Yeast Mapping Analysis Pipeline : An online pipeline for the analysis of yeast genomic datasets.
MIT License
3 stars 0 forks source link

Recommended hardware? #14

Open hbadrane opened 1 year ago

hbadrane commented 1 year ago

Hello, and thank you for this very nice tool. What is the recommended hardware config so we can run let's say a 100 genomes in a reasonable time?

Best Regards, Hassan

darrenabbey commented 1 year ago

My personal computer has an 8 core AMD 64 but CPU and 36 GB of RAM.

When running 7 datasets through YMAP, with 100% allocation of a CPU per dataset, it can take a few hours to get through them all.

Running more datasets at once will slow down execution of each individual dataset, but it still works (assuming no datasets have issues I hadn't accounted for).

If I didn't have anything else to do, I could easily run through 20-30 in a day in batches of 10 at a time, taking about a week to get through 100.

Setting up all the genomes to process at once would mean you wouldn't get any results until everything is done, so I tend not to go this route.

Processing through datasets at a much faster rate would likely require a computer cluster, but I have had no experience with setting up YMAP in that environment.

On Sun, Apr 30, 2023, 11:48 AM hbadrane @.***> wrote:

Hello, and thank you for this very nice tool. What is the recommended hardware config so we can run let's say a 100 genomes in a reasonable time?

Best Regards, Hassan

— Reply to this email directly, view it on GitHub https://github.com/darrenabbey/ymap/issues/14, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACR64RGYCSFF2OSOXXK4LCLXD2JXHANCNFSM6AAAAAAXRA7NXE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

darrenabbey commented 1 year ago

Also, at this time, YMAP does not take advantage of GPUs for faster processing of parallelizable tasks.

On Sun, Apr 30, 2023, 11:48 AM hbadrane @.***> wrote:

Hello, and thank you for this very nice tool. What is the recommended hardware config so we can run let's say a 100 genomes in a reasonable time?

Best Regards, Hassan

— Reply to this email directly, view it on GitHub https://github.com/darrenabbey/ymap/issues/14, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACR64RGYCSFF2OSOXXK4LCLXD2JXHANCNFSM6AAAAAAXRA7NXE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

hbadrane commented 1 year ago

Hey Darren, thank you so much for the inputs. The reason we are looking into installing this locally, is that my colleague was using the UMN server but it looks like they have to take it down for some security reasons, so we can't access it anymore. Will stay in touch...

darrenabbey commented 1 year ago

I'm actively working with the Myer's lab at UMN to get it back online. I'm expecting it'll be up in the next couple weeks.

I'd spend much of the last few months rewriting parts of YMAP to resolve the security issues that had been found.

That said, I can still recommend running it on a local server as data upload times will be reduced and there would be less security concerns in the future.

On Wed, May 3, 2023, 12:12 PM hbadrane @.***> wrote:

Hey Darren, thank you so much for the inputs. The reason we are looking into installing this locally, is that my colleague was using the UMN server but it looks like they have to take it down for some security reasons, so we can't access it anymore. Will stay in touch...

— Reply to this email directly, view it on GitHub https://github.com/darrenabbey/ymap/issues/14#issuecomment-1533414056, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACR64RCCBFEWZQSQKREVPATXEKGYZANCNFSM6AAAAAAXRA7NXE . You are receiving this because you commented.Message ID: @.***>

hbadrane commented 1 year ago

Yes, this is a so nice tool, that we should have a local copy running. Do you think this can work on AWS ?

darrenabbey commented 1 year ago

Right now much of YMAP is Matlab based. I'm not sure Matlab will run in that environment.

Eventually I plan to rewrite the Matlab portions into something that is more accessible, like Python. That would probably be YMAP2. At that point it would easily run on cloud services.

On Wed, May 3, 2023, 2:25 PM hbadrane @.***> wrote:

Yes, this is a so nice tool, that we should have a local copy running. Do you think this can work on AWS ?

— Reply to this email directly, view it on GitHub https://github.com/darrenabbey/ymap/issues/14#issuecomment-1533579666, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACR64REIRATY6BTU7G4X2DTXEKWLBANCNFSM6AAAAAAXRA7NXE . You are receiving this because you commented.Message ID: @.***>

hbadrane commented 1 year ago

I have no knowledge so far about AWS services, but why Matlab won't work? So you mean it's best just to have a local lunix machine?

darrenabbey commented 1 year ago

I haven't played with AWS. I just know setting up Matlab requires a license and that maybe be a complication in that environment.

On Wed, May 3, 2023, 2:58 PM hbadrane @.***> wrote:

I have no knowledge so far about AWS services, but why Matlab won't work? So you mean it's best just to have a local lunix machine?

— Reply to this email directly, view it on GitHub https://github.com/darrenabbey/ymap/issues/14#issuecomment-1533652831, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACR64RFGIVX6P2CZSDAHXWLXEK2GBANCNFSM6AAAAAAXRA7NXE . You are receiving this because you commented.Message ID: @.***>

hbadrane commented 1 year ago

I see ... thanks.