ngiangre
commented
7 years ago I agree that having a simple pipeline is more appealing, and having illumina advocate it is a huge plus. I’m gonna get through what I can tonight and then we can decide tomorrow where to focus our efforts. I think for this project having something that takes advantage of HPCs and can scale is imperitive, and R unfortunately isn’t the most scalable environment. However it is great for data science-y/visualization methods, so like you were saying for downstream things, I think R would come in handy there. Maybe we can make it I/O compatible i.e. make python output able to be loaded into R.
From: Eric Karlins notifications@github.com Reply-To: NCBI-Hackathons/Global_Screening_Arrays reply@reply.github.com Date: Monday, March 20, 2017 at 7:59 PM To: NCBI-Hackathons/Global_Screening_Arrays Global_Screening_Arrays@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [NCBI-Hackathons/Global_Screening_Arrays] Tentative plan for creating gtc pipeline (#8)
Even with an existing R package that has similar functions, a working command line pipeline starting with idats (ideally) or gtc files could make this process a lot easier to run. Wrapping this in a workflow management system like Snakemake, which can submit jobs to an HPC cluster and track I/O will make parallel processing easy and run time should be really quick, after the initial upfront cost of generating an egt file in Genome Studio. Luckily, groups dealing with Illumina data will already be comfortable with Illumina files so generating an egt and using a bpm should be pretty straightforward. Hopefully, after that the pipeline will run with essentially one command.
Another plus for this approach is that if we contact Illumina about our tool, as Ben suggested, they may like that we are using some of their existing software as part of it (like the gtc parsing library). He's right, if they mention our tool it could mean a lot of exposure. In addition, I think this approach has a clearer implementation than the R approach. And since we need to get most of this project done tomorrow, that's appealing to me.
I have some experience with PennCNV, so I'll start with that. But adding other command line CNV callers should be pretty easy as well. And since they will all run in parallel, it doesn't add much cost (if you have the compute) to run more CNV callers. Since CNV calling in general has a fairly high FP rate, using multiple callers may be the way to go.
Though a lot of the code will be python, it will also involve a lot of testing of command line programs. Both testing CNV callers that we download, like PennCNV, and also testing scripts that we write. So knowing how to write python code is not the only thing needed in order to get this pipeline working.
Here's a rough idea of what the pipeline would look like:
| wine AutoConvert.exe /path/to/idats /path/to/out/dir manifest.bpm clusterFile.egt ##hopefully this works | \/ gtc files | \/ "scripts/gtc2PennCNV.py" ##I've already written this script, tested it, and pushed it to this repo | ||
|---|---|---|---|---|
| / run PennCnv ##There will also be generation of a couple of PennCNV specific reference files which is another up front cost. |
|---|
/ There are some existing scripts for post-processing on PennCNV output that we can try if we have time.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
ekarlins
DCGenomics
slsevilla
Even with an existing R package that has similar functions, a working command line pipeline starting with idats (ideally) or gtc files could make this process a lot easier to run. Wrapping this in a workflow management system like Snakemake, which can submit jobs to an HPC cluster and track I/O will make parallel processing easy and run time should be really quick, after the initial upfront cost of generating an egt file in Genome Studio. Luckily, groups dealing with Illumina data will already be comfortable with Illumina files so generating an egt and using a bpm should be pretty straightforward. Hopefully, after that the pipeline will run with essentially one command.
Another plus for this approach is that if we contact Illumina about our tool, as Ben suggested, they may like that we are using some of their existing software as part of it (like the gtc parsing library). He's right, if they mention our tool it could mean a lot of exposure. In addition, I think this approach has a clearer implementation than the R approach. And since we need to get most of this project done tomorrow, that's appealing to me.
I have some experience with PennCNV, so I'll start with that. But adding other command line CNV callers should be pretty easy as well. And since they will all run in parallel, it doesn't add much cost (if you have the compute) to run more CNV callers. Since CNV calling in general has a fairly high FP rate, using multiple callers may be the way to go.
Though a lot of the code will be python, it will also involve a lot of testing of command line programs. Both testing CNV callers that we download, like PennCNV, and also testing scripts that we write. So knowing how to write python code is not the only thing needed in order to get this pipeline working.
Here's a rough idea of what the pipeline would look like:
wine AutoConvert.exe /path/to/idats /path/to/out/dir manifest.bpm clusterFile.egt ##hopefully this works
"scripts/gtc2PennCNV.py" ##I've already written this script, tested it, and pushed it to this repo
run PennCnv ##There will also be generation of a couple of PennCNV specific reference files which is another up front cost.
There are some existing scripts for post-processing on PennCNV output that we can try if we have time.