Open SHuang-Broad opened 2 years ago
Hi Steve,
Interesting suggestion! I have to admit I don't immediately know on how to adapt the code for this. Since you ask for pysam you are mainly interested in bam/cram files as input? Which you would then specify using an URL?
Cheers, Wouter
Our current pipeline uses Google Cloud Storage (gs://...
), but I could see users benefit from support for all major cloud service providers, e.g. AWS, Azure.
If Nanoplot only access the BAM through pysam
, then probably that's the dependency that needs to support streaming. And the change will be minimal.
This is definitely an optimization, so it's not an urgent need.
Regarding supporting gs://...
path, I think the following link might be useful.
https://github.com/pysam-developers/pysam/issues/592
Do you have such a (public?) gs://... path for me to test things on? All our data is processed locally.
we don't have any public data to share (definitely because downloading data from cloud storage incurs costs on the owner of the data unless something like requester pay is specified, so this could easily be abused by malicious actors).
I think these from DeepVariant team themselves might work, but may require you to set up a google cloud account: https://console.cloud.google.com/storage/browser/deepvariant/pacbio-case-study-testdata?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false
I'm sorry if this is too much trouble. Thanks for getting on top of this! Steve
Hi,
we are routinely using Nanoplot in our cloud-native pipelines and would love to see Nanoplot support streaming from cloud strages.
Based on a quick glimpse of the code, it looks like that would require at least one dependency, i.e.
pysam
to support that. Are there any other "patches" necessary to support the streaming?Thanks, Steve