Open mevers opened 4 years ago
This bug also appears to affect S3.remote in version 6.0.5.
This is slightly old but I feel that I should mention that I would not set up a job that will run frequently to pull data from ensembl every time. You should pull it once and use it often. We should try to respect their bandwidth. Perhaps pull it to an S3 bucket and hit that over and over?
@xguse The whole point of incorporating a rule into snakemake
to download data from external resources is to be mindful of bandwidth: only download once and in the presence of upstream changes. As such, I don't see the point of suggesting an S3 bucket. If anything this introduces the need for and dependence on another API-specific process.
This question is a follow-up/bump to unresolved issue #1275 on BitBucket, which I cannot access anymore (old link).
Issue
FTP remote slows down DAG building. This happens every time regardless of whether the file has already been downloaded or not. Running
snakemake
with--debug-dag
suggests thatFTP.remote
slows down the DAG building step.I'm curious to hear what the recommended way is to download files through FTP. An alternative could be the use of
wget
orcurl
butsnakemake.remote.FTP
seems to be the moresnakemake
-canonical approach.Minimal example
Consider the following
Snakefile
which downloads two chromosome sequences from Ensembl through FTPRunning the minimal workflow with
--debug-dag
snakemake
version