search

ESPRI-Mod / synda

ESGF Downloader (this is a deprecated repository, the tool has now moved to https://github.com/ESGF/esgf-download)
https://espri-mod.github.io/synda/
21 stars 11 forks source link

Kill download job #75

Open painter1 opened 6 years ago

painter1 commented 6 years ago

Right now, the only clean way to stop a download in progress is with "synda remove...". But this also removes all the files which have been downloaded. Sometimes you want to stop a download job and keep the data which has already come.

painter1 commented 6 years ago

An example is if the GridFTP server fails and you want to switch to another GridFTP node, or use HTTP.

glevava commented 6 years ago

synda remove doesn't really "stop" the download. It purges the involved datasets from the database and remove the corresponding data on the filesystem. To stop the download process, the only way is to stop the service/daemon : service sdt stop. In addition, synda remove is able to delete data at the "dataset" granularity using synda remove <dataset_id>. In the case of a fallback on HTTP protocol when GridFTP downloads failed you can add http_fallback=true in your sdt.conf.

painter1 commented 6 years ago

Yes, the real problem is that you can't stop a download without purging the database and data. In production use, there will be several simultaneous download jobs, from several nodes. So stopping the daemon isn't a practical option. Thanks for telling me about the http_fallback setting. Does that use the same server as for GridFTP? Sometimes they will be different.

painter1 commented 6 years ago

I just tried stopping the daemon. When I restarted the daemon, the download started again. Is there any way to kill the download without losing what's in the database and file system?

painter1 commented 6 years ago

"synda reset" will stop all downloads without purging the database and data. This is enough to get by on, but it would be very helpful if it could be restricted to data defined by a particular selection file.

RuthPetrie commented 6 years ago

I would find this a useful feature request also, as Jeff says there is no way to just stop a particular download. This is fine in testing you can stop the daemon, mess about with the database but when moving to a production sense which we haven't done at CEDA for any projects it would be great to say synda install -s , then you might want synda pause -s , synda stop -s .