RitchieLabIGH / IRFinder

MIT License
13 stars 10 forks source link

IRBase2.0 Batch Download #31

Open celinehohzm opened 1 year ago

celinehohzm commented 1 year ago

Hi, I am trying to batch download all the samples on https://irbase.igh.cnrs.fr/samples But I keep running into the error "Http failure response for https://irbase-server.igh.cnrs.fr/api/bulkDownload?samples=683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710: 500 OK" I tried reducing the number of samples selected and/or using a different browser, and sometimes it works, but most of the time it doesn't. May I know how to easily batch download it?

CloXD commented 1 year ago

Hello, I apologize for the inconvenience but It was the first web application I developed in a little time and there was little time to identify and solve those bugs in production.

Here are two tricks to download all the files and metadata:

Metadata as json file:

wget -O ./samples.json "https://irbase-server.igh.cnrs.fr/api/sample?asc=true&pageNumber=0&size=1000&totalElements=944"

To download all the files:

SAMPLE_N=1
while [ $SAMPLE_N -le 948 ] ; do
    wget -O ./Sample_${SAMPLE_N}_nondir.txt.gz "https://irbase-server.igh.cnrs.fr/api/download/Sample_${SAMPLE_N}/IRFinder-IR-nondir.txt.gz"
    SAMPLE_N=$((SAMPLE_N + 1 ))
done

I hope this helps. Cheers Claudio

celinehohzm commented 1 year ago

Thank you so much for your prompt response. I just tried it and it works, thanks a lot!