ynop / audiomate

Python library for handling audio datasets.
https://audiomate.readthedocs.io/
MIT License
131 stars 26 forks source link

Retry download if closed connection without response #134

Open CostanzoPablo opened 4 years ago

CostanzoPablo commented 4 years ago

If server close connection, you dont have a "response code" and fail into a Exception.

Proof's:

`/usr/local/lib/python3.6/dist-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location. Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0. from numba.decorators import jit as optional_jit Downloading voxforge-es ... INFO:audiomate:Found 2244 available archives to download Download Files: 0%| | 0/2244 [00:00<?, ?it/s]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/ARI-20100216-hds.tgz" with size: 1361655 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AbdielNavarro-20130409-xvp.tgz" with size: 2625640 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AGOs-20100831-jdm.tgz" with size: 1915692 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AGOs-20100831-tmi.tgz" with size: 2009016 B INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Adrian-20120203-lit.tgz" with size: 2607824 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AdrianRamirezEscalante-20111202-pgl.tgz" with size: 2594505 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 0%| | 1/2244 [00:12<7:30:05, 12.04s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Adrin-20120203-uup.tgz" with size: 2230035 B INFO:audiomate:Finished download Download Files: 0%|▎ | 5/2244 [00:18<5:31:54, 8.89s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Aguatacuperche-20100823-zvv.tgz" with size: 2132406 B INFO:audiomate:Finished download Download Files: 0%|▍ | 7/2244 [00:21<4:11:57, 6.76s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Agus-20120528-sza.tgz" with size: 1533718 B INFO:audiomate:Finished download Download Files: 0%|▌ | 8/2244 [00:28<4:06:45, 6.62s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AlexVixgeck-20090827-qms.tgz" with size: 1105585 B INFO:audiomate:Finished download Download Files: 0%|▌ | 9/2244 [00:29<3:05:11, 4.97s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AlanNaiman-20121014-tke.tgz" with size: 2065020 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alejandra-20111109-oaf.tgz" with size: 1311472 B INFO:audiomate:Finished download Download Files: 0%|▋ | 10/2244 [00:37<3:37:16, 5.84s/it]INFO:audiomate:Finished download Download Files: 0%|▋ | 11/2244 [00:38<2:45:46, 4.45s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091127-dls.tgz" with size: 2144399 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091127-wrq.tgz" with size: 2084240 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AlexanderRubioCaceres-20090902-set.tgz" with size: 1260802 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-fan.tgz" with size: 2444151 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091127-dis.tgz" with size: 2013014 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-ozc.tgz" with size: 2397126 B INFO:audiomate:Finished download Download Files: 1%|▊ | 13/2244 [00:46<2:39:44, 4.30s/it]INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 1%|▉ | 14/2244 [00:53<3:14:10, 5.22s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20100520-svz.tgz" with size: 2348385 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-ygd.tgz" with size: 2095451 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-zoj.tgz" with size: 2226773 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20100520-wvo.tgz" with size: 2189445 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-suc.tgz" with size: 2266510 B INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 1%|█▏ | 19/2244 [01:06<2:43:43, 4.42s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20110915-mnf.tgz" with size: 2442328 B INFO:audiomate:Finished download Download Files: 1%|█▍ | 23/2244 [01:08<2:00:56, 3.27s/it]INFO:audiomate:Finished download Download Files: 1%|█▌ | 24/2244 [01:09<1:38:17, 2.66s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20110915-sff.tgz" with size: 2128985 B INFO:audiomate:Finished download Download Files: 1%|█▋ | 25/2244 [01:12<1:44:04, 2.81s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20110915-wfh.tgz" with size: 2021008 B INFO:audiomate:Finished download Download Files: 1%|█▋ | 26/2244 [01:20<2:35:21, 4.20s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alfredo-20090314-you.tgz" with size: 2117498 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alfredo-20090314-bwm.tgz" with size: 1780407 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alfredo-20090314-nui.tgz" with size: 2000040 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreaBurgos-20120424-pef.tgz" with size: 1929018 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 1%|█▊ | 27/2244 [01:32<4:01:22, 6.53s/it]INFO:audiomate:Finished download Download Files: 1%|█▊ | 28/2244 [01:33<2:58:04, 4.82s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130708-inn.tgz" with size: 1721089 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-dtw.tgz" with size: 1736300 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130708-eaz.tgz" with size: 1909391 B INFO:audiomate:Finished download Download Files: 1%|█▉ | 30/2244 [01:38<2:35:58, 4.23s/it]INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-bhv.tgz" with size: 1776404 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-iqt.tgz" with size: 1986977 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 1%|██ | 31/2244 [01:45<3:04:53, 5.01s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-ttq.tgz" with size: 1749826 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-stu.tgz" with size: 1753877 B INFO:audiomate:Finished download Download Files: 1%|██▏ | 33/2244 [01:49<2:27:32, 4.00s/it]INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-rat.tgz" with size: 1553412 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 2%|██▎ | 36/2244 [01:55<2:07:09, 3.46s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-twg.tgz" with size: 1853932 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130720-mkx.tgz" with size: 1973814 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130720-gix.tgz" with size: 1687520 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130711-jje.tgz" with size: 1589503 B INFO:audiomate:Finished download Download Files: 2%|██▌ | 39/2244 [02:02<1:55:15, 3.14s/it]INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AntonioH-20160531-gwl.tgz" with size: 849178 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 2%|██▌ | 40/2244 [02:08<2:22:21, 3.88s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Andros-20120408-age.tgz" with size: 2205975 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Argorn-20141126-igm.tgz" with size: 2165540 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Ases-20120213-esa.tgz" with size: 2552878 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130720-slz.tgz" with size: 1946756 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AntonioMedina-20121208-kzq.tgz" with size: 2329390 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 2%|██▊ | 43/2244 [02:22<2:33:13, 4.18s/it]INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Ashrey-20100714-yhd.tgz" with size: 1937711 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-kwe.tgz" with size: 2128918 B INFO:audiomate:Finished download Download Files: 2%|██▉ | 46/2244 [02:26<2:01:53, 3.33s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-tsj.tgz" with size: 1742771 B INFO:audiomate:Finished download Download Files: 2%|███▏ | 49/2244 [02:32<1:44:45, 2.86s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-vvx.tgz" with size: 2222457 B INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Azul4-20131019-ssk.tgz" with size: 2093735 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Campotrigo-20130607-bnz.tgz" with size: 3093233 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/CarlosEspinoAngulo-20140525-vip.tgz" with size: 2302281 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-xzd.tgz" with size: 1994095 B INFO:audiomate:Finished download INFO:audiomate:Finished download Download Files: 2%|███▎ | 50/2244 [02:49<4:22:27, 7.18s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/CarlosMedina-20130524-wqx.tgz" with size: 2357299 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/CarlosFiebigCosmelli-20121128-aud.tgz" with size: 1756883 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Carlos_Daniel-20120325-khj.tgz" with size: 2414791 B INFO:audiomate:Finished download INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Condor-20120214-prz.tgz" with size: 1716730 B INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Bernardo-20101210-tiq.tgz" with size: 1849584 B INFO:audiomate:Finished download INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Cixert-20150207-ywe.tgz" with size: 1980326 B INFO:audiomate:Finished download Download Files: 3%|███▊ | 58/2244 [03:05<1:56:45, 3.20s/it] Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse response.begin() File "/usr/lib/python3.6/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.6/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 727, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 403, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/local/lib/python3.6/dist-packages/urllib3/packages/six.py", line 734, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse response.begin() File "/usr/lib/python3.6/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.6/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/DeepSpeech/deepspeech-polyglot/preprocessing/download_data.py", line 108, in main() File "/DeepSpeech/deepspeech-polyglot/preprocessing/download_data.py", line 97, in main dl.download(os.path.join(args.target_path, "voxforge")) File "/usr/local/lib/python3.6/dist-packages/audiomate/corpus/io/base.py", line 40, in download return self._download(target_path) File "/usr/local/lib/python3.6/dist-packages/audiomate/corpus/io/voxforge.py", line 73, in _download downloaded = self.download_files(available, temp_folder) File "/usr/local/lib/python3.6/dist-packages/audiomate/corpus/io/voxforge.py", line 110, in download_files dl_result = download.download_files(url_to_target, num_threads=self.num_workers) File "/usr/local/lib/python3.6/dist-packages/audiomate/utils/download.py", line 35, in download_files description='Download Files' File "/usr/local/lib/python3.6/dist-packages/audiomate/logutil.py", line 21, in progress for x in tqdm.tqdm(iterable, desc=description, total=total): File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1165, in iter for obj in iterable: File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next raise value File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, kwds)) File "/usr/local/lib/python3.6/dist-packages/audiomate/utils/download.py", line 43, in _download_file return download_file(item[0], item[1]) File "/usr/local/lib/python3.6/dist-packages/audiomate/utils/download.py", line 62, in download_file r = requests.get(url, stream=True) File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 530, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))`