potential tqdm issue

[maxim-h]

Failing to download dataset due to tqdm

Description of the bug

Installed pyega3 (from pip) and tried to download a dataset:

pyega3 -c 30 -cf CREDENTIALS_FILE fetch EGAD00001011283 --output-dir EGAD00001011283

But kept getting multiple errors even on re-try, so I cancelled the download.

Used versions (please complete the following information)

it is in the log 👇

• [x] Please, confirm you have tested your PyEGA3 installation (follow instructions):

To Reproduce

pyega3 -c 30 -cf CREDENTIALS_FILE fetch EGAD00001011283 --output-dir EGAD00001011283

Observed behaviour

Errors and no download

Expected behaviour

No errors and start of the download

Screenshots and error messages

pyega3_output.log

``` [2024-01-30 13:50:04 +0100] pyEGA3 - EGA python client version 5.1.0 (https://github.com/EGA-archive/ega-download-client) [2024-01-30 13:50:04 +0100] Parts of this software are derived from pyEGA (https://github.com/blachlylab/pyega) by James Blachly [2024-01-30 13:50:04 +0100] Python version : 3.12.1 [2024-01-30 13:50:04 +0100] OS version : Linux #1 SMP Wed Jul 21 11:57:15 UTC 2021 [2024-01-30 13:50:04 +0100] Server URL: https://ega.ebi.ac.uk:8443/v2 [2024-01-30 13:50:04 +0100] Session-Id: 2772275406 [2024-01-30 13:50:17 +0100] [2024-01-30 13:50:18 +0100] pyEGA3 - EGA python client version 5.1.0 (https://github.com/EGA-archive/ega-download-client) [2024-01-30 13:50:18 +0100] Parts of this software are derived from pyEGA (https://github.com/blachlylab/pyega) by James Blachly [2024-01-30 13:50:18 +0100] Python version : 3.12.1 [2024-01-30 13:50:18 +0100] OS version : Linux #1 SMP Wed Jul 21 11:57:15 UTC 2021 [2024-01-30 13:50:18 +0100] Server URL: https://ega.ebi.ac.uk:8443/v2 [2024-01-30 13:50:18 +0100] Session-Id: 2829523645 [2024-01-30 13:50:19 +0100] [2024-01-30 13:50:19 +0100] Authentication success for user 'maksim.kholmatov@embl.de' [2024-01-30 13:50:19 +0100] File Id: 'EGAF00008204178'(7061552690 bytes). [2024-01-30 13:50:19 +0100] Total space : 77397.05 GiB [2024-01-30 13:50:19 +0100] Used space : 69862.83 GiB [2024-01-30 13:50:19 +0100] Free space : 7534.22 GiB [2024-01-30 13:50:19 +0100] Download starting [using 30 connection(s), file size 7061552674 and chunk length 104857600]... [2024-01-30 13:50:19 +0100] max() iterable argument is empty Traceback (most recent call last): File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 339, in _download_whole_file self.download_file(output_file, num_connections, max_slice_size) File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 134, in download_file with tqdm(total=int(file_size), unit='B', unit_scale=True) as pbar: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 863, in __init__ self.pos = self._get_free_pos(self) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 470, in _get_free_pos return max(inst.pos for inst in cls._instances ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: max() iterable argument is empty [2024-01-30 13:51:20 +0100] retry attempt 1 [2024-01-30 13:51:21 +0100] Download starting [using 30 connection(s), file size 7061552674 and chunk length 104857600]... [2024-01-30 13:51:21 +0100] 'tqdm' object has no attribute 'pos' Traceback (most recent call last): File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 339, in _download_whole_file self.download_file(output_file, num_connections, max_slice_size) File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 134, in download_file with tqdm(total=int(file_size), unit='B', unit_scale=True) as pbar: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 863, in __init__ self.pos = self._get_free_pos(self) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 470, in _get_free_pos return max(inst.pos for inst in cls._instances ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 470, in return max(inst.pos for inst in cls._instances ^^^^^^^^ AttributeError: 'tqdm' object has no attribute 'pos' ```

Additional context

I have solved the issue by upgrading the tqdm package via pip install tqdm --upgrade. Not that it didn't complain about it:

pip install tqdm

```sh $ pip install tqdm --upgrade Requirement already satisfied: tqdm in /[prefix]/.micromamba/envs/pyEGA3/lib/python3.12/site-packages (4.19.6) Collecting tqdm Downloading tqdm-4.66.1-py3-none-any.whl.metadata (57 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 3.4 MB/s eta 0:00:00 Downloading tqdm-4.66.1-py3-none-any.whl (78 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 1.7 MB/s eta 0:00:00 Installing collected packages: tqdm Attempting uninstall: tqdm Found existing installation: tqdm 4.19.6 Uninstalling tqdm-4.19.6: Successfully uninstalled tqdm-4.19.6 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following depend ency conflicts. pyega3 5.1.0 requires tqdm==4.19.6, but you have tqdm 4.66.1 which is incompatible. Successfully installed tqdm-4.66.1 ```

Despite the apparent error above after the upgrade the initial command worked and the download is currently working.

System information

I use micromamba with default channels just to get the base python installation. Then I install everything with pip.

micromamba list

``` List of packages in environment: "/[prefix]/.micromamba/envs/pyEGA3" Name Version Build Channel ────────────────────────────────────────────────────────── _libgcc_mutex 0.1 main _openmp_mutex 5.1 1_gnu bzip2 1.0.8 h7b6447c_0 ca-certificates 2023.12.12 h06a4308_0 expat 2.5.0 h6a678d5_0 ld_impl_linux-64 2.38 h1181459_1 libffi 3.4.4 h6a678d5_0 libgcc-ng 11.2.0 h1234567_1 libgomp 11.2.0 h1234567_1 libstdcxx-ng 11.2.0 h1234567_1 libuuid 1.41.5 h5eee18b_0 ncurses 6.4 h6a678d5_0 openssl 3.0.12 h7f8727e_0 pip 23.3.1 py312h06a4308_0 python 3.12.1 h996f2a0_0 readline 8.2 h5eee18b_0 setuptools 68.2.2 py312h06a4308_0 sqlite 3.41.2 h5eee18b_0 tk 8.6.12 h1ccaba5_0 tzdata 2023d h04d1e81_0 wheel 0.41.2 py312h06a4308_0 xz 5.4.5 h5eee18b_0 zlib 1.2.13 h5eee18b_0 ```

pip freeze

``` certifi==2023.11.17 charset-normalizer==2.0.12 htsget==0.2.5 humanize==4.9.0 idna==3.6 psutil==5.6.6 pyega3==5.1.0 requests==2.26.0 setuptools==68.2.2 six==1.16.0 tqdm==4.66.1 urllib3==1.26.7 wheel==0.41.2 ```

[joshuamschmidt]

@maxim-h I ran into the same issue with python 3.12. Issue was resolved by downgrading to 3.6

[AndGob]

Thank you @joshuamschmidt and @maxim-h, I encountered the same problem in a conda enviroment. Downgrading python works also for me.
NOTE: All package need to be reisntalled after python downgrading.

[CIBRChenLab]

python=3.8 works for me, I also encountered error related to tqdm and request when using python 3.12