Add progress bars for downloading

[cthoyt]

Closes #60

• [x] urllib support (inspired by https://gist.github.com/leimao/37ff6e990b3226c2c9670a2cd1e4a6f5)
• [x] requests support (inspired by https://stackoverflow.com/a/63831344/5775947)

Example usage:

import pystow

if __name__ == '__main__':
    url = "https://zenodo.org/record/7633479/files/emojis.zip?download=1"
    path = pystow.ensure(
        "test",
        force=True,
        url=url,
        name="emojis.zip",
        download_kwargs=dict(progress_bar=True, backend="requests", tqdm_kwargs=dict(leave=True)),
    )
    print("download succeeded to ", path)

[codecov-commenter]

Codecov Report

Merging #61 (b18bec0) into main (d7fa850) will decrease coverage by 0.19%. The diff coverage is 56.25%.

@@            Coverage Diff             @@
##             main      #61      +/-   ##
==========================================
- Coverage   66.98%   66.80%   -0.19%     
==========================================
  Files          10       10              
  Lines         933      946      +13     
  Branches      194      198       +4     
==========================================
+ Hits          625      632       +7     
- Misses        287      291       +4     
- Partials       21       23       +2     

Impacted Files	Coverage Δ
src/pystow/utils.py	61.91% <56.25%> (-0.34%)	:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[cthoyt]

Hi @sgbaird @glass-ships, could you please test this out and let me know what you think?

[glass-ships]

will do, thanks @cthoyt !!

[glass-ships]

Hmm, I don't notice any change in behavior.
Is it because I'm using methods like ensure_open_sqlite_gz()? (Does download() get called by these other ensure methods / could it be implemented?)

For reference, the code is just ensuring a zipped SQL db and Solr db

[cthoyt]

This behavior isn't enabled by default, you'll have to do something like:

import pystow

if __name__ == '__main__':
    url = "https://zenodo.org/record/7633479/files/emojis.zip?download=1"
    path = pystow.ensure(
        "test",
        force=True,
        url=url,
        name="emojis.zip",
        download_kwargs=dict(progress_bar=True, backend="requests", tqdm_kwargs=dict(leave=True)),
    )
    print("download succeeded to ", path)

[glass-ships]

This behavior isn't enabled by default, you'll have to do something like:

import pystow

if __name__ == '__main__':
    url = "https://zenodo.org/record/7633479/files/emojis.zip?download=1"
    path = pystow.ensure(
        "test",
        force=True,
        url=url,
        name="emojis.zip",
        download_kwargs=dict(progress_bar=True, backend="requests", tqdm_kwargs=dict(leave=True)),
    )
    print("download succeeded to ", path)

Hmmm, still no luck. Sorry I think I'm missing something here. Updated the call to:

with monarchstow.ensure_open_sqlite_gz(
    "sql", 
    url=SQL_DATA_URL, 
    force=update, 
    download_kwargs=dict(progress_bar=True, backend="requests", tqdm_kwargs=dict(leave=True))
  ) as db:

But I'm getting TypeError: urlretrieve() got an unexpected keyword argument 'progress_bar' or TypeError: Session.request() got an unexpected keyword argument 'progress_bar'

Full traceback

```python ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ D:\dev\tislab\monarch-py\src\monarch_py\sql_cli.py:20 in entity │ │ │ │ 17 │ │ update (bool): = Whether to re-download the Monarch KG. Default False │ │ 18 │ """ │ │ 19 │ data = SQLImplementation() │ │ ❱ 20 │ entity = data.get_entity(id, update) │ │ 21 │ if not entity: │ │ 22 │ │ print(f"\nEntity '{id}' not found.\n") │ │ 23 │ │ typer.Abort() │ │ │ │ ╭───────── locals ─────────╮ │ │ │ data = │ │ │ ╰───────────────────────────────────────────────────────────────────────────╯ │ │ │ │ D:\dev\tislab\monarch-py\.venv\lib\site-packages\pystow\impl.py:1485 in ensure_open_sqlite_gz │ │ │ │ 1482 │ │ """ │ │ 1483 │ │ import sqlite3 │ │ 1484 │ │ │ │ ❱ 1485 │ │ path = self.ensure_gunzip( │ │ 1486 │ │ │ *subkeys, url=url, name=name, force=force, download_kwargs=download_kwargs │ │ 1487 │ │ ) │ │ 1488 │ │ with closing(sqlite3.connect(path.as_posix())) as conn: │ │ │ │ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │ │ │ download_kwargs = { │ │ │ │ │ 'progress_bar': True, │ │ │ │ │ 'backend': 'requests', │ │ │ │ │ 'tqdm_kwargs': {'leave': True} │ │ │ │ } │ │ │ │ force = False │ │ │ │ name = None │ │ │ │ self = │ │ │ │ sqlite3 = │ │ │ │ subkeys = ('sql',) │ │ │ │ url = 'https://data.monarchinitiative.org/monarch-kg-dev/latest/monarch-kg.db.g… │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ D:\dev\tislab\monarch-py\.venv\lib\site-packages\pystow\impl.py:297 in ensure_gunzip │ │ │ │ 294 │ │ gunzipped_path = self.join(*subkeys, name=gunzipped_name, ensure_exists=True) │ │ 295 │ │ if gunzipped_path.is_file() and not force: │ │ 296 │ │ │ return gunzipped_path │ │ ❱ 297 │ │ path = self.ensure( │ │ 298 │ │ │ *subkeys, │ │ 299 │ │ │ url=url, │ │ 300 │ │ │ name=name, │ │ │ │ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │ │ │ autoclean = True │ │ │ │ download_kwargs = { │ │ │ │ │ 'progress_bar': True, │ │ │ │ │ 'backend': 'requests', │ │ │ │ │ 'tqdm_kwargs': {'leave': True} │ │ │ │ } │ │ │ │ force = False │ │ │ │ gunzipped_name = 'monarch-kg.db' │ │ │ │ gunzipped_path = WindowsPath('C:/Users/glass/.data/monarch/sql/monarch-kg.db') │ │ │ │ name = 'monarch-kg.db.gz' │ │ │ │ self = │ │ │ │ subkeys = ('sql',) │ │ │ │ url = 'https://data.monarchinitiative.org/monarch-kg-dev/latest/monarch-kg.db.g… │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ D:\dev\tislab\monarch-py\.venv\lib\site-packages\pystow\impl.py:171 in ensure │ │ │ │ 168 │ │ if name is None: │ │ 169 │ │ │ name = name_from_url(url) │ │ 170 │ │ path = self.join(*subkeys, name=name, ensure_exists=True) │ │ ❱ 171 │ │ utils.download( │ │ 172 │ │ │ url=url, │ │ 173 │ │ │ path=path, │ │ 174 │ │ │ force=force, │ │ │ params = None │ │ │ │ url = 'https://data.monarchinitiative.org/monarch-kg-dev/latest/monarch-kg.db.gz' │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ D:\dev\tislab\monarch-py\.venv\lib\site-packages\requests\api.py:59 in request │ │ │ │ 56 │ # avoid leaving sockets open which can trigger a ResourceWarning in some │ │ 57 │ # cases, and look like a memory leak in others. │ │ 58 │ with sessions.Session() as session: │ │ ❱ 59 │ │ return session.request(method=method, url=url, **kwargs) │ │ 60 │ │ 61 │ │ 62 def get(url, params=None, **kwargs): │ │ │ │ ╭─────────────────────────────────────── locals ────────────────────────────────────────╮ │ │ │ kwargs = { │ │ │ │ │ 'params': None, │ │ │ │ │ 'progress_bar': True, │ │ │ │ │ 'tqdm_kwargs': {'leave': True}, │ │ │ │ │ 'stream': True │ │ │ │ } │ │ │ │ method = 'get' │ │ │ │ session = │ │ │ │ url = 'https://data.monarchinitiative.org/monarch-kg-dev/latest/monarch-kg.db.gz' │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────╯ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ ```

[cthoyt]

@glass-ships you'll have to install your code from the branch where I implemented this, since it's neither merged nor released

[glass-ships]

@glass-ships you'll have to install your code from the branch where I implemented this, since it's neither merged nor released

I thought I had done exactly that by issuing the following:

$ poetry run pip uninstall pystow 
$ cd </pystow/path> 
$ poetry -C </my/library/path> run pip install . 

A pip freeze | grep 'pystow' returns pystow @ file:///D:/dev/pystow though, which I guess I expected to be the version num..

Apologies for my stumbling around, I'm still getting used to manually managing dependencies with Poetry.

[glass-ships]

Ok, sorry again, looks like I've sorted my issue - my system appeared to be caching an older version and claiming it was up to date!

I nuked my environment and started again, and everything appears kosher!

(monarch-py-py3.10) PS D:\dev\pystow> monarch sql entity --id MONDO:0012933
Downloading monarch-kg.db.gz: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 475M/475M [00:04<00:00, 110MB/s]

# Rest of output
}

[cthoyt]

FYI @glass-ships I have made a release 0.5.0 that includes this feature