Unidata / MetPy

MetPy is a collection of tools in Python for reading, visualizing and performing calculations with weather data.
https://unidata.github.io/MetPy/
BSD 3-Clause "New" or "Revised" License
1.25k stars 415 forks source link

`Level3File` import is a bit bloated #1901

Closed dlaw closed 3 years ago

dlaw commented 3 years ago

I am using the Level3File class to process some radar data on a server. I do not use any other features from MetPy. I have two issues with the import proess:

  1. Upon importing Level3File, I always receive an error Cannot import USCOUNTIES and USSTATES without Cartopy installed. I intentionally do not have cartopy installed, because I don't need it. Perhaps the cartopy import could be made lazy, so that you don't get the warning unless you try to use USCOUNTIES or USSTATES? As a workaround, I have resorted to doing this:

    import logging
    logging.disable(logging.ERROR)
    from metpy.io.nexrad import Level3File
    logging.disable(logging.NOTSET)
  2. The import process is very slow. I have a particular script which takes 1.3 seconds to execute, of which 1.2 seconds is spent importing Level3File and 0.1 seconds is spent opening a level 3 file and doing something with the data. It's not totally obvious to me where this delay originates, but I would guess we end up importing some other modules which are not actually needed. Perhaps a lazy evaluation of whatever slows this down would be the best solution?

    In [1]: time from metpy.io.nexrad import Level3File
    Cannot import USCOUNTIES and USSTATES without Cartopy installed.
    CPU times: user 1.14 s, sys: 89.6 ms, total: 1.23 s
    Wall time: 1.23 s
dopplershift commented 3 years ago

@dlaw Thanks for reporting this. So with us dropping support for Python 3.7, we can finally use a module-level __getattr__() to handle access to USCOUNTIES and USSTATES and do much better with that warning, and we should do so.

However, the fact that your single import is even triggering that led me to dig in--we're doing something wrong. Importing from metpy.io is pulling in metar.py (fine), which then ends up triggering an import from metpy.plots (this is no good). Fixing that eliminated the warning AND dropped the import time for my system from ~1.37s to ~1.17s. I also found it was importing from metpy.calc--hiding that a bit dropped the time for from metpy.io import Level3File to ~1.05s on my system. I'll put in a PR with some of that for 1.1.

You might also see some benefit with using the current main branch, which made the loading of station data information a lazy process.

Also, FYI, we don't recommend the use of from metpy.io.nexrad import Level3File, but instead use from metpy.io import Level3File. The module layout below e.g. metpy.io is considered an implementation detail and is subject to change without warning.