pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.07k stars 295 forks source link

Support timezone-aware datetime objects in `find_files_and_readers` #1166

Open gerritholl opened 4 years ago

gerritholl commented 4 years ago

Feature Request

Is your feature request related to a problem? Please describe.

If my datetime is timezone-aware, Satpy can't use it:

satpy.readers.find_files_and_readers(
    datetime.datetime(2017, 3, 14, 4, 10, tzinfo=datetime.timezone.utc),
    datetime.datetime(2017, 3, 14, 4, 10, tzinfo=datetime.timezone.utc),
    "/tmp/manyabi",
    "abi_l1b")

Results in:

Traceback (most recent call last):
  File "mwe36.py", line 3, in <module>
    satpy.readers.find_files_and_readers(
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/__init__.py", line 652, in find_files_and_readers
    loadables = list(
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 505, in filter_selected_filenames
    for fn, _ in filename_iter:
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 486, in filter_filenames_by_info
    if self.metadata_matches(filename_info):
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 448, in metadata_matches
    if not self.time_matches(
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 439, in time_matches
    if start_time and fend and fend < start_time:
TypeError: can't compare offset-naive and offset-aware datetimes

Describe the solution you'd like

In a satellite context, perhaps all times that don't have timezone info could be assumed UTC. This may be risky if users use times in non-UTC timezones, although with timezone-naive times the risk of mixups already exists.

Describe any changes to existing user workflow

I think none.

Additional context

I got my timezone-aware datetime because I started out with a pandas datetime constructed from the string 2020-01-01T00:00:00Z. As a workaround it is easy to strip away the timezone info.

djhoese commented 4 years ago

There are two possible solutions to this, right?

  1. The user strips away the tz info.
  2. Satpy adds UTC timezones to all of its datetime objects.

I am not against the idea of adding UTC time zones in Satpy, at least in principle. The main complication I see is that we often have datetimes created from filenames. These datetimes are given to satpy from trollsift which I don't think has any knowledge of timezones (uses datetime.strptime). Additionally, many data filenames don't have a timezone included and are expected to be UTC.

My main question for whether or not this should be added is: does it provide any functionality/utility to Satpy users. I think the answer is no. It theoretically slows down processing (adding UTC time zone, every datetime comparison now has to check the timezone, etc). Additionally, other pytroll packages will need to support time zones (ex. pyorbital). This seems like a lot of work for solution 2 when the work for solution 1 is one line of code (ok 2 for start and end) and is a single up-front cost.

I'm not sold on it, but like I said not completely against it in the sense that it is more information, so Satpy provides a more complete set of metadata.

gerritholl commented 4 years ago

I agree that this is low priority. I just wanted to put it here in case anyone else runs into it.

Note that if any data filenames do have timezone included then strptime can result in a timezone-aware datetime:

datetime.datetime.strptime("20200101000000Z", "%Y%m%d%H%M%S%z")

gives

datetime.datetime(2020, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)

but I'm not sure if any satellite products that Satpy supports have timezone information in filenames.

mraspaud commented 4 years ago

How about just converting to naive while issuing a warni g? That keeps the existing functionality and implies only a small change to the code.