search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
10.71k stars 1.33k forks source link

Cannot use dates before 1969-12-31 12:00:00 (i.e. negative epochs < -43200) in partitions on Windows #22311

Closed lsim-aegeri closed 2 days ago

lsim-aegeri commented 1 month ago

Dagster version

1.7.4

What's the issue?

Attempting to use a date before 1969-12-31 12:00:00 (i.e. negative epochs < -43200) results in the following error: 

Operation name: PartitionHealthQuery

Message: [Errno 22] Invalid argument

Path: ["assetNodeOrError","assetPartitionStatuses"]

Locations: [{"line":11,"column":7}]

Stack Trace:
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\graphql\execution\execute.py", line 521, in execute_field
    result = resolve_fn(source, info, **args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster_graphql\schema\asset_graph.py", line 1139, in resolve_assetPartitionStatuses
    ) = get_partition_subsets(
        ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster_graphql\implementation\fetch_assets.py", line 429, in get_partition_subsets
    updated_cache_value = get_and_update_asset_status_cache_value(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_core\storage\partition_status_cache.py", line 422, in get_and_update_asset_status_cache_value
    updated_cache_value = _build_status_cache(
                          ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_core\storage\partition_status_cache.py", line 290, in _build_status_cache
    get_validated_partition_keys(
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_core\storage\partition_status_cache.py", line 218, in get_validated_partition_keys
    validated_partitions = {
                           ^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_core\storage\partition_status_cache.py", line 221, in <setcomp>
    if partitions_def.has_partition_key(pk, current_time=current_time)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_core\definitions\time_window_partitions.py", line 995, in has_partition_key
    partition_start_time = self.start_time_for_partition_key(partition_key)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_core\definitions\time_window_partitions.py", line 570, in start_time_for_partition_key
    return next(iter(self._iterate_time_windows(partition_key_dt))).start
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_core\definitions\time_window_partitions.py", line 923, in _iterate_time_windows
    prev_time = next(iterator)
                ^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_utils\schedules.py", line 821, in cron_string_iterator
    yield from _croniter_string_iterator(
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_utils\schedules.py", line 839, in _croniter_string_iterator
    next_date = next(reverse_cron)
                ^^^^^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\dagster\_utils\schedules.py", line 538, in _timezone_aware_cron_iter
    start_datetime = pendulum.from_timestamp(start_timestamp, tz=timezone_str)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\pendulum\__init__.py", line 295, in from_timestamp
    dt = _datetime.datetime.utcfromtimestamp(timestamp)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I'm pretty sure this comes up becuase datetime.datetime.utcfromtimestamp() will not accept large negative epochs on Windows machines. For example: 

import datetime
datetime.datetime.utcfromtimestamp(-43250)

returns

Traceback (most recent call last):
  File "C:\Users\lucien.simpfendoerfe\micromamba\envs\weather-dagster\Lib\site-packages\IPython\core\interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-31-e07f26fe4ab9>", line 1, in <module>
    datetime.datetime.utcfromtimestamp(-43250)
OSError: [Errno 22] Invalid argument

When I run an identical snippet on Linux, I get the expected result. 

>>> import datetime
>>> datetime.datetime.utcfromtimestamp(-43250)
datetime.datetime(1969, 12, 31, 11, 59, 10)

What did you expect to happen?

I expect to be able to create partitions that start or end before 1969-12-31 12:00:00.

How to reproduce?

I'm not sure it's helpful, but here's the contents of my anaconda environment. 

  Name                       Version             Build                    Channel
---------------------------------------------------------------------------------------
  _libavif_api               1.0.4               h57928b3_4               conda-forge
  adal                       1.2.7               pyhd8ed1ab_0             conda-forge
  adlfs                      2024.4.1            pyhd8ed1ab_0             conda-forge
  affine                     2.4.0               pyhd8ed1ab_0             conda-forge
  aiobotocore                2.13.0              pyhd8ed1ab_0             conda-forge
  aiohttp                    3.9.5               py311ha68e1ae_0          conda-forge
  aioitertools               0.11.0              pyhd8ed1ab_0             conda-forge
  aiosignal                  1.3.1               pyhd8ed1ab_0             conda-forge
  alembic                    1.13.1              pyhd8ed1ab_1             conda-forge
  aniso8601                  9.0.1               pyhd8ed1ab_0             conda-forge
  annotated-types            0.7.0               pyhd8ed1ab_0             conda-forge
  anyio                      4.3.0               pyhd8ed1ab_0             conda-forge
  aom                        3.9.0               he0c23c2_0               conda-forge
  asciitree                  0.3.3               py_2                     conda-forge
  asttokens                  2.4.1               pyhd8ed1ab_0             conda-forge
  attrs                      23.2.0              pyh71513ae_0             conda-forge
  aws-c-auth                 0.7.22              ha390a07_2               conda-forge
  aws-c-cal                  0.6.14              h750c3ff_1               conda-forge
  aws-c-common               0.9.19              h2466b09_0               conda-forge
  aws-c-compression          0.2.18              h750c3ff_6               conda-forge
  aws-c-event-stream         0.4.2               he05106e_12              conda-forge
  aws-c-http                 0.8.1               h55c4ea9_17              conda-forge
  aws-c-io                   0.14.8              ha255af7_5               conda-forge
  aws-c-mqtt                 0.10.4              heb8cf2b_4               conda-forge
  aws-c-s3                   0.5.9               haf5e0f7_3               conda-forge
  aws-c-sdkutils             0.1.16              h750c3ff_2               conda-forge
  aws-checksums              0.1.18              h750c3ff_6               conda-forge
  aws-crt-cpp                0.26.9              hb2f7e64_0               conda-forge
  aws-sdk-cpp                1.11.329            h752cf48_3               conda-forge
  azure-core                 1.30.1              pyhd8ed1ab_0             conda-forge
  azure-core-cpp             1.11.1              h249a519_1               conda-forge
  azure-datalake-store       0.0.51              pyh9f0ad1d_0             conda-forge
  azure-identity             1.16.0              pyhd8ed1ab_0             conda-forge
  azure-identity-cpp         1.6.0               h91493d7_1               conda-forge
  azure-storage-blob         12.20.0             pyhd8ed1ab_0             conda-forge
  azure-storage-blobs-cpp    12.10.0             h91493d7_1               conda-forge
  azure-storage-common-cpp   12.5.0              h91493d7_4               conda-forge
  backoff                    2.2.1               pyhd8ed1ab_0             conda-forge
  blinker                    1.8.2               pyhd8ed1ab_0             conda-forge
  blosc                      1.21.5              hbd69f2e_1               conda-forge
  bokeh                      3.4.1               pyhd8ed1ab_0             conda-forge
  boto3                      1.34.106            pyhd8ed1ab_0             conda-forge
  botocore                   1.34.106            pyge310_1234567_0        conda-forge
  bottleneck                 1.3.8               py311h59ca53f_0          conda-forge
  brotli                     1.1.0               hcfcfb64_1               conda-forge
  brotli-bin                 1.1.0               hcfcfb64_1               conda-forge
  brotli-python              1.1.0               py311h12c1d0e_1          conda-forge
  bzip2                      1.0.8               hcfcfb64_5               conda-forge
  c-ares                     1.28.1              hcfcfb64_0               conda-forge
  c-blosc2                   2.14.4              h183a6f4_1               conda-forge
  ca-certificates            2024.6.2            h56e8100_0               conda-forge
  cached-property            1.5.2               hd8ed1ab_1               conda-forge
  cached_property            1.5.2               pyha770c72_1             conda-forge
  cachetools                 5.3.3               pyhd8ed1ab_0             conda-forge
  cads-api-client            1.0.3               pyhd8ed1ab_0             conda-forge
  cairo                      1.18.0              h1fef639_0               conda-forge
  cartopy                    0.23.0              py311hcf9f919_1          conda-forge
  cdsapi                     0.7.0               pyhd8ed1ab_0             conda-forge
  certifi                    2024.2.2            pyhd8ed1ab_0             conda-forge
  cffi                       1.16.0              py311ha68e1ae_0          conda-forge
  cfgrib                     0.9.12.0            pyhd8ed1ab_0             conda-forge
  cfitsio                    4.4.0               h9b0cee5_1               conda-forge
  cftime                     1.6.3               py311h0a17f05_1          conda-forge
  charls                     2.4.2               h1537add_0               conda-forge
  charset-normalizer         3.3.2               pyhd8ed1ab_0             conda-forge
  click                      8.1.7               win_pyh7428d3b_0         conda-forge
  click-plugins              1.1.1               py_0                     conda-forge
  cligj                      0.7.2               pyhd8ed1ab_1             conda-forge
  cloudpickle                3.0.0               pyhd8ed1ab_0             conda-forge
  colorama                   0.4.6               pyhd8ed1ab_0             conda-forge
  coloredlogs                14.0                pyhd8ed1ab_3             conda-forge
  comm                       0.2.2               pyhd8ed1ab_0             conda-forge
  contourpy                  1.2.1               py311h005e61a_0          conda-forge
  cramjam                    2.8.3               py311h9a9e57f_0          conda-forge
  croniter                   2.0.5               pyhd8ed1ab_0             conda-forge
  cryptography               42.0.7              py311hfd75b31_0          conda-forge
  curl                       8.8.0               h0dd56e1_0               conda-forge
  cycler                     0.12.1              pyhd8ed1ab_0             conda-forge
  cytoolz                    0.12.3              py311ha68e1ae_0          conda-forge
  dagster                    1.7.4               pyhd8ed1ab_11            conda-forge
  dagster-graphql            1.7.4               pyhd8ed1ab_0             conda-forge
  dagster-pipes              1.7.4               pyhd8ed1ab_0             conda-forge
  dagster-webserver          1.7.4               pyhd8ed1ab_0             conda-forge
  dask                       2024.5.2            pyhd8ed1ab_0             conda-forge
  dask-core                  2024.5.2            pyhd8ed1ab_0             conda-forge
  dask-expr                  1.1.2               pyhd8ed1ab_0             conda-forge
  dav1d                      1.2.1               hcfcfb64_0               conda-forge
  debugpy                    1.8.1               py311h12c1d0e_0          conda-forge
  decorator                  5.1.1               pyhd8ed1ab_0             conda-forge
  distributed                2024.5.2            pyhd8ed1ab_0             conda-forge
  docstring_parser           0.16                pyhd8ed1ab_0             conda-forge
  eccodes                    2.35.0              h97caf96_0               conda-forge
  exceptiongroup             1.2.0               pyhd8ed1ab_2             conda-forge
  executing                  2.0.1               pyhd8ed1ab_0             conda-forge
  expat                      2.6.2               h63175ca_0               conda-forge
  fasteners                  0.17.3              pyhd8ed1ab_0             conda-forge
  fastparquet                2024.5.0            py311h0a17f05_0          conda-forge
  filelock                   3.14.0              pyhd8ed1ab_0             conda-forge
  findlibs                   0.0.5               pyhd8ed1ab_0             conda-forge
  fmt                        10.2.1              h181d51b_0               conda-forge
  font-ttf-dejavu-sans-mono  2.37                hab24e00_0               conda-forge
  font-ttf-inconsolata       3.000               h77eed37_0               conda-forge
  font-ttf-source-code-pro   2.038               h77eed37_0               conda-forge
  font-ttf-ubuntu            0.83                h77eed37_2               conda-forge
  fontconfig                 2.14.2              hbde0cde_0               conda-forge
  fonts-conda-ecosystem      1                   0                        conda-forge
  fonts-conda-forge          1                   0                        conda-forge
  fonttools                  4.53.0              py311he736701_0          conda-forge
  freeglut                   3.2.2               h63175ca_2               conda-forge
  freetype                   2.12.1              hdaf720e_2               conda-forge
  freexl                     2.0.0               h8276f4a_0               conda-forge
  frozenlist                 1.4.1               py311ha68e1ae_0          conda-forge
  fsspec                     2024.6.0            pyhff2d567_0             conda-forge
  gcsfs                      2024.6.0            pyhd8ed1ab_0             conda-forge
  geojson                    3.1.0               pyhd8ed1ab_0             conda-forge
  geos                       3.12.1              h1537add_0               conda-forge
  geotiff                    1.7.3               hfbd0779_0               conda-forge
  giflib                     5.2.2               h64bf75a_0               conda-forge
  glib                       2.80.2              h0df6a38_0               conda-forge
  glib-tools                 2.80.2              h2f9d560_0               conda-forge
  google-api-core            2.19.0              pyhd8ed1ab_0             conda-forge
  google-auth                2.29.0              pyhca7485f_0             conda-forge
  google-auth-oauthlib       1.2.0               pyhd8ed1ab_0             conda-forge
  google-cloud-core          2.4.1               pyhd8ed1ab_0             conda-forge
  google-cloud-storage       2.16.0              pyhca7485f_0             conda-forge
  google-crc32c              1.1.2               py311h48d9ab2_5          conda-forge
  google-resumable-media     2.7.0               pyhd8ed1ab_0             conda-forge
  googleapis-common-protos   1.63.1              pyhd8ed1ab_0             conda-forge
  gql                        3.5.0               pyhd8ed1ab_1             conda-forge
  gql-with-requests          3.5.0               pyhd8ed1ab_1             conda-forge
  graphene                   3.3                 pyhd8ed1ab_0             conda-forge
  graphql-core               3.2.3               pyhd8ed1ab_0             conda-forge
  graphql-relay              3.2.0               pyhd8ed1ab_0             conda-forge
  greenlet                   3.0.3               py311h12c1d0e_0          conda-forge
  grpcio                     1.62.2              py311h5bc0dda_0          conda-forge
  grpcio-health-checking     1.62.2              pyhd8ed1ab_0             conda-forge
  gst-plugins-base           1.24.4              hba88be7_0               conda-forge
  gstreamer                  1.24.4              h5006eae_0               conda-forge
  h11                        0.14.0              pyhd8ed1ab_0             conda-forge
  h3-py                      3.7.7               py311h12c1d0e_0          conda-forge
  h5netcdf                   1.3.0               pyhd8ed1ab_0             conda-forge
  h5py                       3.11.0              nompi_py311h67016bb_101  conda-forge
  hdf4                       4.2.15              h5557f11_7               conda-forge
  hdf5                       1.14.3              nompi_h2b43c12_104       conda-forge
  herbie-data                2024.5.0            pyhd8ed1ab_0             conda-forge
  httptools                  0.6.1               py311ha68e1ae_0          conda-forge
  humanfriendly              10.0                py311h1ea47a8_5          conda-forge
  icu                        73.2                h63175ca_0               conda-forge
  idna                       3.7                 pyhd8ed1ab_0             conda-forge
  imagecodecs                2024.6.1            py311ha49d32f_0          conda-forge
  imageio                    2.34.1              pyh4b66e23_0             conda-forge
  importlib-metadata         7.1.0               pyha770c72_0             conda-forge
  importlib_metadata         7.1.0               hd8ed1ab_0               conda-forge
  importlib_resources        6.4.0               pyhd8ed1ab_0             conda-forge
  intel-openmp               2024.1.0            h57928b3_966             conda-forge
  ipykernel                  6.29.3              pyha63f2e9_0             conda-forge
  ipython                    8.25.0              pyh7428d3b_0             conda-forge
  isodate                    0.6.1               pyhd8ed1ab_0             conda-forge
  jasper                     4.2.4               hcb1a123_0               conda-forge
  jedi                       0.19.1              pyhd8ed1ab_0             conda-forge
  jinja2                     3.1.4               pyhd8ed1ab_0             conda-forge
  jmespath                   1.0.1               pyhd8ed1ab_0             conda-forge
  joblib                     1.4.2               pyhd8ed1ab_0             conda-forge
  jupyter_client             8.6.2               pyhd8ed1ab_0             conda-forge
  jupyter_core               5.7.2               py311h1ea47a8_0          conda-forge
  jxrlib                     1.1                 hcfcfb64_3               conda-forge
  kealib                     1.5.3               h6c43f9b_1               conda-forge
  kerchunk                   0.2.5               pyhd8ed1ab_0             conda-forge
  kiwisolver                 1.4.5               py311h005e61a_1          conda-forge
  krb5                       1.21.2              heb0366b_0               conda-forge
  lazy_loader                0.4                 pyhd8ed1ab_0             conda-forge
  lcms2                      2.16                h67d730c_0               conda-forge
  lerc                       4.0.0               h63175ca_0               conda-forge
  libabseil                  20240116.2          cxx17_h63175ca_0         conda-forge
  libaec                     1.1.3               h63175ca_0               conda-forge
  libarchive                 3.7.4               haf234dc_0               conda-forge
  libarrow                   16.1.0              h35ad6a0_6_cpu           conda-forge
  libarrow-acero             16.1.0              he0c23c2_6_cpu           conda-forge
  libarrow-dataset           16.1.0              he0c23c2_6_cpu           conda-forge
  libarrow-substrait         16.1.0              h1f0e801_6_cpu           conda-forge
  libavif16                  1.0.4               hac688ae_4               conda-forge
  libblas                    3.9.0               22_win64_mkl             conda-forge
  libboost-headers           1.85.0              h57928b3_1               conda-forge
  libbrotlicommon            1.1.0               hcfcfb64_1               conda-forge
  libbrotlidec               1.1.0               hcfcfb64_1               conda-forge
  libbrotlienc               1.1.0               hcfcfb64_1               conda-forge
  libcblas                   3.9.0               22_win64_mkl             conda-forge
  libclang13                 18.1.6              default_hf64faad_0       conda-forge
  libcrc32c                  1.1.2               h0e60522_0               conda-forge
  libcurl                    8.8.0               hd5e4a3a_0               conda-forge
  libdeflate                 1.20                hcfcfb64_0               conda-forge
  libevent                   2.1.12              h3671451_1               conda-forge
  libexpat                   2.6.2               h63175ca_0               conda-forge
  libffi                     3.4.2               h8ffe710_5               conda-forge
  libgdal                    3.8.5               h4f813f3_7               conda-forge
  libglib                    2.80.2              h0df6a38_0               conda-forge
  libgoogle-cloud            2.24.0              h5e7cea3_0               conda-forge
  libgoogle-cloud-storage    2.24.0              hce61461_0               conda-forge
  libgrpc                    1.62.2              h5273850_0               conda-forge
  libhwloc                   2.10.0              default_h8125262_1001    conda-forge
  libiconv                   1.17                hcfcfb64_2               conda-forge
  libintl                    0.22.5              h5728263_2               conda-forge
  libintl-devel              0.22.5              h5728263_2               conda-forge
  libjpeg-turbo              3.0.0               hcfcfb64_1               conda-forge
  libkml                     1.3.0               haf3e7a6_1018            conda-forge
  liblapack                  3.9.0               22_win64_mkl             conda-forge
  libnetcdf                  4.9.2               nompi_h07c049d_113       conda-forge
  libogg                     1.3.4               h8ffe710_1               conda-forge
  libparquet                 16.1.0              h178134c_6_cpu           conda-forge
  libpng                     1.6.43              h19919ed_0               conda-forge
  libpq                      16.3                hab9416b_0               conda-forge
  libprotobuf                4.25.3              h503648d_0               conda-forge
  libre2-11                  2023.09.01          hf8d8778_2               conda-forge
  librttopo                  1.1.0               h94c4f80_15              conda-forge
  libsodium                  1.0.18              h8d14728_1               conda-forge
  libspatialite              5.1.0               h1b08085_6               conda-forge
  libsqlite                  3.45.3              hcfcfb64_0               conda-forge
  libssh2                    1.11.0              h7dfc565_0               conda-forge
  libthrift                  0.19.0              ha2b3283_1               conda-forge
  libtiff                    4.6.0               hddb2be6_3               conda-forge
  libutf8proc                2.8.0               h82a8f57_0               conda-forge
  libvorbis                  1.3.7               h0e60522_0               conda-forge
  libwebp-base               1.4.0               hcfcfb64_0               conda-forge
  libxcb                     1.15                hcd874cb_0               conda-forge
  libxml2                    2.12.7              h73268cd_0               conda-forge
  libzip                     1.10.1              h1d365fa_3               conda-forge
  libzlib                    1.3.1               h2466b09_1               conda-forge
  libzopfli                  1.0.3               h0e60522_0               conda-forge
  llvmlite                   0.41.1              py311h5bc0dda_0          conda-forge
  locket                     1.0.0               pyhd8ed1ab_0             conda-forge
  looseversion               1.3.0               pyhd8ed1ab_0             conda-forge
  lz4                        4.3.3               py311haddf500_0          conda-forge
  lz4-c                      1.9.4               hcfcfb64_0               conda-forge
  lzo                        2.10                hcfcfb64_1001            conda-forge
  m2w64-gcc-libgfortran      5.3.0               6                        conda-forge
  m2w64-gcc-libs             5.3.0               7                        conda-forge
  m2w64-gcc-libs-core        5.3.0               7                        conda-forge
  m2w64-gmp                  6.1.0               2                        conda-forge
  m2w64-libwinpthread-git    5.0.0.4634.697f757  2                        conda-forge
  mako                       1.3.5               pyhd8ed1ab_0             conda-forge
  markdown-it-py             3.0.0               pyhd8ed1ab_0             conda-forge
  markupsafe                 2.1.5               py311ha68e1ae_0          conda-forge
  matplotlib                 3.8.4               py311h1ea47a8_2          conda-forge
  matplotlib-base            3.8.4               py311h9b31f6e_2          conda-forge
  matplotlib-inline          0.1.7               pyhd8ed1ab_0             conda-forge
  mdurl                      0.1.2               pyhd8ed1ab_0             conda-forge
  metpy                      1.6.2               pyhd8ed1ab_0             conda-forge
  minizip                    4.0.6               hb638d1e_0               conda-forge
  mkl                        2024.1.0            h66d3029_692             conda-forge
  msal                       1.28.0              pyhd8ed1ab_0             conda-forge
  msal_extensions            1.1.0               py311h1ea47a8_1          conda-forge
  msgpack-python             1.0.8               py311h3257749_0          conda-forge
  msys2-conda-epoch          20160418            1                        conda-forge
  multidict                  6.0.5               py311ha68e1ae_0          conda-forge
  multiurl                   0.3.1               pyhd8ed1ab_0             conda-forge
  munkres                    1.1.4               pyh9f0ad1d_0             conda-forge
  nest-asyncio               1.6.0               pyhd8ed1ab_0             conda-forge
  netcdf4                    1.6.5               nompi_py311hb75404a_101  conda-forge
  networkx                   3.3                 pyhd8ed1ab_1             conda-forge
  numba                      0.58.1              py311h2c0921f_0          conda-forge
  numcodecs                  0.12.1              py311hda3d55a_1          conda-forge
  numpy                      1.26.4              py311h0b4df5a_0          conda-forge
  oauthlib                   3.2.2               pyhd8ed1ab_0             conda-forge
  openjpeg                   2.5.2               h3d672ee_0               conda-forge
  openssl                    3.3.0               h2466b09_3               conda-forge
  orc                        2.0.1               h7e885a9_1               conda-forge
  packaging                  24.0                pyhd8ed1ab_0             conda-forge
  pandas                     2.2.2               py311hcf9f919_1          conda-forge
  parso                      0.8.4               pyhd8ed1ab_0             conda-forge
  partd                      1.4.2               pyhd8ed1ab_0             conda-forge
  pcre2                      10.43               h17e33f8_0               conda-forge
  pendulum                   3.0.0               py311h633b200_0          conda-forge
  pickleshare                0.7.5               py_1003                  conda-forge
  pillow                     10.3.0              py311h6819b35_0          conda-forge
  pint                       0.23                pyhd8ed1ab_1             conda-forge
  pip                        24.0                pyhd8ed1ab_0             conda-forge
  pixman                     0.43.4              h63175ca_0               conda-forge
  platformdirs               4.2.2               pyhd8ed1ab_0             conda-forge
  ply                        3.11                pyhd8ed1ab_2             conda-forge
  pooch                      1.8.1               pyhd8ed1ab_0             conda-forge
  poppler                    24.04.0             h747fd5a_0               conda-forge
  poppler-data               0.4.12              hd8ed1ab_0               conda-forge
  portalocker                2.8.2               py311h1ea47a8_1          conda-forge
  postgresql                 16.3                h7f155c9_0               conda-forge
  proj                       9.4.0               he13c7e8_1               conda-forge
  prompt-toolkit             3.0.42              pyha770c72_0             conda-forge
  proto-plus                 1.23.0              pyhd8ed1ab_0             conda-forge
  protobuf                   4.25.3              py311hb041b4a_0          conda-forge
  psutil                     5.9.8               py311ha68e1ae_0          conda-forge
  pthread-stubs              0.4                 hcd874cb_1001            conda-forge
  pthreads-win32             2.9.1               hfa6e2cd_3               conda-forge
  pure_eval                  0.2.2               pyhd8ed1ab_0             conda-forge
  pyarrow                    16.1.0              py311h3810d55_1          conda-forge
  pyarrow-core               16.1.0              py311h65ac0bd_1_cpu      conda-forge
  pyarrow-hotfix             0.6                 pyhd8ed1ab_0             conda-forge
  pyasn1                     0.6.0               pyhd8ed1ab_0             conda-forge
  pyasn1-modules             0.4.0               pyhd8ed1ab_0             conda-forge
  pycparser                  2.22                pyhd8ed1ab_0             conda-forge
  pydantic                   2.7.3               pyhd8ed1ab_0             conda-forge
  pydantic-core              2.18.4              py311h533ab2d_0          conda-forge
  pygments                   2.18.0              pyhd8ed1ab_0             conda-forge
  pygrib                     2.1.5               py311h23b68ad_2          conda-forge
  pyjwt                      2.8.0               pyhd8ed1ab_1             conda-forge
  pyopenssl                  24.0.0              pyhd8ed1ab_0             conda-forge
  pyparsing                  3.1.2               pyhd8ed1ab_0             conda-forge
  pyproj                     3.6.1               py311hc1951a7_6          conda-forge
  pyqt                       5.15.9              py311h125bc19_5          conda-forge
  pyqt5-sip                  12.12.2             py311h12c1d0e_5          conda-forge
  pyreadline3                3.4.1               py311h1ea47a8_3          conda-forge
  pysheds                    0.4                 py311h814a670_1          conda-forge
  pyshp                      2.3.1               pyhd8ed1ab_0             conda-forge
  pysocks                    1.7.1               pyh0701188_6             conda-forge
  python                     3.11.9              h631f459_0_cpython       conda-forge
  python-dateutil            2.9.0               pyhd8ed1ab_0             conda-forge
  python-dotenv              1.0.1               pyhd8ed1ab_0             conda-forge
  python-eccodes             1.7.0               py311h0a17f05_2          conda-forge
  python-tzdata              2024.1              pyhd8ed1ab_0             conda-forge
  python_abi                 3.11                4_cp311                  conda-forge
  pytz                       2024.1              pyhd8ed1ab_0             conda-forge
  pyu2f                      0.1.5               pyhd8ed1ab_0             conda-forge
  pywavelets                 1.4.1               py311h59ca53f_1          conda-forge
  pywin32                    306                 py311h12c1d0e_2          conda-forge
  pywin32-on-windows         0.1.0               pyh07e9846_2             conda-forge
  pyyaml                     6.0.1               py311ha68e1ae_1          conda-forge
  pyzmq                      26.0.3              py311h484c95c_0          conda-forge
  qt-main                    5.15.8              hcef0176_21              conda-forge
  rasterio                   1.3.10              py311h41753ef_1          conda-forge
  rav1e                      0.6.6               h975169c_2               conda-forge
  re2                        2023.09.01          hd3b24a8_2               conda-forge
  requests                   2.32.3              pyhd8ed1ab_0             conda-forge
  requests-oauthlib          2.0.0               pyhd8ed1ab_0             conda-forge
  requests-toolbelt          1.0.0               pyhd8ed1ab_0             conda-forge
  rich                       13.7.1              pyhd8ed1ab_0             conda-forge
  rioxarray                  0.15.5              pyhd8ed1ab_0             conda-forge
  rsa                        4.9                 pyhd8ed1ab_0             conda-forge
  s3fs                       2024.6.0            pyhd8ed1ab_0             conda-forge
  s3transfer                 0.10.1              pyhd8ed1ab_0             conda-forge
  scikit-image               0.23.2              py311hbc92ba2_0          conda-forge
  scikit-learn               1.5.0               py311hdcb8d17_1          conda-forge
  scipy                      1.13.1              py311hd4686c6_0          conda-forge
  setuptools                 70.0.0              pyhd8ed1ab_0             conda-forge
  shapely                    2.0.4               py311hcd532c9_1          conda-forge
  sip                        6.7.12              py311h12c1d0e_0          conda-forge
  six                        1.16.0              pyh6c4a22f_0             conda-forge
  snappy                     1.2.0               hfb803bf_1               conda-forge
  sniffio                    1.3.1               pyhd8ed1ab_0             conda-forge
  snuggs                     1.4.7               py_0                     conda-forge
  sortedcontainers           2.4.0               pyhd8ed1ab_0             conda-forge
  spdlog                     1.13.0              h64d2f7d_0               conda-forge
  sqlalchemy                 2.0.30              py311he736701_0          conda-forge
  sqlite                     3.45.3              hcfcfb64_0               conda-forge
  stack_data                 0.6.2               pyhd8ed1ab_0             conda-forge
  starlette                  0.37.2              pyhd8ed1ab_0             conda-forge
  structlog                  24.2.0              pyhd8ed1ab_0             conda-forge
  svt-av1                    2.1.0               he0c23c2_0               conda-forge
  tabulate                   0.9.0               pyhd8ed1ab_1             conda-forge
  tbb                        2021.12.0           hc790b64_1               conda-forge
  tblib                      3.0.0               pyhd8ed1ab_0             conda-forge
  threadpoolctl              3.5.0               pyhc1e730c_0             conda-forge
  tifffile                   2024.5.22           pyhd8ed1ab_0             conda-forge
  tiledb                     2.23.0              h98cc7b1_2               conda-forge
  time-machine               2.14.1              py311ha68e1ae_0          conda-forge
  timezonefinder             6.4.1               py311ha68e1ae_0          conda-forge
  tk                         8.6.13              h5226925_1               conda-forge
  toml                       0.10.2              pyhd8ed1ab_0             conda-forge
  tomli                      2.0.1               pyhd8ed1ab_0             conda-forge
  toolz                      0.12.1              pyhd8ed1ab_0             conda-forge
  toposort                   1.10                pyhd8ed1ab_0             conda-forge
  tornado                    6.4                 py311ha68e1ae_0          conda-forge
  tqdm                       4.66.4              pyhd8ed1ab_0             conda-forge
  traitlets                  5.14.3              pyhd8ed1ab_0             conda-forge
  typing-extensions          4.12.1              hd8ed1ab_0               conda-forge
  typing_extensions          4.12.1              pyha770c72_0             conda-forge
  tzdata                     2024a               h0c530f3_0               conda-forge
  ucrt                       10.0.22621.0        h57928b3_0               conda-forge
  ujson                      5.10.0              py311hda3d55a_0          conda-forge
  universal_pathlib          0.2.2               pyhd8ed1ab_0             conda-forge
  uriparser                  0.9.8               h5a68840_0               conda-forge
  urllib3                    2.2.1               pyhd8ed1ab_0             conda-forge
  uvicorn                    0.30.1              py311h1ea47a8_0          conda-forge
  uvicorn-standard           0.30.1              h1ea47a8_0               conda-forge
  vc                         14.3                ha32ba9b_20              conda-forge
  vc14_runtime               14.38.33135         h835141b_20              conda-forge
  vs2015_runtime             14.38.33135         h22015db_20              conda-forge
  watchdog                   4.0.1               py311h1ea47a8_0          conda-forge
  watchfiles                 0.22.0              py311h533ab2d_0          conda-forge
  wcwidth                    0.2.13              pyhd8ed1ab_0             conda-forge
  websockets                 11.0.3              py311ha68e1ae_1          conda-forge
  wheel                      0.43.0              pyhd8ed1ab_1             conda-forge
  win_inet_pton              1.1.0               pyhd8ed1ab_6             conda-forge
  wrapt                      1.16.0              py311ha68e1ae_0          conda-forge
  xarray                     2024.5.0            pyhd8ed1ab_0             conda-forge
  xerces-c                   3.2.5               h63175ca_0               conda-forge
  xorg-libxau                1.0.11              hcd874cb_0               conda-forge
  xorg-libxdmcp              1.1.3               hcd874cb_0               conda-forge
  xyzservices                2024.4.0            pyhd8ed1ab_0             conda-forge
  xz                         5.2.6               h8d14728_0               conda-forge
  yaml                       0.2.5               h8ffe710_2               conda-forge
  yarl                       1.9.4               py311ha68e1ae_0          conda-forge
  zarr                       2.18.2              pyhd8ed1ab_0             conda-forge
  zeromq                     4.3.5               he1f189c_4               conda-forge
  zfp                        1.0.1               h63175ca_0               conda-forge
  zict                       3.0.0               pyhd8ed1ab_0             conda-forge
  zipp                       3.17.0              pyhd8ed1ab_0             conda-forge
  zlib                       1.3.1               h2466b09_1               conda-forge
  zlib-ng                    2.0.7               hcfcfb64_0               conda-forge
  zstd                       1.5.6               h0ea2cb4_0               conda-forge

The particular partitions that are failing are: 

monthly_partition = MonthlyPartitionsDefinition(
    start_date=datetime.datetime(1940, 1, 1),
    end_offset=1,
)

yearly_partition = TimeWindowPartitionsDefinition(
    cron_schedule="0 0 1 1 *",
    fmt="%Y-%m-%d",
    start=datetime.datetime(1940, 1, 1),
    end_offset=1,
)

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

jamiedemaria commented 3 weeks ago

cc @smackesey if you've run across this before or have any ideas? @gibsondan you've also been looking at pendulum stuff recently right?

From what I can find through searching around, pendulum claims to have fixed this in https://github.com/sdispater/pendulum/issues/53, but it was part of the 0.6.2 release in 2016, so you should definitely have that code.

Other libraries that ran into the same issue, seem to write custom code to special case negative time on windows:

This PR from py-arrow uses timedelta if the date is negative and the OS is windows

Another example from faker https://github.com/joke2k/faker/pull/1436

smackesey commented 3 weeks ago

IIUC @gibsondan has a PR that removes pendulum entirely so hopefully that will make this issue moot

gibsondan commented 3 weeks ago

We are going to remove pendulum, but we will still be using UNIX timestamps after that, so I'm not positive that will resolve this.

gibsondan commented 3 weeks ago

It's possible that the workaround suggested here could resolve this, but I have not tried it: https://stackoverflow.com/questions/37494983/python-fromtimestamp-oserror/41400321#41400321

jamiedemaria commented 3 weeks ago

yeah, after some internal discussion, this is something we aren't going to try to fix. We assume we can use unix timestamps all over our code base, so trying to work around this issue is infeasible at the time. Closing this issue