orion-junkins / river-level-forecasting

Exploring various neural network architectures for river level forecasting
MIT License
3 stars 3 forks source link

Broken AWSWeatherProvider Test #152

Closed douglasdennis closed 1 year ago

douglasdennis commented 1 year ago

Looks like #138 broke the test_fetch_historical_with_consistent_columns test for the AWSWeatherProvider.

================================================================================== FAILURES ==================================================================================
_______________________________________________________________ test_fetch_historical_with_consistent_columns ________________________________________________________________

weather_provider = <rlf.forecasting.data_fetching_utilities.weather_provider.aws_weather_provider.AWSWeatherProvider object at 0x7eff80d33460>

    @pytest.mark.aws
    @pytest.mark.slow
    def test_fetch_historical_with_consistent_columns(weather_provider):
        expected_columns = [
            'soil_moisture_level_1',
            'soil_moisture_level_2',
            'soil_moisture_level_3',
            'soil_moisture_level_4',
            'soil_temperature_level_1',
            'soil_temperature_level_2',
            'soil_temperature_level_3',
            'soil_temperature_level_4',
        ]

        requested_columns = [
            'soil_moisture_level_1',
            'soil_moisture_level_2',
            'soil_moisture_level_3',
            'soil_moisture_level_4',
            'soil_temperature_level_1',
            'soil_temperature_level_2',
            'soil_temperature_level_3',
            'soil_temperature_level_4',
        ]

        weather_provider.set_timestamp(CURRENT_TESTING_TIMESTAMP)
>       weather_datums = weather_provider.fetch_historical(columns=requested_columns)

tests/forecasting/data_fetching_utilities/weather_provider/test_aws_weather_provider.py:178:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/rlf/forecasting/data_fetching_utilities/weather_provider/aws_weather_provider.py:80: in fetch_historical
    datums = self.download_datums_from_aws(dir_path="historical", columns=columns)
src/rlf/forecasting/data_fetching_utilities/weather_provider/aws_weather_provider.py:53: in download_datums_from_aws
    datum = self.aws_dispatcher.download_datum(
src/rlf/aws_dispatcher.py:144: in download_datum
    hourly_parameters = self.download_df_from_parquet(folder_name, "data", columns=columns)
src/rlf/aws_dispatcher.py:102: in download_df_from_parquet
    table = dataset.read(columns=columns)
venv/lib/python3.10/site-packages/pyarrow/parquet/core.py:2601: in read
    table = self._dataset.to_table(
pyarrow/_dataset.pyx:369: in pyarrow._dataset.Dataset.to_table
    ???
pyarrow/_dataset.pyx:336: in pyarrow._dataset.Dataset.scanner
    ???
pyarrow/_dataset.pyx:2576: in pyarrow._dataset.Scanner.from_dataset
    ???
pyarrow/_dataset.pyx:2484: in pyarrow._dataset.Scanner._make_scan_options
    ???
pyarrow/_dataset.pyx:2385: in pyarrow._dataset._populate_builder
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   pyarrow.lib.ArrowInvalid: No match for FieldRef.Name(soil_moisture_0_to_7cm) in temperature_2m: double
E   relativehumidity_2m: int64
E   dewpoint_2m: double
E   apparent_temperature: double
E   pressure_msl: double
E   surface_pressure: double
E   precipitation: double
E   rain: double
E   snowfall: double
E   cloudcover: int64
E   cloudcover_low: int64
E   cloudcover_mid: int64
E   cloudcover_high: int64
E   shortwave_radiation: double
E   direct_radiation: double
E   diffuse_radiation: double
E   direct_normal_irradiance: double
E   windspeed_10m: double
E   windspeed_100m: double
E   winddirection_10m: int64
E   winddirection_100m: int64
E   windgusts_10m: double
E   et0_fao_evapotranspiration: double
E   vapor_pressure_deficit: double
E   soil_temperature_level_1: double
E   soil_temperature_level_2: double
E   soil_temperature_level_3: double
E   soil_temperature_level_4: double
E   soil_moisture_level_1: double
E   soil_moisture_level_2: double
E   soil_moisture_level_3: double
E   soil_moisture_level_4: double
E   time: timestamp[us, tz=UTC]
E   __fragment_index: int32
E   __batch_index: int32
E   __last_in_fragment: bool
E   __filename: string

pyarrow/error.pxi:100: ArrowInvalid
========================================================================== short test summary info ===========================================================================
FAILED tests/forecasting/data_fetching_utilities/weather_provider/test_aws_weather_provider.py::test_fetch_historical_with_consistent_columns - pyarrow.lib.ArrowInvalid: No match for FieldRef.Name(soil_moisture_0_to_7cm) in temperature_2m: double