Update data through 3/31/24

[ehmatthes]

In preparation for some upcoming presentations, update the analysis through the end of last month (3/31/24). See #11 for process.

• [x] Run current tests, and address any issues.
• [x] master -> main
• [x] Update all libraries?
• [x] Download updated data.
• [x] Append to current data set.
• [x] Update README in ir_data_clean/.
• [x] Check consistency of format.
• [x] Merge.
• [ ] Share streamlit dashboard as interactive PR.
• [ ] Should I be working in UTC?
• [ ] Review open issues on this repository, and on irg_realtime repository.

[ehmatthes]

Downloading new data

• [x] Go to USGS Current Conditions for USGS 15087700 INDIAN R AT SITKA AK
  • [x] Click Legacy real-time page, to end up here
• [x] Check Discharge, Gage height, and Tab-separated boxes
• [x] Set the date range so it overlaps the most recent data in ir_data_clean/irva_akdt_022016-102019_arch_format.txt, or most recent data file by one day.
  • [x] 12/30/20-03/31/24
• [x] Copy and paste the output to a text editor, and save with a descriptive filename in ir_data_other, so we have this specific data in its own file.
  • [x] Saved as irva_akdt_123020-033124_raw.txt.
  • [x] I believe this is "arch" format.
• [x] Append this data to the most recent data file in ir_data_clean. Do this by adding a couple blank lines to the end of the existing data file. Then paste in the data starting at the next timestamp. Examine the data across the blank lines, and check for continuity.
• [x] When you're confident this represents an accurate transition, remove the blank lines. Save with an updated filename to represent the new date range, and remove the existing data file.
• [x] Modify process_hx_data.py to use the new data file. This is in the data_files = [...] block.
• [x] Run tests. (They'll fail, because tests will run over the new dataset.)
• [x] Process the new data, and note any irregularities.
• [x] Make sure to add any known slides during this period, as described in readme.
  • [x] Make sure to run python slide_event.py after updating this file. Otherwise the new slides won't be included in the analysis and output.
• [x] It's always hard going back and trying to pinpoint landslide event times and information. It would be a really good habit to open an issue on this project any time there's a critical period, and any time there's a known slide. That's a perfect place to note any information that will be useful in later analysis.
  • [x] When researching past events on facebook, can go to anyone's timeline, on their Posts line click Filters, and can see their timeline starting at a given date.

[ehmatthes]

Run current tests

• [x] Build venv
  • [x] Fails on 3.12 and 3.11; try unpinning requirements and running. (Generates plots)
• [x] Test fails, can't import plot_heights. Import path issue? (Fixed by setting pythonpath in pytest.ini.)

[ehmatthes]

Update tests

• [x] Tests are brittle, because it's just running process_hx_data(), where the data to process is defined. So if you update the data source, tests break. Tests should specify which data to run against.

Tests pass for these files in process_hx_data():

    data_files = [
        'ir_data_clean/irva_utc_072014-022016_hx_format.txt',
        'ir_data_clean/irva_akdt_022016-123120_arch_format.txt',
        # 'ir_data_other/irva_akdt_123020-033124_arch_format.txt',
    ]

[ehmatthes]

Update for current data set

• [x] Failing on these lines; implement a try-except block to catch this.

USGS    15087700    2023-11-20 19:00    AKST    22.42   P   500 P
USGS    15087700    2023-11-20 19:15    AKST    22.45   P   514 P
USGS    15087700    2023-11-20 19:30    AKST    Eqp P   Eqp P
USGS    15087700    2023-11-20 19:45    AKST    Eqp P   Eqp P
USGS    15087700    2023-11-27 11:30    AKST    Eqp P   Eqp P
USGS    15087700    2023-11-27 11:45    AKST    21.85   P   275 P
USGS    15087700    2023-11-27 12:00    AKST    21.82   P   265 P

[ehmatthes]

Updating slides

• [ ] Does SSSC have a public record of recent slides?
  • [x] Emailed Ron 4/22/24.
• [ ] Are there any records of the recent slides near Starrigavan/ Katlian Bay mouth?

[ehmatthes]

Building a streamlit app

For data exploration, and helping people understand the project.

• [ ] Cache the actual dataset?

[ehmatthes]

Current output

--- Final Results ---

Data analyzed from: 09/05/2014 to 09/28/2023
  Critical rise used: 2.5 feet
  Critical rise rate used: 0.5 ft/hr

Notifications Issued: 20

True Positives: 6
  South Kramer Slide 8/2015 - Notification time: 41 minutes
  HPR Slide 9/2017 - Notification time: 60 minutes
  Medvejie Slide 9/2019 - Notification time: 35 minutes
  Olga Strait slides 10/26/20 - Notification time: 405 minutes
  Sand Dollar Drive Slide 11/2/2020 - Notification time: 25 minutes
  Crawfish Inlet slide 8/12/23 - Notification time: 330 minutes

False Positives: 14
  09/06/2014 21:00:00
  09/13/2016 13:00:00
  09/15/2016 05:45:00
  09/27/2017 03:15:00
  11/17/2018 21:49:00
  01/26/2019 14:30:00
  08/10/2020 06:35:00
  08/13/2021 16:00:00
  02/17/2022 02:30:00
  08/17/2022 21:00:00
  09/26/2022 21:15:00
  10/15/2022 10:00:00
  09/21/2023 06:30:00
  09/28/2023 00:00:00

False Negatives: 3
  Second Sand Dollar Drive Slide 11/2/2020
  HPR Slide 9/2016 (minor slide)
  Starrigavan Slide 9/2014 (time of slide unknown)

Slides outside range: 2
  Beaver Lake Slide 11/2011 (wind and snowmelt?)
  Redoubt Slide 5/2013 (not on Sitka road system)