[Bug] Today's data not availabe on sleep sensor

[nitobuendia]

Originally posted by @akeslo in https://github.com/nitobuendia/oura-custom-component/issues/10#issuecomment-1379645988

My Config

  - platform: oura
    access_token: XXXXREMOVEDXXXXX
    scan_interval: 7200
    sensors:
     sleep:
       name: oura_sleep_metrics
       max_backfill: 0
       monitored_dates:
        - 0d_ago
        - 1d_ago
        - 7d_ago 
     sleep_periods:
       name: oura_sleep_periods
       max_backfill: 0
       monitored_dates:
        - 0d_ago
        - 1d_ago
        - 7d_ago       
     sleep_score:
       name: oura_sleep_score
       max_backfill: 0 
       monitored_dates:
        - 0d_ago
        - 1d_ago
        - 7d_ago

sleep (oura_sleep_metrics)

• 0d_ago is returning null for today however I have data via cloud.ouraring.com for 2023-01-11

[nitobuendia]

Checking some of the code, 0d_ago should be working and returns data for my own sensor:

0d_ago: 
day: '2023-01-15'
lowest_heart_rate: 50
total_sleep_duration_in_hours: 6.07

which matches the Cloud UI (* 6.07h = 6h and 4 min)

Since I cannot currently reproduce, it's hard to know the root cause. If I had to make a guess, I would think this has something to do to how we do the mapping of dates to data points. However, I also cannot see anything there since parse_individual_data_point does not modify the day for the sleep sensor.

Myabe it has something to do with how the API reports the date based on the time you went to bed. For example, my bedtime was past midnight, so the sleep period appeared today. Perhaps if I had gone bed before midnight it would not appear on today's date but yesterday's. I do recall something like this when I first created this component, but that's long ago.

I will have another look next week weekend.

In the meantime, @akeslo, one thing you can try is comparing the data from the API to what the sensors report. You can call the API like this from your terminal:

curl --location --request GET 'https://api.ouraring.com/v2/usercollection/sleep?start_date=<START_DATE>&end_date=<END_DATE>' \
--header 'Authorization: Bearer <TOKEN>'

Replacing:

• with a few days ago in YYYY-MM-DD.
• with today's date in YYYY-MM-DD.
• with your personal token from Oura.

The data returned should match sleep_periods sensor. One of the long_sleep entries for a given day will be selected for the sleep sensor.

[akeslo]

Time of this post: 21:46 Eastern Time

Sleep Sensor for Today: 2023-01-17

When I hit the API via CURL:

1 - For today as start/end date - I get no results.

curl --location --request GET 'https://api.ouraring.com/v2/usercollection/sleep?start_date=2023-01-17&end_date=2023-01-17' \
--header 'Authorization: Bearer <TOKEN>

Results: {"data":[],"next_token":null}

2. If I extend until tomorrow, I get data for today.

   curl --location --request GET 'https://api.ouraring.com/v2/usercollection/sleep?start_date=2023-01-17&end_date=2023-01-18' \
   --header 'Authorization: Bearer <TOKEN>

   This returns a data array with 2 objects both dated today:

This is an interesting thought but even with the dataset above, the bedtime_start attributes on both are before midnight:

Myabe it has something to do with how the API reports the date based on the time you went to bed. For example, my bedtime was past midnight, so the sleep period appeared today. Perhaps if I had gone bed before midnight it would not appear on today's date but yesterday's. I do recall something like this when I first created this component, but that's long ago.

Here's the curl output I'm getting:

{
    "data": [
        {
            "average_breath": 13.875,
            "average_heart_rate": 66.625,
            "average_hrv": null,
            "awake_time": 720,
            "bedtime_end": "2023-01-16T19:29:29-05:00",
            "bedtime_start": "2023-01-16T19:16:29-05:00",
            "day": "2023-01-17",
            "deep_sleep_duration": 0,
            "efficiency": 8,
            "heart_rate": {
                "interval": 300.0,
                "items": [
                    88.0,
                    null,
                    null
                ],
                "timestamp": "2023-01-16T19:16:29.000-05:00"
            },
            "hrv": {
                "interval": 300.0,
                "items": [
                    28.0,
                    null,
                    null
                ],
                "timestamp": "2023-01-16T19:16:29.000-05:00"
            },
            "latency": 0,
            "light_sleep_duration": 60,
            "low_battery_alert": false,
            "lowest_heart_rate": 88,
            "movement_30_sec": "22221111322111111111132233",
            "period": 0,
            "readiness": null,
            "readiness_score_delta": null,
            "rem_sleep_duration": 0,
            "restless_periods": 2,
            "sleep_phase_5_min": "421",
            "sleep_score_delta": null,
            "time_in_bed": 780,
            "total_sleep_duration": 60,
            "type": "sleep"
        },
        {
            "average_breath": 13.875,
            "average_heart_rate": 69.625,
            "average_hrv": 59,
            "awake_time": 6090,
            "bedtime_end": "2023-01-17T07:03:00-05:00",
            "bedtime_start": "2023-01-16T21:41:00-05:00",
            "day": "2023-01-17",
            "deep_sleep_duration": 6780,
            "efficiency": 82,
            "heart_rate": {
                "interval": 300.0,
                "items": [
                    null,
                    null,
                    85.0,
                    82.0,
                    80.0,
                    80.0,
                    79.0,
                    74.0,
                    72.0,
                    73.0,
                    73.0,
                    74.0,
                    73.0,
                    74.0,
                    75.0,
                    75.0,
                    73.0,
                    72.0,
                    70.0,
                    76.0,
                    74.0,
                    74.0,
                    75.0,
                    71.0,
                    72.0,
                    67.0,
                    66.0,
                    69.0,
                    71.0,
                    72.0,
                    71.0,
                    69.0,
                    67.0,
                    65.0,
                    65.0,
                    68.0,
                    66.0,
                    68.0,
                    68.0,
                    68.0,
                    70.0,
                    72.0,
                    73.0,
                    73.0,
                    70.0,
                    68.0,
                    67.0,
                    68.0,
                    64.0,
                    63.0,
                    66.0,
                    65.0,
                    65.0,
                    65.0,
                    64.0,
                    65.0,
                    67.0,
                    67.0,
                    66.0,
                    67.0,
                    66.0,
                    66.0,
                    68.0,
                    69.0,
                    75.0,
                    76.0,
                    73.0,
                    70.0,
                    69.0,
                    69.0,
                    67.0,
                    70.0,
                    71.0,
                    69.0,
                    68.0,
                    69.0,
                    67.0,
                    64.0,
                    65.0,
                    65.0,
                    63.0,
                    64.0,
                    65.0,
                    67.0,
                    65.0,
                    65.0,
                    63.0,
                    62.0,
                    63.0,
                    65.0,
                    63.0,
                    64.0,
                    66.0,
                    68.0,
                    68.0,
                    66.0,
                    66.0,
                    68.0,
                    66.0,
                    64.0,
                    61.0,
                    61.0,
                    61.0,
                    62.0,
                    63.0,
                    63.0,
                    61.0,
                    61.0,
                    63.0,
                    64.0,
                    62.0,
                    null,
                    null
                ],
                "timestamp": "2023-01-16T21:41:00.000-05:00"
            },
            "hrv": {
                "interval": 300.0,
                "items": [
                    null,
                    null,
                    18.0,
                    19.0,
                    22.0,
                    22.0,
                    25.0,
                    34.0,
                    33.0,
                    28.0,
                    29.0,
                    29.0,
                    31.0,
                    28.0,
                    41.0,
                    49.0,
                    47.0,
                    54.0,
                    63.0,
                    27.0,
                    34.0,
                    41.0,
                    41.0,
                    51.0,
                    54.0,
                    60.0,
                    59.0,
                    53.0,
                    36.0,
                    36.0,
                    48.0,
                    70.0,
                    60.0,
                    67.0,
                    71.0,
                    47.0,
                    56.0,
                    46.0,
                    48.0,
                    43.0,
                    50.0,
                    54.0,
                    36.0,
                    34.0,
                    57.0,
                    68.0,
                    63.0,
                    65.0,
                    92.0,
                    95.0,
                    88.0,
                    86.0,
                    84.0,
                    67.0,
                    68.0,
                    50.0,
                    52.0,
                    51.0,
                    52.0,
                    73.0,
                    70.0,
                    75.0,
                    63.0,
                    55.0,
                    22.0,
                    19.0,
                    34.0,
                    37.0,
                    45.0,
                    44.0,
                    78.0,
                    46.0,
                    56.0,
                    51.0,
                    50.0,
                    45.0,
                    50.0,
                    73.0,
                    66.0,
                    65.0,
                    68.0,
                    63.0,
                    56.0,
                    62.0,
                    79.0,
                    69.0,
                    69.0,
                    71.0,
                    61.0,
                    59.0,
                    80.0,
                    78.0,
                    74.0,
                    85.0,
                    64.0,
                    80.0,
                    69.0,
                    72.0,
                    78.0,
                    81.0,
                    82.0,
                    82.0,
                    71.0,
                    81.0,
                    68.0,
                    73.0,
                    93.0,
                    89.0,
                    87.0,
                    105.0,
                    97.0,
                    null,
                    null
                ],
                "timestamp": "2023-01-16T21:41:00.000-05:00"
            },
            "latency": 1950,
            "light_sleep_duration": 17100,
            "low_battery_alert": false,
            "lowest_heart_rate": 61,
            "movement_30_sec": "33221211121122321111122111111211111111111111121211111111111211231111111111111111112111111111111111111111111111111111111111111221111111111111111111111111122122211111222223121111111112222111112222211111112121122121122121111111111233121121111111111121123343111111211221111111111123211111111111111111111111111111212111112321223111111111111111111111111111211211111111111111111111111111111111111111111111111111111111321221111111111222111111121111111111112221111111112212111111111111111121132111111111111111111112121121111111111111111321111112111111111211111111111111111111111111111111111111111111111111131111111111111111113111212121111112211111121111111111111123112323211112332211111111111122111111111111132122211111111111221111122211111121111111121112211221111111111111111111132211111111111111111111111111111111111111111111112122211111111111112211211131111111222211111111111111111111111111111111111111111111222111111112211111111111111111113221111112111111111111111112111111121211211111222121111111122122111111111211111111111111111111111111111111111111111111111111111111311111221111111111111111111111111321111211211111122222323311",
            "period": 1,
            "readiness": {
                "contributors": {
                    "activity_balance": 88,
                    "body_temperature": 69,
                    "hrv_balance": 93,
                    "previous_day_activity": 90,
                    "previous_night": 82,
                    "recovery_index": 92,
                    "resting_heart_rate": 41,
                    "sleep_balance": 75
                },
                "score": 70,
                "temperature_deviation": 0.42,
                "temperature_trend_deviation": 0.15
            },
            "readiness_score_delta": 0,
            "rem_sleep_duration": 3750,
            "restless_periods": 280,
            "sleep_phase_5_min": "44444441221112114422344424222111122221111443222222212222111112222344334223322122222322222212221322222222333223224",
            "sleep_score_delta": 0,
            "time_in_bed": 33720,
            "total_sleep_duration": 27630,
            "type": "long_sleep"
        }
    ],
    "next_token": null
}

[nitobuendia]

First, I had a look at my own data again now that I have data points with different start and end dates.

I just tried on the API on (2023-01-18) while going on 2023-01-17 to bed, the day for data point is marked as 2023-01-18. As such today (or "0d_ago") should work. If not, there's a potential bug.

API Data for today:

{
    "data": [
        {
            "average_breath": 13.5,
            "average_heart_rate": 57.5,
            "average_hrv": 66,
            "awake_time": 5040,
            "bedtime_end": "2023-01-18T07:56:44+08:00",
            "bedtime_start": "2023-01-17T23:53:44+08:00",
            "day": "2023-01-18",
(...)

In my case, however, the sensor is indeed loading the data and it matches today:

0d_ago: 
  average_breath: 13.5
  average_heart_rate: 57.5
  day: '2023-01-18'
  awake_duration_in_hours: 1.4
  bedtime_end_hour: '07:56'
  bedtime_start_hour: '23:53'

I can't still reproduce the issue, unfortunately.

---

@akeslo Looking at your data and tests, the only thing I can think of is (1) timezones or (2) we're sending today's data as the end date.

I can do (2) to see if that solve it. However, my API behaviour is different.

• If I send a request for today ?start_date=2023-01-18&end_date=2023-01-18, I get no data.
• If I send a request from today to tomorrow ?start_date=2023-01-18&end_date=2023-01-19, I get no data.
• If I send a request from yesterday to today ?start_date=2023-01-17&end_date=2023-01-18, I get the data I shared above.

I am still not able to reproduce your issue, even on the API.

For (1), I am not sure what would be the solution, or how to explore as of right now. If I look at API documentation, the default setting for end_date is YYYY-MM-DD of the "current UTC date". Maybe they are using UTC for the date?

Looking at my data:

            "bedtime_end": "2023-01-18T07:56:44+08:00",
            "bedtime_start": "2023-01-17T23:53:44+08:00",
            "day": "2023-01-18",

translated to UTC would be:

            "bedtime_end": "2023-01-17T23:56:44+00:00",
            "bedtime_start": "2023-01-17T15:53:44+00:00",
            "day": "2023-01-18",

Both dates would fall on 17th, so it doesn't make sense that this is the reason as I'd expect the day to be 2023-01-17 then.

Looking at your data:

// data 1
            "bedtime_end": "2023-01-16T19:29:29-05:00",
            "bedtime_start": "2023-01-16T19:16:29-05:00",
            "day": "2023-01-17",

// data 2
            "bedtime_end": "2023-01-17T07:03:00-05:00",
            "bedtime_start": "2023-01-16T21:41:00-05:00",
            "day": "2023-01-17",

translate to UTC would be:

// data 1
            "bedtime_end": "2023-01-17T00:29:29+00:00",
            "bedtime_start": "2023-01-17T00:16:29+00:00",
            "day": "2023-01-17",

// data 2
            "bedtime_end": "2023-01-17T12:03:00+00:00",
            "bedtime_start": "2023-01-17T02:41:00+00:00",
            "day": "2023-01-17",

All the dates are on 2023-01-17, so that could explain why the 16th data is on 17th if you convert it to UTC. It's not confirmed if that's the reason, though.

All in all, I can try just adding a +1 on the end date of the data requested. For users who don't have this problem, this should not affect - but it seems this may fix your use case.

[nitobuendia]

@akeslo to test the hypothesis on the previous comment, you would need to change sensor_base.py > _get_monitored_date_range function on line 244 for this one:

  def _get_monitored_date_range(self):
    """Returns tuple containing start and end date based on monitored dates.

    Returns:
      (start_date, end_date) in YYYY-MM-DD
    """
    sensor_dates = self._get_monitored_name_days()

    today_date = datetime.datetime.today().strftime('%Y-%m-%d')
    sensor_dates = sensor_dates.values()
    start_date = (
        min(sensor_dates) if len(sensor_dates) > 0 else today_date)
    end_date = (
        max(sensor_dates) if len(sensor_dates) > 0 else today_date)

    # Add an extra week to retrieve past week in case current week data is
    # missing.
    start_date = date_helper.add_days_to_string_date(start_date, -7)

    # Add an extra day to retrieve today's date in case of timezone difference.
    end_date = date_helper.add_days_to_string_date(end_date, 1)

    return (start_date, end_date)

Would you be able to try out and see if that works (after restarting, etc)?

The main difference here is this line: end_date = date_helper.add_days_to_string_date(end_date, 1)

This would replicate your behaviour of asking for tomorrow instead of today for the end_date. For me, it has no impact, as 0d_ago is working fine. That's somehow a good thing. However, for you, it could mean being able to load 0d_ago data.

Let me know if you're willing to try and if it works, I am willing to commit the code.

Thanks again!

[akeslo]

Awesome, that does it!

It also reminded me that I landed on something similar on my crude fork, though I don't think I knew at the time why :).

From fork, it was using a range from yest to tom:

        yest_string = _get_date_string(-1)
        tom_string = _get_date_string(1)
        daily_sleep_response = api.get_data(
            self._oura_token, oura_api.OuraURLs.DAILY_SLEEP, yest_string, tom_string
        )

Thanks for looking into this!

[nitobuendia]

Thanks for confirming @akeslo - I will publish the change when I can. Yesterday to today logic should work if you're looking only for one yesterday/today data. This should do the same but from the max range needed from the monitored_dates.