Feature request: sleeplog: Expose more statistics

[myxor]

I would like to collect a list of possible further statistics data which the sleeplog app could deliver in future:

• Last Asleep Time
• Last Awake Duration
• Last Sleep Duration
• Total Awake Duration
• Total Sleep Duration

This could be done via the already existing global.sleeplog object.

Pinging @storm64 as creator of sleeplog.

Open for discussion and ideas.

[storm64]

This shouldn't be a big deal.

For the first two values I can simply add sleeplog.awakeSince as a timestamp from the last status change to awake (status == 2) and reset this value to undefined on the first detection of a sleeping state (status > 2).

• Last Asleep Time:
  Date(sleeplog.awakeSince)
• Last Awake Duration [ms]:
  Date.now() - sleeplog.awakeSince

To determine the other three values it is necessary to do little calculations over the logged data.

I assume you want to use this data on a watch face and therefore it could be the best approach to add these values as properties of sleeplog, too.

There are a few ways at wich point to calculate the values:

1. Each time the logfile is read,
2. when a new value is added to the logfile,
3. at a specific time of day,
4. on every request.

My preferred way would be the second one: On every new log entry. It's easy to implement and the fact that the logfile needs to be loaded already makes it less power consuming.

The more tricky question is how the last three values should be calculated. The total values are heavily depending on the used time span. I would suggest to use the same preferences as used for the app, wich would result in the following values for your requested values:

• Last Sleep Duration = consecutive sleeping as displayed inside the app
• Total Sleep Duration ~= true sleeping as displayed in the app

Or do you want to have the sum of all logged awake / sleeping periods?

I'm actually working on a more detailed sleep detection with a light and deep sleep phase and some code improvements. Therefore I need to make a lot of changes to the code and it will be no problem to implement these ideas.

Thanks for your suggestions, those are some great ideas.

[myxor]

Thanks for answering.

2. when a new value is added to the logfile I would prefer this way as well.

I would suggest to use the same preferences as used for the app, wich would result in the following values for your requested values

Fine for me. I guess it would be the best if the values via the "API" match the ones being shown in the app.

Therefore I need to make a lot of changes to the code and it will be no problem to implement these ideas.

if you could integrate some parts of this would be great :)

[GrandVizierOlaf]

@myxor I hope you don't mind my piggybacking here, but I am excited to hear that @storm64 is working on the detailed sleep detection; it's something I was hoping for. I can open a separate issue if that would be better.

@storm64, while you're working on that do you think you could add a debug option to log more details, rather than just the state changes? I was thinking that could be helpful for when a user is trying to dial in their personal thresholds for more accurate classification of the states. I found that I had to connect to my watch with the IDE and dump the sleeplog state to set the nomothreshold. It might also be possible to capture that data overnight in some kind of learning mode and then modify the values for the user before going back to non-debug mode automatically.

[myxor]

@GrandVizierOlaf sure that is fine.

@storm64 i am really looking forward to your rework of the sleeplog app. This night I had another idea: it would be great if we somehow could make a connection between the Quiet Mode and the current sleeping state. I would like the notification mode be toggled to "silent" or "alarm" while sleep is detected. Just as an idea :)

[storm64]

I'm happy to here that you are having fun with the app.

@GrandVizierOlaf: I stepped on the same problem while trying to figure out why no sleeping is recognized on @juanjgit's watch.

To tell you more about what I am working on, here a list of the major changes:

1. Deep and Light Sleep: The previous calculated sleeping is now called deep sleep. In addition to that I added a light sleep. A light sleep is detected if you are already sleeping but the movement/stddev value has exceeded the deep sleep threshold but is lower than a separate light sleep threshold.
2. ESS Calculation: While all the testing, I thought it might be helpful to add a option to ignore a specific amount of times the deep sleep threshold is exceeded.
3. Statistics: I am reworking the calculation of statistics to be done continuous and as soon as possible. For now this is done only and each time the app displays the data. This will speed up displaying, reduce the overall load and possible to make it available inside the sleeplog object. The available statistics wold look like following:
   • sleeplog.info The values are only calculated if not available and will then be changed according to status changes. awakeSince - timestamp of the last change from sleep to awake sumSleep - sum of all logged sleeping periods in seconds sumAwake - sum of all logged awake periods in seconds firstLogDate - timestamp of the first log entry
   • sleeplog.stats This values are calculated on every status change from sleep to awake. calculatedAt - timestamp of the calculation deepSleep - deep sleep duration of the last day lightSleep - light sleep duration of the last day consecSleep - consecutive sleep duration of the last day
4. Consecutive Sleep: As written above, for now the consecutive sleep status is only available through the app. To make the calculation of statistics more easy and displaying the app faster, I want to calculate the status on each logging and store the value within the log.
5. Log: The logged values are changing to: [timestamp, status, sleep]
   • status: 0 = unknown, 1 = not worn, 2 = awake, 3 = light sleep, 4 = deep sleep
   • sleep: 0 = unknown, 1 = not consecutive, 2 = consecutive
6. Start Debugging: There will be a new setDebug function in the module where you can enable debugging with the following options:
   • file (bool/string) filename of the StorageFile where debugging info is appended to;
     if omitted no debugging is written to storage and only displayed in the console; on true the filename will be set to debug_{hours since 1970}.log
   • append (bool) if not true a existing StorageFile would be erased
   • duration / writeUntil specify how long the debugging info will appended to the StorageFile;
     debugging to the console will only stop the debug property of sleeplog is deleted
   • interval set a interval to only write changes every x minutes to console and storage; the maximal values of value and exceeded will be outputted
7. Debug Output: Will look like this: 07:55:00.180 on ESS > status: 3, value: 0.1408, temp: 32.5 °C , times exceeded: 3
   • 07:55:00.180 timestamp
   • on ESS/PWM > power saving or ESS mode
   • status: 3, 0 = unknown, 1 = not worn, 2 = awake, 3 = light sleep, 4 = deep sleep
   • value: 0.1408, maximal movement value on PSM or stddev on ESS inside the interval
   • temp: 32.5 °C, value of E.getTemperature() at the timestamp
   • times exceeded: 3 only for ESS, maximal count a exceeded threshold is ignored (see 2.)
8. Toggle Quiet Mode: This could be a really nice idea. @myxor do you already have in mind how to realize this? Maybe calling require("qmsched").setMode(sleeping ? 1 : 2) from Quiet Mode Schedule and Widget could be enough?

At last I want to share a thought related to the future of ESS mode:

For comparison I installed the sleeplog app twice, each app using one mode (ESS/PSM). (I installed the second instance by replacing "sleeplog" with "sleeptest".) In my opinion the ESS calculation brings no real benefit, obviously except for a faster feedback on a status change to light sleep or awake but is extremely more power consuming and harder to set up correct. On the next version I would enable power saving by default. I could imagine to use the ESS calculation only when in deep sleep to detect a status change faster. What are your thoughts on this?

[GrandVizierOlaf]

@storm64 that all sounds fantastic. Let me know if you need any help testing it.

When you mention "the last day" with regards to sleeplog.stats, is that the current day, the previous day, the last 24 hours, or something else?

Reading through it again reminds me of an issue I ran into yesterday and a potential solution; the ambient temperature was 81°F (27 C) and my watch was unplugged and charging, but was detected as worn. I bumped the temperature threshold, but it gets above body temperature here in the summer and I worry that it will mess with the detection then. One potential shortcut is to check if the watch is charging and, if it is, know that it is not worn. You could also check the HRM stats every so often and see if that gives more accurate worn/not worn than the temperature.

[myxor]

Toggle Quiet Mode: This could be a really nice idea. @myxor do you already have in mind how to realize this? Maybe calling require("qmsched").setMode(sleeping ? 1 : 2) from Quiet Mode Schedule and Widget could be enough?

Yes indeed it looks like this could be enough to toggle the quiet mode! I will try this out in the next days.

[storm64]

When you mention "the last day" with regards to sleeplog.stats, is that the current day, the previous day, the last 24 hours, or something else?

I would stay with the same day periods as in the app. The duration is fixed to 24 hours. Start and end time is set by the "Break ToD" value in settings. As a result "the last day" is the period from second last to last time of day equal to "Break ToD".

@GrandVizierOlaf, I was unsatisfied with the "not wearing" check from the beginning. Checking for the charging status is a good idea and for checking the status if not charging I might have found a solution with minimal HRM up time:

function onWearingCheck(isWearing) {
  print("Wearing status:", isWearing);
}

function wearingCheck() {
  // define function to read wearing status
  var hrmListener = hrm => global.checkIsWearing = hrm.isWearing;
  // enable HRM
  Bangle.setHRMPower(true, "sleeplog");
  // wait for initialisation
  setTimeout(() => {
    // add HRM listener
    Bangle.on('HRM-raw', hrmListener);
    // set default wearing value
    global.checkIsWearing = false;
    // wait for two cycles (HRM working on 60Hz)
    setTimeout(() => {
      // disable HRM and remove listener
      Bangle.setHRMPower(false, "sleeplog");
      Bangle.removeListener('HRM-raw', hrmListener);
      // execute follow-up function with the result
      onWearingCheck(checkIsWearing);
      // clear cached status
      delete global.checkIsWearing;
    }, 34);
  }, 2500);
}

wearingCheck();

[myxor]

@storm64 any news on the sleeplog rework? :)

[storm64]

I'm on 80% for "public" testing. For now the last part to rewrite is the app.js for displaying the logged data.

[storm64]

I am proud to present the new sleeplog app: version 0.10 🎉 ✨ 🎊 https://storm64.github.io/BangleApps/?id=sleeplog

Sorry that it took so long but hopefully most of the early bugs are sorted out and the app should be ready to be use and get tested!

I would love to hear about your impressions and like to know your choice of thresholds, to set the default values as optimized as possible.

The last piece of work is to rewrite the README.md to show how to use it and show the restrictions and possibilities. But here are some explanations how to use the app and settings:

• On the app screen:

  • swipe left & right to change the displayed day
  • touch on the "title" (e.g. Night to Fri 20/05/2022) to enter a day selection prompt
  • touch on the info area (by default displaying consecutive and true sleeping) to change the displayed information
  • touch on the wrench (upper right corner) to enter the settings
  • exit the app with the UI back button widget on the upper left corner

• Inside the settings:

  • the threshold values are accessible through a submenu
  • app timeout lets you specify a separate lockTimeout and backlightTimeout only for the sleeplog app
  • debug settings are available in a submenu down at the end:
  • display log is not implemented yet
  • options Enable and write File should be self explaining
  • the Duration specifies how long data should be written into the .csv file
  • the .csv file loggs the following data (timestamps are in days since 1900-01-01 00:00 UTC as used by office software):
    timestamp, movement, status, consecutive, asleepSince, awakeSince, bpm, bpmConfidence

• Timestamps and files:

  1. externally visible/usable timestamps (in global.sleeplog) are formatted as Bangle timestamps:
     milliseconds since 1970-01-01 00:00 UTC
  2. internally used and logged (to sleeplog.log (StorageFile)) is within the highest available resolution:
     10 minutes since 1970-01-01 00:00 UTC (Bangle / (10 * 60 * 1000))
  3. debug .csv file ID (sleeplog_123456.csv) has a hourly resolution: hours since 1970-01-01 00:00 UTC (Bangle / (60 * 60 * 1000))
  4. logged timestamps inside the debug .csv file are formatted for office calculation software: days since 1900-01-01 00:00 UTC (Bangle / (24 * 60 * 60 * 1000) + 25569)
  5. every 14 days the sleeplog.log (StorageFile) is reduced and old entries are moved into separat files for each fortnight (sleeplog_1234.log) but still accessible though the app:
     fortnights since 1970-01-04 12:00 UTC (converted with require("sleeplog").msToFn(Bangle) and require("sleeplog").fnToMs(fortnight))

• Logfiles from before 0.10:
  timestamps and sleeping status of old logfiles are automatically converted on your first consecutive sleep or manually by require("sleeplog").convertOldLog()

• View logged data:
  if you'd like to view your logged data in the IDE, you can access it with require("sleeplog").printLog(since, until) or require("sleeplog").readLog(since, until) to view the raw data
  since & until in Bangle timestamp, e.g. require("sleeplog").printLog(Date()-24*60*60*1000, Date()) for the last 24h

[myxor]

Thank you @storm64, i will try it out the next days (nights).

[storm64]

I would like to collect a list of possible further statistics data which the sleeplog app could deliver in future:

The solutions with the new app:

• Last Asleep Time [Date]:
  Date(sleeplog.awakeSince)
• Last Awake Duration [ms]:
  Date() - sleeplog.awakeSince

• ~Last Sleep Duration~ Last Stats [object]:

  // get stats of the last night (period as displayed inside the app)
  //  as this might be the mostly used function the data is cached inside the global object 
  sleeplog.getStats();
  
  // get stats of the last 24h
  require("sleeplog").getStats(0, 24*60*60*1000);
  // same as
  require("sleeplog").getStats(Date.now(), 24*60*60*1000);
  // output as object, timestamps as UNIX timestamp, durations in minutes
  ={ calculatedAt: 1653123553810, deepSleep: 250, lightSleep: 150, awakeSleep: 10,
  consecSleep: 320, awakeTime: 1030, notWornTime: 0, unknownTime: 0, logDuration: 1440,
  firstDate: 1653036600000, lastDate: 1653111600000 }
  
  // to get the start of a period defined by "Break TOD" of any date
  var startOfBreak = require("sleeplog").getLastBreak();
  // same as
  var startOfBreak = require("sleeplog").getLastBreak(Date.now());
  // output as date
  =Date: Sat May 21 2022 12:00:00 GMT+0200
  
  // get stats of this period as displayed inside the app
  require("sleeplog").getStats(require("sleeplog").getLastBreak(), 24*60*60*1000);
  // or any other day
  require("sleeplog").getStats(require("sleeplog").getLastBreak(Date(2022,4,10)), 24*60*60*1000);

• ~Total Awake Duration + Total Sleep Duration~ Total Stats [object]:

  // use with caution, may take a long time !
  require("sleeplog").getStats(0, 0, require("sleeplog").readLog());

---

I would like the notification mode be toggled to "silent" or "alarm" while sleep is detected.

For now I forgot about this but would like to implement it.

Toggle Quiet Mode: This could be a really nice idea. @myxor do you already have in mind how to realize this? Maybe calling require("qmsched").setMode(sleeping ? 1 : 2) from Quiet Mode Schedule and Widget could be enough?

Yes indeed it looks like this could be enough to toggle the quiet mode! I will try this out in the next days.

Have you tried if require("qmsched").setMode(sleeping ? 1 : 2) is the correct approach?

And how should it be triggered:

1. set to 1 on
   1. first change to deep sleep
   2. first change to consecutive sleep (delayed by the "Min Consecutive" threshold)
2. set to 2 on
   1. first change to awake (might be in the middle of the night)
   2. first change to non consecutive sleep (delayed by the "Max Awake" threshold)

In my opinion I would prefer a option in the settings (showing if qmsched is installed) like

               never     -> off
set Quiet on   first     -> 1.i.
               consec.   -> 1.ii.

[storm64]

@gfwilliams I'm working on sending the sleep state to gadgetbridge and found the following two java files that might be interesting for this:

.../model/ActivityKind.java defining the different kinds of activities.
A list of the interesting kinds for sleeplog:

1. TYPE_UNKNOWN
2. TYPE_NOT_WORN
3. TYPE_ACTIVITY
4. TYPE_LIGHT_SLEEP
5. TYPE_DEEP_SLEEP

and

.../service/devices/banglejs/BangleJSDeviceSupport.java handling the communication with Bangle.js.
Where the interesting part for activities could be altered just a bit:

  372            case "act": {
  373                BangleJSActivitySample sample = new BangleJSActivitySample();
- 374                sample.setTimestamp((int) (GregorianCalendar.getInstance().getTimeInMillis() / 1000L));
+                    int ts = GregorianCalendar.getInstance().getTimeInMillis() / 1000L;
  375                int hrm = 0;
  376                int steps = 0;
+                    int activity = BangleJSSampleProvider.TYPE_NOT_MEASURED;
+                    if (json.has("ts")) ts = json.getLong("ts") / 1000L;
  377                if (json.has("hrm")) hrm = json.getInt("hrm");
  378                if (json.has("stp")) steps = json.getInt("stp");
- 379                int activity = BangleJSSampleProvider.TYPE_ACTIVITY;
- 380                /*if (json.has("act")) {
+                    if (json.has("act")) {
  381                    String actName = "TYPE_" + json.getString("act").toUpperCase();
  382                    try {
  383                        Field f = ActivityKind.class.getField(actName);
  384                        try {
  385                            activity = f.getInt(null);
  386                        } catch (IllegalAccessException e) {
  387                            LOG.info("JSON activity '"+actName+"' not readable");
  388                        }
  389                    } catch (NoSuchFieldException e) {
  390                        LOG.info("JSON activity '"+actName+"' not found");
  391                    }
- 392                }*/
+                    }
+                    sample.setTimestamp(ts);
- 393                sample.setRawKind(activity);
  394                sample.setHeartRate(hrm);
  395                sample.setSteps(steps);
+                    sample.setRawKind(activity);
  396                try (DBHandler dbHandler = GBApplication.acquireDB()) {
  397                    Long userId = getUser(dbHandler.getDaoSession()).getId();
  398                    Long deviceId = DBHelper.getDevice(getDevice(), dbHandler.getDaoSession()).getId();
  399                    BangleJSSampleProvider provider = new BangleJSSampleProvider(getDevice(), dbHandler.getDaoSession());
  400                    sample.setDeviceId(deviceId);
  401                    sample.setUserId(userId);
  402                    provider.addGBActivitySample(sample);
  403                } catch (Exception ex) {
  404                    LOG.warn("Error saving activity: " + ex.getLocalizedMessage());
  405                }
  406                // push realtime data
  407                if (realtimeHRM || realtimeStep) {
  408                    Intent intent = new Intent(DeviceService.ACTION_REALTIME_SAMPLES)
  409                            .putExtra(DeviceService.EXTRA_REALTIME_SAMPLE, sample);
  410                    LocalBroadcastManager.getInstance(getContext()).sendBroadcast(intent);
  411                }
  412            } break;

The according code snippet to trigger a status change inside sleeplog would be as following:

      // send status to gadgetbridge
      var gb_kind = "unknown,not_worn,activity,light_sleep,deep_sleep";
      Bluetooth.println(JSON.stringify({
        t: "act",
        act: gb_kind.split(",")[data.status],
        ts: data.timestamp // as UNIX timestamp in ms
      }));

I am not sure if this works as easy es I might think/hope, especially the behavior from gadgetbridge receiving two separate actions (one with steps+hrm and one with the activity), but this could be altered within Bangle itself if neccessary.

I would really like to here your thoughts and am excited to take the sleeplog app a step further.

[gfwilliams]

Yes, this looks promising - and the code is already there, just disabled? However is the timestamp thing really required? Can't we just send the update when the activity type actually changes?

[storm64]

Due to the fact that i need to evaluate the movement collected over the last 10 minutes, a status change had occurred 10 minutes ago. This is taken into account in the sleeplog app and to use the same base of data, I would recommend to hand over the corrected timestamp.

[gfwilliams]

Ok, thanks. I guess it also has the benefit that maybe the Bangle can 'catch up' if it's been disconnected from Gadgetbridge for a while

[myxor]

@storm64 just wanted to let you know that i saw this exception happening today a few times:

Uncaught Error: Module "sleeplog" not found
 at line 105 col 4013 in .boot0
...les.removeCached("sleeplog");}
                              ^
in function "setStatus" called from line 105 col 2258 in .boot0
...;}else{sleeplog.setStatus(data);}

[storm64]

Sry, that this took me so long, but the module not found error should be fixed with the new beta04.

[storm64]

Hi @myxor,

just wanted to remind you of my question:

Toggle Quiet Mode: This could be a really nice idea. @myxor do you already have in mind how to realize this? Maybe calling require("qmsched").setMode(sleeping ? 1 : 2) from Quiet Mode Schedule and Widget could be enough?

Yes indeed it looks like this could be enough to toggle the quiet mode! I will try this out in the next days.

Have you tried if require("qmsched").setMode(sleeping ? 1 : 2) is the correct approach?

And how should it be triggered:

1. set to 1 on
   1. first change to deep sleep
   2. first change to consecutive sleep (delayed by the "Min Consecutive" threshold)
2. set to 2 on
   1. first change to awake (might be in the middle of the night)
   2. first change to non consecutive sleep (delayed by the "Max Awake" threshold)

In my opinion I would prefer a option in the settings (showing if qmsched is installed) like

               never     -> off
set Quiet on   first     -> 1.i.
               consec.   -> 1.ii.

[myxor]

In my opinion I would prefer a option in the settings (showing if qmsched is installed) like

               never     -> off
set Quiet on   first     -> 1.i.
               consec.   -> 1.ii.

This sounds to me as the best approach as well.

Have you tried if require("qmsched").setMode(sleeping ? 1 : 2) is the correct approach?

I did not yet try it but from my view on the qmsched code this looks correct.

[bobrippling]

I think we can close this issue now, if there's nothing else outstanding?

[gfwilliams]

Looks good - we can always reopen if there is something specific