What should be the default unit in logging interval?

[thinrope]

I was planning to propose to change the default logging interval unit to seconds, but got sidetracked... Now the code has been re-factored and my patch (see commit 2058008edf9e9dacf3347b41a38b7560e8a180c) will need update a bit.

The reasons for seconds vs. minutes is that it it is a finer grained unit and will save some conversion for display. Sometimes getting more than one reading per minute is desirable and impossible with current stable firmware.

Of course most people don't know what 86400s means, but for most small intervals it is easy enough to calculate. Plus emerging GUIs can tackle that.

Anybody against that change?

[elafargue]

I would suggest leaving a long logging interval by default (30 minutes), so that new users don't end up with a full log too fast, or end up just wearing out the flash.

On the other hand, if a user actually needs to log faster, they should be able to select shorter intervals - does less than 1 minute actually make sense with a 30 seconds integration window?

[nokton]

In the bGeigie we use two windows: 60 seconds and 5 seconds. In areas with high radiation the 5 second window will better contour any sudden changes in the field under measurement, while in areas with low radiation tiny hotspots can be better located as one can see exactly when a spike was recorded.

Pieter

Nokton Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On 2013年4月28日Sunday at 10:46, elafargue wrote:

I would suggest leaving a long logging interval by default (30 minutes), so that new users don't end up with a full log too fast, or end up just wearing out the flash. On the other hand, if a user actually needs to log faster, they should be able to select shorter intervals - does less than 1 minute actually make sense with a 30 seconds integration window?

— Reply to this email directly or view it on GitHub (https://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17126697).

[new299]

Perhaps we have data entered in MM:SS format as we do on the time setting screen?

I agree that the default should stay at 30mins, 30mins should give us a logging mode battery life of a few months which was one of our design goals.

@thinrope what are the use-cases for a <1min logging interval? Would you also need an option to change the logging integration time?

[thinrope]

First, a clarification: this is about the unit (=seconds or minutes) used for display for the logging interval: I suggest forgetting minutes at all and using seconds.

I have absolutely no issue with default being 1800s (=30min), it should just be displayed as 1800s. But I want to be able to specify say 5s or even 1s and effectively be able to use it as a mobile sensor (knowing it will fill up the flash quickly and eventually kill it). Also, using the same semantics for streaming data over serial instead of logging (issue #19) will avoid that.

Now, I hadn't figured out that there is also "integration time" and that needs to be fiddled with as well... Looking at how the counter is implemented, I guess the easiest thing is to output the number of counts since "last log" divided by minutes since last log. This can be as trivial as get_total_count(), although that can be reset. To avoid rounding errors, I'd rather log the raw count for the interval and let display software handle division or whether to display CPM or CPS. It will also make arbitrary moving averages easier (integer math), as well as changing the logging interval on the fly independent. And finally it solves (actually avoids) dealing with uncertainty of the measurement since counts is a directly observed phenomenon (i.e. raw data) while any kind of "integration windows", cpm/cps, etc. are calculated.

[Frangible]

Interesting convergence. That is ultimately exactly what I ended up doing for Geiger Bot / the Safecast app... primary logging value is the total number of counts since the last logging interval. (also how I did the graphs, for better or worse.)

I also went with an integer value in seconds as the logging interval, and later locked it down to be selected from a list, with the key the number of seconds, and display text the value in hh:mi:ss. Simplistic, but the list seemed to encompass all the intervals people wanted.

And that's why you weren't the only one who knew how long 86400 seconds was...

On Apr 29, 2013, at 12:21 AM, Kalin KOZHUHAROV notifications@github.com wrote:

First, a clarification: this is about the unit (=seconds or minutes) used for display for the logging interval: I suggest forgetting minutes at all and using seconds.

I have absolutely no issue with default being 1800s (=30min), it should just be displayed as 1800s. But I want to be able to specify say 5s or even 1s and effectively be able to use it as a mobile sensor (knowing it will fill up the flash quickly and eventually kill it). Also, using the same semantics for streaming data over serial instead of logging (issue #19) will avoid that.

Now, I hadn't figured out that there is also "integration time" and that needs to be fiddled with as well... Looking at how the counter is implemented, I guess the easiest thing is to output the number of counts since "last log" divided by minutes since last log. This can be as trivial as get_total_count(), although that can be reset. To avoid rounding errors, I'd rather log the raw count for the interval and let display software handle division or whether to display CPM or CPS. It will also make arbitrary moving averages easier (integer math), as well as changing the logging interval on the fly independent. And finally it solves (actually avoids) dealing with uncertainty of the measurement since counts is a directly observed phenomenon (i.e. raw data) while any kind of "integration windows", cpm/cps, etc. are calculated.

\ Reply to this email directly or view it on GitHub.

[new299]

Unfortunately the device is completely asleep for ("Log Interval" mins) between logging periods, so we can't acquire all pulses between logging periods (the logging period is always 30seconds). During the sleep period (Log Interval) the geiger HV board is not power on, and the CPU would not respond to interrupts generated by geiger pulses in any case.

Here's what currently happens when the device is in logging mode (switch in "off" position):

1. User flicks the switch to sleep mode.
2. STM32 insures a RTC alarm is set for the next logging time (current time+"Log Interval").
3. STM32 goes into standby mode. The CPU is powered off, CPU register contents are lost, memory contents is lost.
4. STM32 wakes up on the RTC alarm. This is basically a reset, but by checking some registers you can tell that the STM32 woke up on an RTC alarm.
5. HV board is powered back up, the device reads geiger pulses for 30seconds.
6. After 30 seconds it writes the log entry to flash.
7. Sets a new RTC alarm for the next log time (last log time + "Log Interval").
8. STM32 goes into standby mode, back to step 3.

[thinrope]

Thanks for that explanation! Now that I think I understand it, my last commit ad02ac79e225a750b29897d99f6ab8db6d645193 may not be working in sleep mode... Will do a test.

[new299]

@thinrope yes, the interval and count stuff in log_data_t shouldn't work as you might expect it to. If I've read the commit correctly it will only contain 30second of data. If it does contain more we have a problem because the device isn't going into standby mode correctly. :)

[LucidOne]

Based on all of the above, I think it makes more sense that the default log interval units for flash are in minutes, this helps indicate to the user that shorter times are not practical.

Logging to serial however should have a resolution of seconds.

[thinrope]

Oh well... Github didn't refresh the page and I submitted this without reading the last comment... So still not clear then...

Well, I did the test with current devel and guess what? I got a log every 30 seconds in sleep mode ;-) So, this means we are not sleeping 30min but 30s... One more reason to move everything into seconds (except date/time display and logging).

Since this issue has grown too big in note very related comments and I haven't heard anybody against switching to seconds (and keeping 30*60s default logging interval), I'll rediff my commit and push it to devel. The rest of the issues will be filed/dealt separately.

[LucidOne]

On the UX side, it may be worth throwing a warning if you try logging more often than once a minute.

[nokton]

If you're at it - to use the Onyx as a bGeigie, could we have the Onyx log, say every 5 seconds and use a generic datalogger (e.g. iPhone/Andoird generic app) and merge the two data streams into a bGeigie stream (could be a databot@safecast.org function for that matter)

Low tech way to get a bGeigie out of an Onyx.

All we need would be a simple mounting 'sleeve' to put the Onyx on the car window.

Pieter

On Apr 29, 2013, at 9:45 PM, Kalin KOZHUHAROV notifications@github.com wrote:

Well, I did the test with current devel and guess what? I got a log every 30 seconds in sleep mode ;-) So, this means we are not sleeping 30min but 30s... One more reason to move everything into seconds (except date/time display and logging).

Since this issue has grown too big in note very related comments and I haven't heard anybody against switching to seconds (and keeping 30*60s default logging interval), I'll rediff my commit and push it to devel. The rest of the issues will be filed/dealt separately.

— Reply to this email directly or view it on GitHub.

[elafargue]

Given the limited r/w cycles of the ST32's flash - we're at least one order of magnitude below dedicated flash chips here and i'm not sure there is any wear levelling built in - , I'd suggest not logging to flash under a minute or more at the very minimum, and use data streaming for external logging if a faster rate is required, as you described above.

On Mon, Apr 29, 2013 at 12:16 PM, Pieter Franken notifications@github.comwrote:

If you're at it - to use the Onyx as a bGeigie, could we have the Onyx log, say every 5 seconds and use a generic datalogger (e.g. iPhone/Andoird generic app) and merge the two data streams into a bGeigie stream (could be a databot@safecast.org function for that matter)

Low tech way to get a bGeigie out of an Onyx.

All we need would be a simple mounting 'sleeve' to put the Onyx on the car window.

Pieter

On Apr 29, 2013, at 9:45 PM, Kalin KOZHUHAROV notifications@github.com wrote:

Well, I did the test with current devel and guess what? I got a log every 30 seconds in sleep mode ;-) So, this means we are not sleeping 30min but 30s... One more reason to move everything into seconds (except date/time display and logging).

Since this issue has grown too big in note very related comments and I haven't heard anybody against switching to seconds (and keeping 30*60s default logging interval), I'll rediff my commit and push it to devel. The rest of the issues will be filed/dealt separately.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17187801 .

[nokton]

Not sure about - can you check with bunnie what this was specified for.

Pieter

On Apr 30, 2013, at 4:45 AM, elafargue notifications@github.com wrote:

Given the limited r/w cycles of the ST32's flash - we're at least one order of magnitude below dedicated flash chips here and i'm not sure there is any wear levelling built in - , I'd suggest not logging to flash under a minute or more at the very minimum, and use data streaming for external logging if a faster rate is required, as you described above.

On Mon, Apr 29, 2013 at 12:16 PM, Pieter Franken notifications@github.comwrote:

If you're at it - to use the Onyx as a bGeigie, could we have the Onyx log, say every 5 seconds and use a generic datalogger (e.g. iPhone/Andoird generic app) and merge the two data streams into a bGeigie stream (could be a databot@safecast.org function for that matter)

Low tech way to get a bGeigie out of an Onyx.

All we need would be a simple mounting 'sleeve' to put the Onyx on the car window.

Pieter

On Apr 29, 2013, at 9:45 PM, Kalin KOZHUHAROV notifications@github.com wrote:

Well, I did the test with current devel and guess what? I got a log every 30 seconds in sleep mode ;-) So, this means we are not sleeping 30min but 30s... One more reason to move everything into seconds (except date/time display and logging).

Since this issue has grown too big in note very related comments and I haven't heard anybody against switching to seconds (and keeping 30*60s default logging interval), I'll rediff my commit and push it to devel. The rest of the issues will be filed/dealt separately.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17187801 .

— Reply to this email directly or view it on GitHub.

[elafargue]

ST says it's spec'ed at 10k cycles ( http://www.st.com/st-web-ui/static/active/en/resource/technical/document/datasheet/CD00191185.pdf, page 63, Nend).

On Mon, Apr 29, 2013 at 12:47 PM, Pieter Franken notifications@github.comwrote:

Not sure about - can you check with bunnie what this was specified for.

Pieter

On Apr 30, 2013, at 4:45 AM, elafargue notifications@github.com wrote:

Given the limited r/w cycles of the ST32's flash - we're at least one order of magnitude below dedicated flash chips here and i'm not sure there is any wear levelling built in - , I'd suggest not logging to flash under a minute or more at the very minimum, and use data streaming for external logging if a faster rate is required, as you described above.

On Mon, Apr 29, 2013 at 12:16 PM, Pieter Franken notifications@github.comwrote:

If you're at it - to use the Onyx as a bGeigie, could we have the Onyx log, say every 5 seconds and use a generic datalogger (e.g. iPhone/Andoird generic app) and merge the two data streams into a bGeigie stream (could be a databot@safecast.org function for that matter)

Low tech way to get a bGeigie out of an Onyx.

All we need would be a simple mounting 'sleeve' to put the Onyx on the car window.

Pieter

On Apr 29, 2013, at 9:45 PM, Kalin KOZHUHAROV < notifications@github.com> wrote:

Well, I did the test with current devel and guess what? I got a log every 30 seconds in sleep mode ;-) So, this means we are not sleeping 30min but 30s... One more reason to move everything into seconds (except date/time display and logging).

Since this issue has grown too big in note very related comments and I haven't heard anybody against switching to seconds (and keeping 30*60s default logging interval), I'll rediff my commit and push it to devel. The rest of the issues will be filed/dealt separately.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub< https://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17187801> .

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17189569 .

[nokton]

That's a lot of drives.

On Apr 30, 2013, at 4:53 AM, elafargue notifications@github.com wrote:

ST says it's spec'ed at 10k cycles ( http://www.st.com/st-web-ui/static/active/en/resource/technical/document/datasheet/CD00191185.pdf, page 63, Nend).

On Mon, Apr 29, 2013 at 12:47 PM, Pieter Franken notifications@github.comwrote:

Not sure about - can you check with bunnie what this was specified for.

Pieter

On Apr 30, 2013, at 4:45 AM, elafargue notifications@github.com wrote:

Given the limited r/w cycles of the ST32's flash - we're at least one order of magnitude below dedicated flash chips here and i'm not sure there is any wear levelling built in - , I'd suggest not logging to flash under a minute or more at the very minimum, and use data streaming for external logging if a faster rate is required, as you described above.

On Mon, Apr 29, 2013 at 12:16 PM, Pieter Franken notifications@github.comwrote:

If you're at it - to use the Onyx as a bGeigie, could we have the Onyx log, say every 5 seconds and use a generic datalogger (e.g. iPhone/Andoird generic app) and merge the two data streams into a bGeigie stream (could be a databot@safecast.org function for that matter)

Low tech way to get a bGeigie out of an Onyx.

All we need would be a simple mounting 'sleeve' to put the Onyx on the car window.

Pieter

On Apr 29, 2013, at 9:45 PM, Kalin KOZHUHAROV < notifications@github.com> wrote:

Well, I did the test with current devel and guess what? I got a log every 30 seconds in sleep mode ;-) So, this means we are not sleeping 30min but 30s... One more reason to move everything into seconds (except date/time display and logging).

Since this issue has grown too big in note very related comments and I haven't heard anybody against switching to seconds (and keeping 30*60s default logging interval), I'll rediff my commit and push it to devel. The rest of the issues will be filed/dealt separately.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub< https://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17187801> .

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17189569 .

— Reply to this email directly or view it on GitHub.

[elafargue]

It's a bit more complex, since flash is erased by pages, so you end up rewriting the same flash cells multiple times even with small changes. Dedicated storage flash memories use above 100k erase/write cycles and use wear levelling, i.e. try to avoid erasing/writing the same cells all the time - this is transparent.

With 10k cycles, you're 100% fine with upgrading the firmware pretty much as often as you want, but I would have reservations when it comes to implementing fast flash logging, it's just not meant for that. More to the point, if you want to have precise sync with a GPS, you're better off not relying on sync'ing the clock, but do the log merge in a dedicated logger that receives both GPS and Onyx streams...

Does this make sense?

On Mon, Apr 29, 2013 at 1:05 PM, Pieter Franken notifications@github.comwrote:

That's a lot of drives.

On Apr 30, 2013, at 4:53 AM, elafargue notifications@github.com wrote:

ST says it's spec'ed at 10k cycles (

http://www.st.com/st-web-ui/static/active/en/resource/technical/document/datasheet/CD00191185.pdf,

page 63, Nend).

On Mon, Apr 29, 2013 at 12:47 PM, Pieter Franken notifications@github.comwrote:

Not sure about - can you check with bunnie what this was specified for.

Pieter

On Apr 30, 2013, at 4:45 AM, elafargue notifications@github.com wrote:

Given the limited r/w cycles of the ST32's flash - we're at least one order of magnitude below dedicated flash chips here and i'm not sure there is any wear levelling built in - , I'd suggest not logging to flash under a minute or more at the very minimum, and use data streaming for external logging if a faster rate is required, as you described above.

On Mon, Apr 29, 2013 at 12:16 PM, Pieter Franken notifications@github.comwrote:

If you're at it - to use the Onyx as a bGeigie, could we have the Onyx log, say every 5 seconds and use a generic datalogger (e.g. iPhone/Andoird generic app) and merge the two data streams into a bGeigie stream (could be a databot@safecast.org function for that matter)

Low tech way to get a bGeigie out of an Onyx.

All we need would be a simple mounting 'sleeve' to put the Onyx on the car window.

Pieter

On Apr 29, 2013, at 9:45 PM, Kalin KOZHUHAROV < notifications@github.com> wrote:

Well, I did the test with current devel and guess what? I got a log every 30 seconds in sleep mode ;-) So, this means we are not sleeping 30min but 30s... One more reason to move everything into seconds (except date/time display and logging).

Since this issue has grown too big in note very related comments and I haven't heard anybody against switching to seconds (and keeping 30*60s default logging interval), I'll rediff my commit and push it to devel. The rest of the issues will be filed/dealt separately.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub<

https://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17187801>

.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub< https://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17189569> .

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17190651 .

[nokton]

Makes partial sense - I know that we can hook up, but this idea is to not have any need for a cable to the unit low-tech style. A suggestion for people that have a smart phone and Onyx and want to geo log their data with minimal effort.

So for the flash, how many times can you log data to it in real life?

Pieter

On Apr 30, 2013, at 5:14 AM, elafargue notifications@github.com wrote:

It's a bit more complex, since flash is erased by pages, so you end up rewriting the same flash cells multiple times even with small changes. Dedicated storage flash memories use above 100k erase/write cycles and use wear levelling, i.e. try to avoid erasing/writing the same cells all the time - this is transparent.

With 10k cycles, you're 100% fine with upgrading the firmware pretty much as often as you want, but I would have reservations when it comes to implementing fast flash logging, it's just not meant for that. More to the point, if you want to have precise sync with a GPS, you're better off not relying on sync'ing the clock, but do the log merge in a dedicated logger that receives both GPS and Onyx streams...

Does this make sense?

On Mon, Apr 29, 2013 at 1:05 PM, Pieter Franken notifications@github.comwrote:

That's a lot of drives.

On Apr 30, 2013, at 4:53 AM, elafargue notifications@github.com wrote:

ST says it's spec'ed at 10k cycles (

http://www.st.com/st-web-ui/static/active/en/resource/technical/document/datasheet/CD00191185.pdf,

page 63, Nend).

On Mon, Apr 29, 2013 at 12:47 PM, Pieter Franken notifications@github.comwrote:

Not sure about - can you check with bunnie what this was specified for.

Pieter

On Apr 30, 2013, at 4:45 AM, elafargue notifications@github.com wrote:

Given the limited r/w cycles of the ST32's flash - we're at least one order of magnitude below dedicated flash chips here and i'm not sure there is any wear levelling built in - , I'd suggest not logging to flash under a minute or more at the very minimum, and use data streaming for external logging if a faster rate is required, as you described above.

On Mon, Apr 29, 2013 at 12:16 PM, Pieter Franken notifications@github.comwrote:

If you're at it - to use the Onyx as a bGeigie, could we have the Onyx log, say every 5 seconds and use a generic datalogger (e.g. iPhone/Andoird generic app) and merge the two data streams into a bGeigie stream (could be a databot@safecast.org function for that matter)

Low tech way to get a bGeigie out of an Onyx.

All we need would be a simple mounting 'sleeve' to put the Onyx on the car window.

Pieter

On Apr 29, 2013, at 9:45 PM, Kalin KOZHUHAROV < notifications@github.com> wrote:

Well, I did the test with current devel and guess what? I got a log every 30 seconds in sleep mode ;-) So, this means we are not sleeping 30min but 30s... One more reason to move everything into seconds (except date/time display and logging).

Since this issue has grown too big in note very related comments and I haven't heard anybody against switching to seconds (and keeping 30*60s default logging interval), I'll rediff my commit and push it to devel. The rest of the issues will be filed/dealt separately.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub<

https://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17187801>

.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub< https://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17189569> .

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/Safecast/onyxfirmware/issues/25#issuecomment-17190651 .

— Reply to this email directly or view it on GitHub.

[new299]

@nokton we could support a 5s logging interval but it's a substantial rewrite. Basically we can't write to the flash more than once every few mins, so we'd have to add some code to cache log entries in memory. At a 5s logging interval we also can only store about 4hours of data in the STM32 so that's another limitation that needs to be considered. The phone based GPS log merging is a reasonably big project in itself, if someone wants to take this on it's probably best approached on a separate branch as it will obviously involve a fair bit of hacking, and putting together phone apps/scripts/serverside tools to do this.

So, for the moment I don't think we should allow logging intervals of <1min. I think allowing interval input in seconds in fine. I rather this was specified as MM:SS as a UX thing because casual users (who this device is aimed at to some degree) are more likely to understand that.