search

EnviroDIY / ModularSensors

An Arduino library to give environmental sensors a common interface of functions for use with Arduino-framework dataloggers, such as the EnviroDIY Mayfly.
https://envirodiy.github.io/ModularSensors/
Other
79 stars 48 forks source link

Add support for a tipping bucket #84

Closed SRGDamia1 closed 5 years ago

SRGDamia1 commented 6 years ago

Simple reed switch

SRGDamia1 commented 6 years ago

Need a very quick interrupt function to count the number of tips and then add a function in the main "wake up" function to write out the number of tips since the last interrupt.

Do we want to also separately write out exactly the time of the tip as a pendant logger does?

aufdenkampe commented 6 years ago

I need this capability by the end of February. I've been talking to @awickert and @bschulz1701 at UMN and with https://github.com/NorthernWidget, and they've described to me how they did it for their ALog.

@awickert and/or @bschulz1701, could you point us to the code you use for supporting tipping buckets?

SRGDamia1 commented 6 years ago

I have a couple of tipping buckets that I could test this on, but I haven't had any pressing need to do so and I haven't decided exactly how to do it. Partly it's a question of whether to interface directly with a reed switch on the tipper or act as a secondary logger that's pulling results from something else connected directly to the buckets. And then there's the question of recording only a number of tips since the last time interval vs recording the exact time of each individual tip.

bschulz1701 commented 6 years ago

So I did not write the code for the tipping bucket, but the way we do it after looking through is we have the reed switch on a hardware interrupt pin, we then configure this pin to an interrupt function, which just sets a global "new tip" flag. We then put the logger to sleep, and when the logger wakes up (which can be the result of a external timer going off, or of a bucket tip) we check if there has been a tip, if there has we log that in an SD file (along with other measurements if we have that enabled) so we do it based on hardware interrupts and end up logging to the SD for every tip. This is probably the simplest way to go about it.

However, I am not sure of the interrupt pin status on the Mayfly, if hardware interrupts are not available, it would be possible to use a port interrupt (which can be used on any pin) to do the same (which it looks like is already in place in the EnviroDIY github), I have actually done testing on this with PCINT firmware that I wrote, and a rain gauge pulse is slow enough that it is possible to use PCINT type interrupts to wake from sleep, and still catch the pulse (in order to determine that THAT was what triggered the interrupt and the wake up).

Also, as I discussed briefly with Anthony, I have some designs for a counter device which would act as the "secondary logger" as you are describing. Where this device would operate in a very low power state and count pulses and then could be read at set intervals by the main logger. It seems that this does not pay off from a power consumption standpoint (for a single tipping bucket at least) but could be used to simply the software for a rain gauge, and (not to get off topic) but would be very beneficial for power consumption when using an anemometer (which uses the same pulse method, but can easily get to over 100Hz, where as a rain gauge, even in torrential rains, is more like 5Hz)

Hope this is helpful and sorry if it is rambling! Would be happy to talk more about any of this

aufdenkampe commented 6 years ago

@bschulz1701, thanks for that explanation!

A few ideas and questions:

bschulz1701 commented 6 years ago

We do write to the SD with every tip, writing to the SD definitely is an additional power draw, but not that much more than wakeup power. This was mainly done for accuracy I think, in order to get sub-measurement interval rainfall data, but from my standpoint, I don't see any reason you could not just wake up and increment a counter, then read/clear this value at every log interval, this would simply limit your temporal resolution to 1 log cycle, so this might not be enough for some?

Currently this is basically what we do, we record the time of each bucket tip event in a csv file as line item. I think with trying to integrate more real time elements into the system it would pay off to do what you are talking about as well, of having a variable used to hold the rain fall rate (or something analogous to that) for the last logging period

By always on I am assuming you mean that this device constantly records pulses while the logger sleeps and then the logger can wake up at preset intervals and get data from the secondary logger? In this case, not really, at least not with fulfilling the always on requirement. If you took an arduino and set it up to detect pulses, count them, then go back into deep sleep, you could probably get down to an average current of something like 30uA or less assuming not a long or rain (which could run on a watch battery for a while, with a margin of safety, maybe 3 - 6 months), but this would also mean you would need a interrupt pin to be available to the master logger so that it can wake the secondary logger from sleep when it wants to talk (Since the secondly logger will be sleeping most of the time too). Also, electrically speaking, I don't know of any which have a watch battery socket (which is a somewhat trivial thing to add externally, but I know you like to have everything user makeable). To really get the power down on an external counter like this I think you would want to go with a gate level circuit implementation, which is what I have designed mine with, I do not have a hard power number yet (I have not done a hardware test yet), but it should be under 20uA running

SRGDamia1 commented 6 years ago

@aufdenkampe , I don't think the AVR processor can do much of anything while it's in "deep sleep" including increment a counter. A SAMD processor's clock can send an interrupt to another device while the rest of the processor is sleeping, but that's it. Even just incrementing a counter value stored in memory requires the processor memory to be on. And, as @bschulz1701 mentioned, for fast enough waking to catch a tip and know it was a tip, you'd want to use a hardware pin interrupt. (I think pin 10 is the only pin available on the Mayfly.)

To be consistent with all the other data in this library, you definitely do want a single counter value associated with a nice even time interval. The question remains whether or not to stay awake long enough to record the time. I suppose you could create a buffer space in memory for writing times to instead of writing to the SDCard every time, but then you'd have to reserve an unknown amount of memory for storing those values and I'd be afraid of missing tips because that reserved memory filled up or of causing fragmented memory problems. There's no way you'd want to throw exact tip times into the same file where you're recording 5-minute conductivity or something. Parsing a file like that would be a nightmare.

When I was thinking about acting as a secondary logger, I wasn't thinking of trying to write up code for a slave Arduino to both record tips and respond to requests from a master Arduino. I think that would just create hassle. I was thinking about some other lower-power non-arduino as a primary logger.

bschulz1701 commented 6 years ago

I would agree on the increment issue, maybe I was not clear with that, the system would have to be awake for the incrementing process, which would be cause by a wake up due to a hardware interrupt, but then the device could go back to sleep, since the interrupt would be able to wake it up.

And it seems like allocating memory to store that could be tricky, because there is a pretty wide variability, say you write to the SD every 15 minutes, you could have up to 50 tips in that period (with only 2"/hour of rain) and most of the time have none, so it would take some clever dynamic allocation to make sure you don't overrun anything and also have enough region to operate in.

aufdenkampe commented 6 years ago

@SRGDamia1, thanks for those thoughts. I think I did a poor job of explaining some of my ideas. Let me try to clarify.

My thinking on writing to the SD card would be that if we chose to record the time of the tips, then those would get written to a separate file, that would only list tip times. The main data file would only write the number of tips between two time intervals (in a CSV line along with all the other data values) either by reading & analyzing the tip-time file, or by keeping track of tip counts some other way.

My thinking regarding "a way to store the "flag" in memory while in deep sleep" was that the logger would clearly need to wake to record the tip flag (and potentially time), so my question was whether the logger was capable of storing those values from one wake through a sleep to the next wake. It sounds like the answer is yes, but that there could be memory issues.

aufdenkampe commented 6 years ago

@SRGDamia1 & @bschulz1701, is this then the simplest way to go?:

Is that the way to go?

bschulz1701 commented 6 years ago

Yes, I think that would be the easiest implementation. The main improvement to this would be storing the tip information in RAM as opposed to SD flash to reduce power consumption, but this could get hairy, so probably best to start with SD and work back form there.

awickert commented 6 years ago

@aufdenkampe Your method seems a good way to go, for me. We don't yet do your third bullet point, but it should not be too hard to include it. An unsigned integer is small in memory and would hold all of the needed information.

On a related/unrelated note, I have been thinking of working on the problem of how small rainfall rate bins should be to properly average over tips. This should have something to do with the PDF of rainfall.

I wrote the code to log tipping-bucket rain gauges, and can answer any details about it. But it is pretty simple and it seems you all understand it.

SRGDamia1 commented 6 years ago

Yes, that's the way I'd do it. WRT making a component in this library, you'd want a sensor constructor that requires the interrupt pin and probably English vs metric for the tip increments. Then you'd also want to give options for an additional pin for the SD card chip select and the file name/name header. The setup function would have to assign the interrupt and start the extra file. Then you need to write the interrupt function. Most of the other normal sensor functions (sleep/wake, etc) would be blank. The take single measurement would write out the added counter value. The variables would be accumulated tip count, calculated rainfall, and possibly rain rate standardized to per hour.

I'd have to think about how the counting interrupt might interact with the main wake - up for logging. It could get weird if something happens like a tip occurs while the logger is already writing to the other SD card. If the tip occurs while the processor is communicating with an sdi-12 sensor or SoftwareSerial connected sensor, the tip would be completely missed because those libraries both turn off interrupts for the duration of communication. (SDI12 is actually worse than SoftwareSerial because of the super slow baud rate. I keep thinking about ways to improve that, but it's really low on my to-do list.)

Hmm..

Yeah, for purposes of including a tipping bucket in this library, it would be much easier to implement the tips only as an accumulated count number. Then, if you really want exact times of rainfall, just use a separate logger whose only job is to record tip times to a card. The two loggers don't even need to talk to each other, just both have connections to the same tipper.

On Feb 2, 2018 4:29 PM, "Anthony Aufdenkampe" notifications@github.com wrote:

Yes, I think that would be the easiest implementation. The main improvement to this would be storing the tip information in RAM as opposed to SD flash to reduce power consumption, but this could get hairy, so probably best to start with SD and work back form there.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/EnviroDIY/ModularSensors/issues/84#issuecomment-362758432, or mute the thread https://github.com/notifications/unsubscribe-auth/AFIBjuvlNk_YF0nXAKW57_M9Ka7P31gWks5tQ7LRgaJpZM4RFrj1 .

SRGDamia1 commented 6 years ago

Oh, also, WRT storing values in memory during deep sleep, the arduino can retain anything when it goes back to sleep, that isn't the worry at all. The problem is that to remember each individual time, you have to either guess in advance how many times you are going to have to remember and save spaces in memory for that many values or try to dynamically allocate new space when you need it and de-allocate it later. Everything I've read says Arduinos are really bad at memory clean up, so you're likely to crash the logger after a while that way. Just adding to a count is easy. You know from the start that you're going to need just one space in memory for that single number, even if you don't know what the number is. If you want to be really, really safe, you could even allocate 32 bits of space to hold that number, though I would argue that if you're getting 4,294,967,295 tips between logging intervals you might want to decrease your interval a little.

On Feb 2, 2018 8:32 PM, "Sara Damiano" sdamiano@stroudcenter.org wrote:

Yes, that's the way I'd do it. WRT making a component in this library, you'd want a sensor constructor that requires the interrupt pin and probably English vs metric for the tip increments. Then you'd also want to give options for an additional pin for the SD card chip select and the file name/name header. The setup function would have to assign the interrupt and start the extra file. Then you need to write the interrupt function. Most of the other normal sensor functions (sleep/wake, etc) would be blank. The take single measurement would write out the added counter value. The variables would be accumulated tip count, calculated rainfall, and possibly rain rate standardized to per hour.

I'd have to think about how the counting interrupt might interact with the main wake - up for logging. It could get weird if something happens like a tip occurs while the logger is already writing to the other SD card. If the tip occurs while the processor is communicating with an sdi-12 sensor or SoftwareSerial connected sensor, the tip would be completely missed because those libraries both turn off interrupts for the duration of communication. (SDI12 is actually worse than SoftwareSerial because of the super slow baud rate. I keep thinking about ways to improve that, but it's really low on my to-do list.)

Hmm..

Yeah, for purposes of including a tipping bucket in this library, it would be much easier to implement the tips only as an accumulated count number. Then, if you really want exact times of rainfall, just use a separate logger whose only job is to record tip times to a card. The two loggers don't even need to talk to each other, just both have connections to the same tipper.

On Feb 2, 2018 4:29 PM, "Anthony Aufdenkampe" notifications@github.com wrote:

Yes, I think that would be the easiest implementation. The main improvement to this would be storing the tip information in RAM as opposed to SD flash to reduce power consumption, but this could get hairy, so probably best to start with SD and work back form there.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/EnviroDIY/ModularSensors/issues/84#issuecomment-362758432, or mute the thread https://github.com/notifications/unsubscribe-auth/AFIBjuvlNk_YF0nXAKW57_M9Ka7P31gWks5tQ7LRgaJpZM4RFrj1 .

awickert commented 6 years ago

@SRGDamia1 I wrote code to check which interrupts might always be active and to, for example, store information on a tip until after a logging cycle if the logger is busy at the time. We can see how good this is, https://github.com/NorthernWidget/ALog

SRGDamia1 commented 6 years ago

That sounds useful! Thanks!

On Feb 2, 2018 8:52 PM, "Andy Wickert" notifications@github.com wrote:

@SRGDamia1 https://github.com/srgdamia1 I wrote code to check which interrupts might always be active and to, for example, store information on a tip until after a logging cycle if the logger is busy at the time. We can see how good this is, GitHub.com/NorthernWidget/ALog

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/EnviroDIY/ModularSensors/issues/84#issuecomment-362764029, or mute the thread https://github.com/notifications/unsubscribe-auth/AFIBjtczO6_oGTuCKYEpJOXB2bPs5iMuks5tQ7v6gaJpZM4RFrj1 .

aufdenkampe commented 6 years ago

@SRGDamia1, is there any chance you could work on this in the next week or two?

SRGDamia1 commented 6 years ago

You'd need to talk to Dave Arscott. Right now we don't have any need for new rain gauge stations so I don't have a budget to work on this with. The same applies to the other sensor requests.

aufdenkampe commented 6 years ago

@SRGDamia1, thanks for the update. I was planning on doing the other sensors, but I wasn't sure if this one was on your to do list for your own reasons.

aufdenkampe commented 6 years ago

@bschulz1701, I just created this repository for you to use for your initial sketches for the Mayfly: https://github.com/EnviroDIY/TippingBucketRainGauge

Let's start by just pasting in (or linking to) what you are already doing for the ALog, then work from there.

The question for me is whether we use this repo to build stand-alone code, or something that leverages what already exists in ModularSensors (or both). @SRGDamia1, ideas or suggestions?

aufdenkampe commented 6 years ago

@bschulz1701, from this post on the Mayfly (https://envirodiy.org/features-of-the-new-mayfly-v0-5/), I just noticed this, which is relevant to our plan to use D10 to record tips:

An onboard 1M-ohm resistor has been added to the board for pulling up digital pin D10 to 3.3v for applications where you want to use D10 as a hardware interrupt (D10 is one of the 3 hardware interrupt pins on the Mayfly, in addition to D2 and D3, which are usually used for the Xbee coms). The default position is OFF. Add a solder bridge to SJ12 if you want to connect D10 to the onboard pullup resistor. DO NOT use this jumper if you have modified SJ1 to use D10 as the RTC interrupt. Also, DO NOT use this jumper if you are using D10 for any other type of input besides something that needs the pullup. For more information about the solder jumpers, be sure to read this page: https://envirodiy.org/mayfly/hardware/jumper-settings/

bschulz1701 commented 6 years ago

Thanks for pointing this out, we were actually planning on using D11 (INT1) for the tipping bucket as I recall, D10 would work just fine as well, and is on the same grove port, but it is a likely candidate to be used for a hardware interrupt for the RTC, so I figured we could use D11, which so far I have not found any conflicting usage

aufdenkampe commented 6 years ago

@bschulz1701, I'm a bit confused, because the idea of using D11 seems in conflict with that statement:

(D10 is one of the 3 hardware interrupt pins on the Mayfly, in addition to D2 and D3, which are usually used for the Xbee coms)

My read of the schematic for the v0.5 (https://github.com/EnviroDIY/EnviroDIY_Mayfly_Logger/blob/master/hardware/mayfly_v0p5p_schematic.png and also at https://envirodiy.org/mayfly/hardware/), definitely shows that the RTC uses D10, but does that not then use up all of our external interrupts. I see that:

bschulz1701 commented 6 years ago

Ah, I see, I was using a different bootloader pinout description, rather than just reading off of the pin labels on the schematic, and they allocate digital pin values to different port pins for some reason. In this case using D10 should still be fine and we can use that jumper for the pullup, as long as we either do not need interrupts from the RTC for this application, or can use A7 as the interrupt pin for the RTC. Not sure if this is the case?

aufdenkampe commented 6 years ago

We definitely need the interrupts from the RTC for anything in ModularSensors to work, to wake the system from sleep at set intervals (typically every 5 or 10 minutes), then collect, record and transmit data. So we don't want to interfere with that in any way.

For our first implementation of this in ModularSensors, let's just focus on storing tips in memory as an accumulated count number, then reading that when the logger wakes on the set interval to record and transmit data, as described in https://github.com/EnviroDIY/ModularSensors/issues/84#issuecomment-362761279

bschulz1701 commented 6 years ago

If we want to do that, we would still need to use some interrupt to wake the logger from sleep and increment the counter in memory, which brings up the issues mentioned in the comment, interrupts which are asynchronous to logging events could be missed due to shutting off interrupts during some of those operations, also vector interrupts would need to be used if we can't use a hardware interrupt, which could have the potential of causing issues with alt software serial and others which use this.

I wonder if a better solution (at least for now, given that there could be a lot of dependencies to deal with) is to use a secondary logger (as @SRGDamia1 ) eludes to, like a pro-mini or Trinket board, and have them running all the time, or some amount of sleep, then using a dedicated interrupt to read tipping events, and have the main Mayfly talk to this board over I2C to ask for the number of tips during each log event. Power consumption would be slightly increased, but I think both simplicity and robustness would be improved. Simply have a grove connector going to the board, which is wired directly to the output of the tipping bucket. What do you think of this?

aufdenkampe commented 6 years ago

I am very open to having a separate board dedicated to just listening to the tipping bucket, IF it doesn't require a separate solar power system (i.e. can it run on the Mayfly's non-switched power or an independent battery with a 5 y life).

On the other hand, think that we still have the potential to use a pin change interrupt vector without conflicting with other libraries, if it would be fast enough to catch a tip while the system is in sleep. This is because:

@SRGDamia1 and @s-hicks2, what do you think about using a pin change interrupt to record bucket tips?

bschulz1701 commented 6 years ago

I think it would definitely be low enough power to not need that, on the order of mAs, which could run off of the Mayflys power. And then it looks like we would have to use a pin in vector A to fully avoid any conflicts, which we can do, we would just have to plug into the main headers instead of the groves. It does not solve the problem of interrupts being disabled by several of the other libraries, such as SDI-12, which could mean losing data.

aufdenkampe commented 6 years ago

For the record, we decided to develop an external tip counter device, using the Adafruit Pro Trinket continuously powered off the Mayfly and acting as an I2C slave. The code for the this Tip Counting Pro Trinket is here: https://github.com/EnviroDIY/TippingBucketRainGauge

SRGDamia1 commented 6 years ago

Sorry, I was out Thursday and Friday for Easter.

WRT the pin numbers, the Mayfly and the Mbili it was modeled on differ from almost every other AtMega 1280p board. I think it's vectors B and D that are reversed.

WRT interrupt conflicts, external interrupts are different from pin-change interrupts and use completely separate interrupt vectors, so you shouldn't have conflicts with libraries that use pin-change interrupts (like SoftwareSerial/SDI-12/Servo/etc). The interrupt vector to use pin D10 as an external interrupt is INT2_vect, vector A is for pin change interrupts. AltSoftSerial uses timer interrupts, another different type of interrupt vector.

That being said, I'd just use GreyGnome's "EnableInterrupt" library to control the tipping bucket. The library is smart enough to pick the right vector type (external vs pin change) for any given pin, so you wouldn't need to do any of the heavy lifting wrt assigning vectors and making any sketches applicable to a wider audience. The ModularSensors library is already using EnableInterrupt to handle the clock interrupts, so it won't conflict with anything there.

Setting up I2C communication between two loggers sound harder than anything I would do. I'd just make the two loggers completely and totally independent, except for being connected to the same reed switch.

If including the tipping bucket as a module in ModularSensors, I'd use EnableInterrupt with "NeedForSpeed" and only count the number of interrupts. Then just output that number in the GetSingleMeasurementResults fxn.

For the other logger, I'd still use EnableInterrupt, but with a full interrupt function that includes writing the current time to an SD card.

aufdenkampe commented 6 years ago

I talked to Sara yesterday, and she suggests we work off the head of the "timing" branch. I just created that new branch. https://github.com/EnviroDIY/ModularSensors/tree/battery_and_bucket. Let's put all new code and documents everything in there.

aufdenkampe commented 6 years ago

@bschulz1701, I like @SRGDamia1's suggestion I https://github.com/EnviroDIY/ModularSensors/pull/114#issuecomment-380209239 to:

add a "multiplier" input for using with a voltage divider. That makes it more clear and more general.

Could you do that as you clean up that code? We can add documentation that describes what those multipliers should be for the two different gain switches for the Seeed Grove Voltage Divider that we're using.

aufdenkampe commented 5 years ago

This should have been closed in May of 2018 or so, when #116 was closed!