Feature request - flow meter add in

[mattip]

Thanks for this wonderful application.

I have a Seed Studio flow meter http://www.seeedstudio.com/depot/G12-Water-Flow-Sensor-p-635.html hooked into my raspberry pi.

I am using wiringpi like the code posted here https://www.raspberrypi.org/forums/viewtopic.php?f=37&t=94656 to set up an interrupt handler and increment a counter. I see how I can write this as a so and read the counter from Python. I would like to get feedback in SIP about overall flow rate and expected vs actual flow rate (to detect leaks). What would be a strategy to get this into SIP? I see the way the clock time is updated, should I hook into that and update another div under it?

Note that I have not calibrated the flow meter, from some preliminary tests it seems the meter peaks out at about 100 pulses per second, so this might not be an accurate way to measure water quantity rather a qualitative go-no go check of the system, an arduino and maybe a better meter may be needed to get accurate measurements, but that is a different issue.

[Dan-in-CA]

Welcome, Thanks for your interest in the SIP project.

There is a user in Chile who is currently refactoring some of the SIP code to make it easier to interface various hardware with the Raspi and SIP. He is also planning on using flow meters in his system. It would be a good idea if you could post your feature request on the SIP discussion forum: http://nosack.com/sipforum/index.php

I will contact Matias as I think he may be interested in working with you on this. There may also be others who would have an interest in this.

[bkoblenz]

If you use pigpio as the gpio interface, then you can set up a callback function (called wind_cb) below and that callback function has a tally method so you dont have to worry interrupt handling etc.

Just a couple of lines of code in your loop taking samples. (Assumes there was an initial call to set wind_cb prior to the loop.

                    flow = wind_cb.tally()
                    wind_cb = pi.callback(gv.pin_map[3],

pigpio.FALLING_EDGE)

Tally gives you the number of "clicks" since the last time wind_cb was defined.

On Sat, Jan 23, 2016 at 1:11 AM, Matti Picus notifications@github.com wrote:

Thanks for this wonderful application.

I have a Seed Studio flow meter http://www.seeedstudio.com/depot/G12-Water-Flow-Sensor-p-635.html hooked into my raspberry pi.

I am using wiringpi like the code posted here https://www.raspberrypi.org/forums/viewtopic.php?f=37&t=94656 to set up an interrupt handler and increment a counter. I see how I can write this as a so and read the counter from Python. I would like to get feedback in SIP about overall flow rate and expected vs actual flow rate (to detect leaks). What would be a strategy to get this into SIP? I see the way the clock time is updated, should I hook into that and update another div under it?

Note that I have not calibrated the flow meter, from some preliminary tests it seems the meter peaks out at about 100 pulses per second, so this might not be an accurate way to measure water quantity rather a qualitative go-no go check of the system, an arduino and maybe a better meter may be needed to get accurate measurements, but that is a different issue.

— Reply to this email directly or view it on GitHub https://github.com/Dan-in-CA/SIP/issues/143.

[Pelado-Mat]

I plan to add a pressure sensor to do a go-no-go check, as my irrigation water comes from a pump and a cistern I must be sure that the pump is actually moving water and not over-pressure the system.

When we finish the work with Dan, I plan to modify the pressure sensor plugin to log and plot the pressure someway, also to stop all the stations if the pressure drops too much or get to high.

A Flow meter plug in could work in a similar way, it should be easy modification of the pressure adj plugin using bkoblenz code.

[shrakrak]

Has anyone eventually got around to writing a flow meter plugin. i am also interested in this feature and i wanted to check if something is available before i start my own solution.

BTW - i think a flow meter should be part of the core application like the rain sensor. The reason is that it can effect many system functions like logging, active display of flow, accounting. it can also be used to detect solenoid malfunction and even detect leakage/clogging per station (comparing actual station flow Vs expected or historical flow data)

any opinions ?

[DaveSprague]

For what it's worth, three years ago I grabbed a copy of what was called OSPi, at the time, and hacked in support for reading data from an arduino board that interfaces to three (or more) of the Seeed Studio flow sensors. I was in a hurry at the time and it's worked so well and reliability that I haven't taken the time to merge this into the newer versions.

I agree that a flow meter touches several subsystems of the application so it's maybe be best to build support into the core app.

I'm happy to provide more details if that would be useful. Here's an overview:

1) the Arduino code, which I'm happy to share if it's useful, keeps track of multiple pulse counters for each flow sensors. Each time the RPi code reads a counter, via a USB serial interface, it resets that counter back to zero.

2) The RPi code reads the first counter for each flow sensor every few seconds to provide a real-time flow rate display. I use the second counter for each flow meter to track the flow during a program run (by reading that meter's counter it at the start of a program to reset it, the reading the total number of pulses after the program completes to determine the total flow). I then use a third counter to track the amount of flow through each flow sensor over a twenty-four hour period to check if any of the irrigation system's valves is not closing properly.

3) I added the usage info the the log and also created a separate page to view the current flow rate of each channel. At the time, I didn't have enough knowledge or time to integrate that into the primary display pages of the app.

I don't do any automatic checking of usage versus history or automatically checking a bad valve but I agree those would be great features to have. I do email the recently log entries to myself once day to so that when we're away we can check that things seem to be working okay.

The reason I implemented multiple counters per flow meter is that I wasn't sure if the pulse count that was read every few 2-3 seconds for the flow rate display could be accumulated accurately enough over a longer period to get the total flow. Looking at the usage per day though, there's enough unexplained variation in usage for a given program that it' s not clear that this is necessary. I'll try to provide some more quantitive info on the amount of variation in usage from day to day for a particular program.

Another advantage of having multiple counters per flow meter is that it kept the RPi code somewhat simpler since I could devote a separate counter to each function (real-time flow, program usage, and valve malfunction / leak-detection).

I briefly considered implementing the pulse counting on the RPi using pigpio or similar library but at the time decided that using an Arduino to maintain the pulse counts was simpler for me -- particularly since I planned to maintain multiple pulse counters per flow meter.

Dave

[Dan-in-CA]

Hi Dave, This sounds very useful. I would be interested in more details. You can post info on the SIP forum at: http://nosack.com/sipforum/index.php or email me at: dkimberling59@gmail.com or post code on GitHub and provide a link.

Thanks. Dan

[jakkuvaavinash]

the exact code for the pulse counting was not described in the fallowing so please get a valid code for water flow meter YF-S201 Hall Effect Water Flow Meter / Sensor so that will help to my project please refer me as soon as possible thanking you

NOTE : PYTHON PROGRAMMING FOR RPI

avinashvera@gmail.com 8985797658 please contact me as soon as possible

[Dan-in-CA]

I am not sure what code source you are referring to. Can you please provide more detail?

[Dan-in-CA]

Also, the YF-S201 flow meter requires 5V to work. The GPIO pins on the Raspberry Pi are 3.3V max. You will need to connect the flow meter to something like an Arduino.

[DaveSprague]

Following along from my previous post in this thread from back in April, here's the Arduino code I use to count pulses from the flow meters. It uses a library called from here: http://playground.arduino.cc/Main/PinChangeInt, although I used a version of this from 2013 so I'm not certain it will work with the current version without modifications.

This code is written for three flow sensors -- one for each valve in my system. I maintain four counters for each flow meter so that I could accumulate counts over different time durations. I did this because I wasn't sure if I read the sensors every few seconds and accumulated those values in the ospi python code would be accurate for looking at total flow over the course of a few hours. It also makes the python code cleaner in cases where I wanted to keep logs of the total flow during a ospi program even or over a 24 hour period for each channel in addition to the flow during a particular scheduled program even.

I'm not sure this approach is necessary given the accuracy of the flow sensors but I haven't done any analysis of this yet.

I'll send the python code along some time soon, once I've had a chance to relearn it again and put a bit of documentation together. It was based on a version of the ospi code from back in 2013.

` // PinChangeIntExample, version 1.1 Sun Jan 15 06:24:19 CST 2012 // See original example for more details on defines, etc //#define NO_PORTB_PINCHANGES // to indicate that port b will not be used for pi n change interrupts

define NO_PORTC_PINCHANGES // to indicate that port c will not be used for pin

change interrupts

define NO_PORTD_PINCHANGES // to indicate that port d will not be used for pin

change interrupts

// #define DISABLE_PCINT_MULTI_SERVICE

include

int p1 = 8; // digital pins 8, 9, and 10 are all on Port B on the Uno int p2 = 9; int p3 = 10; int led = 13;

// There are four counters maintained for each of three flow sensors. // This allows for keeping track of the total flow over different timeframes // e.g. 5 seconds, 1 minute, full program run, 24 hour period long interrupt_count[4][3] = {{0L, 0L, 0L}, {0L, 0L, 0L}, {0L, 0L, 0L}, {0L, 0L, 0L} }; long unsigned startTime[4]; long elapsedTime; boolean LED_State = HIGH;

int inByte = 0;

void quicfunc1() { digitalWrite(led, HIGH); interrupt_count[0][0]++; interrupt_count[1][0]++; interrupt_count[2][0]++; interrupt_count[3][0]++; };

void quicfunc2() { digitalWrite(led, HIGH); interrupt_count[0][1]++; interrupt_count[1][1]++; interrupt_count[2][1]++; interrupt_count[3][1]++; };

void quicfunc3() { digitalWrite(led, HIGH); interrupt_count[0][2]++; interrupt_count[1][2]++; interrupt_count[2][2]++; interrupt_count[3][2]++; };

long flow_milliliters(long pulse_count, long milliseconds) { if ((pulse_count 1000 / milliseconds) > 1000) { return -1; } long milliliters = pulse_count 1000L 2L / ( 11L 60L); return milliliters; }

void establishContact() { while (Serial.available() <= 0) { Serial.println("0,0,0,"); // send an initial string delay(300); } }

void setup() { pinMode(led, OUTPUT); digitalWrite(led, LOW);

pinMode(p1, INPUT); PCintPort::attachInterrupt(p1, &quicfunc1, FALLING); // add more attachInterr upt code as required

pinMode(p2, INPUT); PCintPort::attachInterrupt(p2, &quicfunc2, FALLING);

pinMode(p3, INPUT); PCintPort::attachInterrupt(p3, &quicfunc3, FALLING);

startTime[0] = millis(); startTime[1] = millis(); startTime[2] = millis(); startTime[3] = millis(); interrupts(); Serial.begin(9600); // while (!Serial) { // ; // for Leonardo board // } // establishContact(); }

void loop() { digitalWrite(led, LOW); long count; if (Serial.available() > 0) { // get incoming string and convert to an integer for which counters are requested digitalWrite(led, HIGH); char c = Serial.read(); int freq = c - '0'; int req = freq; int rst = 1;

// bits 0-2 determine which of four counters is requested.
// if bit 3 is low (default) we'll reset each of the four counters when
//   reading them, otherwise (if bit 3 is high) we don't reset the counters
if (freq >= 8) {
  rst = 0;
  req = freq - 8;
}

elapsedTime = millis() - startTime[req];
String out_str = "";
noInterrupts();
// read the flow amount (in milliliters since the counter was last reset)
//  for each of three flow sensors
for (int i = 0; i < 3; i++) {
  count = interrupt_count[req][i];
  if (rst == 1) {
    interrupt_count[req][i] = 0;
  }
  String flow = String(flow_milliliters(count, elapsedTime), DEC);
  out_str += flow;
  out_str += ", ";
}

if (rst == 1) {
  startTime[req] = millis();
}
interrupts();

// the output string are the values of the total flow for each flow
//  sensors since this level of counter was reset, followed by the
//  elapsed time since the flow sensors was reset.
//  The serial input value as well as it's decoded values (the requested
//  counter and the Reset flag) are also returned for debugging purposes.
out_str += String(elapsedTime, DEC);
out_str += ", ";
out_str += String(freq) + ", " + String(req) + ", " + String(rst);
Serial.println(out_str);
digitalWrite(led, LOW);

} } `

[Pelado-Mat]

I still plan to add flow sensors to my installation, but Im busy with other stuff right now.

My two cents: To get the arduino to connect to SIP, you could use a plugin as I wrote for the pump_control.

I connect the arduino to the pi via i2c and get the sensor readings of a pressure transducer (among other things). See the plugin and arduino code from: https://github.com/Dan-in-CA/sip_plugins/tree/master/pump_control

If you decide to go for that route:

• The i2c code on the arduino side was kind of tricky to get right, specially when you have to send floats or ints via i2c, after reading tons of documents I came across https://github.com/MikeOchtman/Pi_Arduino_I2C that point me to the right direction
• If you power the arduino from any other source.... connect the GND of the ardunio to de GND of the PI. (I fall into that....)

BTW, every time I start thinking about this I feel that SIP do not have the tools to really take advantage of sensors readings. I imagine some nice graph where I could see the different stations running overleaped with my sensors readings (pressure and/or water flow, etc). One path could be add some infrastructure to SIP to make that kind of graphs, using something like http://www.flotcharts.org/ (way over my current programming skills) Other way could be using @DanielCasner mtqq plugin to connect SIP to something like Domoticz, or PiDome and get the flow meters data into that with mySensors

[DaveSprague]

I'm doing a first cut adding Flow Sensor support, based on a simplified version of the implementation I described above and ported to the current code base. The initial version will support flow sensors connected to an Arduino and accessed via USB serial. It will show the real-time flow rate and flow amount while programs (including Run Once Programs) run and I plan to have it add total water usage to the log for each program run.

My plan is to release an early, rough version in the next several days.

I have included a stub that simulates flow sensors so that others can try it out prior to actually hooking up sensors (it just generates some pulse count values rather than accessing an Arduino). I'm also trying to make it general enough that it would be fairly easy to connect flow sensors to directly to the RPi.

My assumption for this first version is that there's a flow sensor for each valve. My own system only has three valves so that's how I've implemented it. For larger systems it may make sense to have one shared flow sensor on the supply line that's shared by multiple valves or some other configuration like that. There should be a fairly simple way to support this type of configuration with some settings options.

One thing I noticed is that the real-time update that's used to show elapsed time when running programs isn't used during Manual operation of the valves so we'd need to somehow add that into the Manual mode in order to be able to see flow rates/amounts when running in Manual mode (which I think would be quite useful).

I agree with one of the comments above that graphing would be useful. My thought is that the webpage associated with the flow_sensor plugin would be a place to show these graphs. It would access log data and ideally be able to show tables and graphs of water usage over different time periods (weekly, monthly, yearly?).

Another feature I'd like to implement is a way to look for stuck or leaking valves by watching for water flow during times when there shouldn't be any.

Let me know your thoughts on what features/configurations would be useful.

[Dan-in-CA]

This sounds really cool! One question I have is how to handle sensors that are some distance from the pi. I have been working with a USB to RS485 serial adapter at the Pi and an RS485 to RS232 adapter attached to an arduuino micro. The micro has a second hardware serial port which makes this easy. RS485 serial can work up to 4000 feet. Of course a lot of folks like to use wireless and since the latest Pi 3s have built-in wifi it should not be too hard.

Regarding data graphing, I have been testing chart.js. and have been able to make a plugin page with nice graphs. There is an intro to chart.js with some examples here: https://www.sitepoint.com/introduction-chart-js-2-0-six-examples/

Looking forward to seeing what you have put together.

Dan

[DaveSprague]

That charting package looks great.

Good question about remote sensors. I really haven't given that any thought so far. In my installation, I have my flow sensors installed right after each valve and since the PI (with the OSPi board, in my case) need to be fairly close to the valves then I assumed the flow sensors would also be close -- although the valves solenoids can probably be driven over a longer distance than the return signals form the flow sensors can travel. So yes, I would think we could/should accommodate both RS485 and WiFi connected Arduinos.

Do you power the Arduino over the RS485 connection or just provide power to it locally?

Just finished writing a simple version of the Arduino code that maintains pulse counters for up to 8 flow sensors. It could be expanded beyond that number fairly simply. I'm using the EnableInterrupts library (https://github.com/GreyGnome/EnableInterrupt) which which can use up to 18 inputs on an Arduino Uno.

If you're trying to accurately monitor the flow from several valves/lines at once, this code may not be fast enough so if that's an important issue, we may need to use the HiSpeed mode of the EnableInterrupt library or write some other custom Arduino code.

My current implementation has the Arduino code just count pulses for each flow sensor and allows all the counters to be reset at once. The conversion to flow rate in Gallons or Liters per second and total flow amounts in Gallons or Liters is done in the flow_sensor plugin. I partitioned it this way since I thought we might want to support flow sensors connected directly to the Pi, using an interrupt routine to count the pulses. In some ways it would be cleaner to have those conversion details encapsulated in the Arduino code but at this point, that's not how I've structured the code.

As you mentioned above, the flow sensors are designed to run at 5v so maybe the directly interface to the Pi isn't something we really need to support?

My testing is using the 1/2" and 3/4" plastic flow sensors similar to this one: http://wiki.seeedstudio.com/wiki/G1/2_Water_Flow_sensor

Dave

[Dan-in-CA]

At this point I have been connecting to the arduinos over standard 18guage sprinkler wire. I use 2 wires for the serial communication and 2 wires carrying 12V DC to power the arduinos. I have been using Arduino micros. They are small and have 2 hardware serial ports. They are soldered to an Adafruit 1/2 size perma-proto board along with the RS485 to RS232 adapter and the whole thing fits into a 1 gang weather proof junction box which makes a nice inexpensive water tight enclosure.

Of course there are larger weather tight junction boxes that could b used for the Arduino Uno.

I have one of the 1/2 inch plastic flow sensors that I can use for testing.

This plugin will be extremely useful for use with drip irrigation systems as it will enable the user to determine when the drip emitters are getting clogged.

I will look into adding the real-time update for the manual mode.

[DaveSprague]

Hi Dan,

That setup looks great. I'll have to pick up an ardiuno micro to try out. Great idea using the junction boxes.

I'll send you the current version tomorrow. What's the best way to give you the code -- I can do a pull request but I can also just send you a zip file of the full SIP directory if that's easier. The changes from your current version on github are small changes in the following files: helpers.py, templates/home.html, templates/log.html, and webpages.py and then of course the addition of the plugins/flow_sensors.py file. Also included will be the Arduino .ino file.

You'll probably want to change the style parameters of the Stations List table so that the flow and total water usage values fit better. I couldn't locate where the width of the columns of that table are set. Right now, the flow rate and amounts, with units, are just tacked on to the time that counts down each second and so they wrap around to a second line, increasing the height of that table row while the program is running.

I'm only updating the instantaneous flow rate and usage values every three seconds in hopes of getting somewhat more accurate flow rate numbers but that's a parameter you can set within the plugin file.

I have not yet add any Options in to support the flow sensors. For now, those are just set up near the top of the flow_sensor plugin file (e.g. the type of flow sensor being used, whether to give rates and amounts in Liters or Gallons, etc). I'm also at this point just supporting a direct mapping from the eight flow sensor counters running on the Arduino and the first eight valves.

Dave

On Tue, Apr 11, 2017 at 9:58 AM, Dan notifications@github.com wrote:

At this point I have been connecting to the arduinos over standard 18guage sprinkler wire. I use 2 wires for the serial communication and 2 wires carrying 12V DC to power the arduinos. I have been using Arduino micros. They are small and have 2 hardware serial ports. They are soldered to an Adafruit 1/2 size perma-proto board along with the RS485 to RS232 adapter and the whole thing fits into a 1 gang weather proof junction box which makes a nice inexpensive water tight enclosure. [image: sensor_box] https://cloud.githubusercontent.com/assets/4706639/24920649/d35e5c32-1e9c-11e7-98fb-01ffa6c8432c.jpg Of course there are larger weather tight junction boxes that could b used for the Arduino Uno.

I have one of the 1/2 inch plastic flow sensors that I can use for testing.

I will look into adding the real-time update for the manual mode.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Dan-in-CA/SIP/issues/143#issuecomment-293327706, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYZCoxgEtFn3pDEw4jeriWmGK0LW7Wwks5ru7EdgaJpZM4HK466 .

[Dan-in-CA]

If you let me know when the SIP code is ready I can clone your fork from GitHub. That way I can pull updates as things progress and you won't need to do anything special. You can either send a zip of the arduino code or post it to a gist here in GitHub. Just click Gist at the very top of your repo page.

I should mention that the Arduino micro has been discontinued by Adafruit who helped design it but it is being manufactured in Italy and is available from Newark Element 14.

[DaveSprague]

My branch of SIP with the flow_sensor support is here: https://github.com/DaveSprague/SIP

The Arduino code is in a public Gist here: https://gist.github.com/DaveSprague/88c2380297e0ae53605ac360299f1b6b

Near the top of the flow_sensors.py plugin you can set the type of sensor you're using (only 1/2" and 3/4" plastic Seeed types are supported now) and whether you want the units in Liters or Gallons.

It currently defaults to flow sensors connect to an Arudino that's connected to the Pi via the USB serial interface. You may need to change the /dev/tty??? name in the flow_sensors.py plugin code to connect to a different serial interface.

Also, near the top of the plugin you can change from using Arduino connected flow sensors to using a very basic simulated flow sensors feature so that the program can be tested even if hardware sensors aren't connected yet.

When you run a Program or Run-Once program you'll see the flow rate and total flow amounts for each station updated every 3 seconds on the Home page. It also adds the water usage amounts for each Program or Run-Once program to the log file.

Doesn't currently show any flow information for Manual Mode yet.

I've left in a bunch of print statements I used for debugging for now so you can use them to identify any issues.

Let me know how it goes and by all means give me feedback on any code issues or functionality issues.

This is my first open source code contribution!

Dave

[Dan-in-CA]

OK. I have cloned your fork, copied the Arduino code and downloaded EnableInterrupt to my arduino library. The Arduino code compiles and I loaded it onto an Arduino Micro that I have on a breadboard.

I needed to install pyserial on the Raspi and the plugin is loading properly. I connected the arduino to the Pi with a USB cable and am getting:

Writing to Arduino
serial input from Arduino is: 0,0,0,0,0,0,0,0

So it looks like they are communicating but I have not connected the flow sensor yet. I have not taken a close look at the Python code yet but this looks like a great start.

[DaveSprague]

Yes looks good so far. Those eight zeros are the counter values for up to eight flow sensors. You can see in the arduino code the pins I currently have set for the flow sensors. As I recall, it's pins 5-12

Dave On Wed, Apr 12, 2017 at 11:32 AM Dan notifications@github.com wrote:

OK. I have cloned your fork, copied the Arduino code and downloaded EnableInterrupt to my arduino library. The Arduino code compiles and I loaded it onto an Arduino Micro that I have on a breadboard.

I needed to install pyserial on the Raspi and the plugin is loading properly. I connected the arduino to the Pi with a USB cable and am getting:

Writing to Arduino serial input from Arduino is: 0,0,0,0,0,0,0,0

So it looks like they are communicating but I have not connected the flow sensor yet. I have not taken a close look at the Python code yet but this looks like a great start.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Dan-in-CA/SIP/issues/143#issuecomment-293668067, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYZCmjYBq3_8JWDsss8nfWWvtkc9_FJks5rvRjRgaJpZM4HK466 .

[Dan-in-CA]

I hooked up the flow sensor I have to the Arduino micro but I didn't get any reading (blowing air through the sensor). After switching to an Arduino Uno it is working.

There is probably some difference between the 2 Arduino models.

[Dan-in-CA]

Looks like the Micro has a different set of interrupt-able pins. I am getting a reading on the Micro with the sensor connected to pin 8 using your original code. It may be necessary to have different versions of the Arduino code for different arduinos.

[DaveSprague]

You might want to try some different pins on the Arduino Micro. The Arduino versions are a bit confusing to me but this page on the EnableInterrupts library talks about what pins can be used on different Arduino versions:

https://github.com/GreyGnome/EnableInterrupt/wiki/Usage#pin--port-bestiary

[DaveSprague]

Yes, or if the Arduinos are "self-aware" enough we might be able to configure, at startup, certain pins to be used based on what type of Arduino it is.

[Dan-in-CA]

Not sure if it is possible to to determine the Arduino model automatically but there are ways to handle it in a single code base. That does not seem to an urgent matter.

[DaveSprague]

Yes, agreed. We can find a way to support a few popular Arduino models pretty easily I think.

[DaveSprague]

The product description of the Arduino Micro, here: http://www.newark.com/arduino/a000053/atmega32u4-arduino-micro-dev-board/dp/63W3544 and the spec sheet you can download there seems to say it's similar to a Leonardo so you might want to check it against this list of pins from the EnableInterrupts usage:

Arduino Leonardo Interrupt Type	Pins
External	0-3 and 7
Pin Change	8-11 and SCK, MOSI, MISO, SS (SS on 3rd party boards)

So you should be able have a flow sensor input on any of these pins (External or Pin Change)

[DaveSprague]

Were you able to get the Micro working okay? I have a Leonardo which appears to be identical except for the form factor so I can do some testing here as well. Here's a link I found on the Leonardo/Micro -- we may have to modify the serial code somewhat.

https://www.arduino.cc/en/Guide/ArduinoLeonardoMicro

[Dan-in-CA]

Yes. The micro works using ports 8-11 with the code as is which is fine for testing.

Considering that a user will need to open the Arduino file so they can upload it it would be easy to have (commented out) lines defining the pins for different models and instructions at the top directing the user to un-comment the line for their particular model.

[DaveSprague]

I just noticed that in home.html, in the updateStatus function, around lines 69-70 I should have put in a test to make sure station.flowrate and station.flowAmt are available before I append them into the display string (to properly handle the case where the flow-sensor plugin isn't being used). Do you want me to make that change?

[shrakrak]

Hi Dave

i have some comments and suggestions regarding the flow meter support

1. Assuming you only operate a single line at a time you can use a single flow meter before the master valve to measure all lines. it is cheap and simple and doesn't impose a strict limitation. So it think this topology should probably be supported.

2. valve malfunction detection - i believe you mentioned this one already. it would be really cool to have that

3. line damage detection - This one is a bit more complex. using historical measurements we can estimate an estimated flow per line using averaging (again assuming single line at a time). If the flow exceed the average by X percent (configurable) some sort of notification is sent. This will allow to detect tears in the hoses or detached droppers. Off course when you add outlets it will false detect a damage but that can be easily handled using some "reset" button.

Thanks for picking up this feature. it will be a great addition. Let me know if you need a beta tester for direct attached flow meter

[Dan-in-CA]

Yes. Go ahead and make the change in your fork. I am also thinking about possibly having some way to load plugin specific JavaScript code.

Right now the plugin files are kept in a separate Github repository: https://github.com/Dan-in-CA/sip_plugins

The README for that repo is used by the plugin manager to list the available plugins. When a user selects a plugin for installation the plugin manager downloads the contents of the folder for the selected plugin(s). There is a manifest file included with the other plugin files. The manifest lists the files that make up the plugin along with their locations in the directly structure. The plugin manager reads the manifest file and places the files in their proper locations. A similar process is used to remove the files if the user uninstalls a plugin.

Until now, none of the existing plugins have had to modify the home page or other pre-defined pages but that should definitely be an option.

Another part of the plugin system is the file init.py in the plugins directory. It does the actual plugin loading when a plugin's group mode is set to executable. That is how plugins can be enabled or disabled.

If it is necessary to modify the base code of one or more of the UI pages to make a plugin usable that is fine with me but it would be nice to be able to load plugin specific JS code that can modify an existing page. That would make code maintenance easier.

[DaveSprague]

@shrakrak agree with your three points. The single flow meter will probably be a popular option, even in cases where there's not a master valve but just a single supply line for some number of valves so we should have a fairly flexible way of assigning meters to valves. I like your idea about line detection too.

@Dan-in-CA Yes, I'll make that change and let you know when it's available. I agree with the desirability of allowing plugins to have javascript code or at least some fairly general way to allow plugins to display real-time data on the home page and to add columns to the log is very desirable.

One way to do this, at least for the log, would be to have a gv.log_columns dictionary that plugins can add entries to. For example, the flow_sensor plugin would add a reference to it's program water usage variable (currently named as gv.plugin_data["fs"]["program_amounts"]) so the key for this entry "Usage" would be used as the column name and the value would be a reference to that "program_amounts" variable. One complicating factor is that currently the program_amounts value is list with an entry for each flow_meter so you would need a way to designate that you want to select the specific amount value corresponding to the currently active valve/program.

Thanks for the description of the plugin architecture. That's very helpful. I'll work on getting the flow_sensors plugin compatible with it.

[Dan-in-CA]

Dave, I have an idea for a way that plugins can modify the pre-defined pages using JavaScript/jQuery to add new display features. It's a little complicated to describe and I have not tested it yet. I'll let you know more after I have a chance to do some tinkering.

The log data is already stored as a series of dictionaries in the SIP/data/log.json file. A plugin could add elements but there would still need to be a way to modify the log page to show the new data and possibly to add the use data to the timeline on the home page.

The log file is updated by the log_run() function in helpers.py line ~321. Perhaps we could add a signal to that function that would be broadcast when a line is added to the log. Then the plugin could update the log with it's own data. The log is updated when a station run is completed. The plugin could get the station ID from the log data, add the usage data and reset the usage counter. -- just some brainstorming.

This is all very exciting. Sometimes a user will request a new feature but when someone actually gets involved to create the feature as you are doing I find it quite motivating. Also the exchange of ideas from different view points leads to innovations that I would not have even thought of.

Dan

[DaveSprague]

Hi Dan, I'm working on one way to allow plugins to add items to the log. I'll send you that once I have something working and see what you think.

I'm not tied to this specific approach but it seems like it will do what we want it to do. The basic idea is to have a variable in gv that I'm calling gv.logged_values that is a list of pairs. The first element of each pair is the name of the value and the second element is a function that is used to compute the logged value at the time the log is updated to the log.json file. (I'm using a list rather than a dictionary since the order is controllable which is important when you display the log.)

I'm initializing the list to be the default logged values as they are in the current code base but it will allow plugins to add other values to the log by adding other name/function pairs. I'll modify the flow_sensor plugin to add it's water usage values.

I considered the signal approach you mentioned but we would still have the issue of the log.html file having to figure out, when displaying the log, what the name of the new values added to the plugins are so I think this approach is comparable to that one. It also avoids having two writes to the log each time the log is updated.

I've got this new version working to the point of updating the logs properly. Now I'm going to update the log display code to use the same approach rather than just having a fixed set of column names.

Once I have something that you can take a look at, I'll move on to making the flow_sensor plugin compatible with the plugin architecture and then hopefully start looking at the charting.

Dave

[Dan-in-CA]

Sounds good! Can't wait to see what you've come up with. I will be off-line for a couple of days so no rush.

[DaveSprague]

I've made some good progress:

Arduino Code Fix First, I found a bug in my Arduino code. I was only using 16b counters for the flow meter pulse counters but 32b counters are required to avoid wrap-around when the water quantity goes above about 140 gallons. I looked back at my previous implemtation and, sure enough, I was using 32bit counters for it. I've updated the Gist archive of the Arduino code to use 32b unsigned counters. https://gist.github.com/DaveSprague/88c2380297e0ae53605ac360299f1b6b

Logged Values from Plugins I've completed the new implementation of the logging function that supports plugins adding values to be logged. There's a new gv variable called gv.logged_values. It is a list of values to be logged. Each value entry is a two-entry list consisting of the name of the value (which is used for the key name in the json file and for the Column name) and a function that is used when the log is created to compute the value to be logged. Here's an example of code in the flow_sensor plugin that places an entry in the log list to include the amount of water used by each program run:

gv.logged_values.append( [_('usage'), lambda : '{:.3f}'.format(gv.plugin_data["fs"]["program_amounts"][gv.lrun[0]]) ])

What this does is it looks uses the current station number, stored in gv.lrun[0], to index into the flow_sensor plugins gv variable that stores the water quantity amounts to get the one for the station that just finished running. It takes that float value and converts it to a string with three digits to the right of the decimal point.

I also added code into the gv.py module that adds the proper values into gv.logged_values to replicate the default log format.

One restriction is that the anonymous function that computes the value to be logged has to return a string so if you need to log a numeric value or boolean you should convert it to a string within the anon. function.

This code is in a separate fork of the SIP project here: https://github.com/DaveSprague/SIPdev.git

Convert Flow Sensor plugin to be compatible with the Plugin Architecture Making good progress on this. I have added the manifest and also added the code to implement a webpage for the plugin that allows you to set the type of interface to the flow sensors (only Simulated and Arduino Serial are supported now), the type of flow sensor used (only 1/2" and 3/4" plastic Seeed type flow meters are supported currently, and whether to log/display Liters or Gallons. Still tracking down a few bugs in this and have yet to test that the plugin can be loaded from the sip_plugins repository.

[Dan-in-CA]

OK. I have gotten your updates and will start exploring and testing.

[DaveSprague]

Thanks Dan. My fork of SIP that's named SIPdev is the one that has the new stuff in it. It looks for plugins in my fork of sip_plugins which is where the flow_sensors plugin is now.

[Dan-in-CA]

Started doing some testing: The plugin install went without a hitch. You have really made progress.

On the plugin settings page I switched from "simulated" to "Arduino-Serial" but it still seems to be running in simulation mode.

I am wondering about support for other flow sensors that users might try to use. Looking at Amazon there are several brands. One vendor has sizes from 1/4 inch to 2 inch: https://www.amazon.com/DIGITEN/b/ref=bl_dp_s_web_14129847011?ie=UTF8&node=14129847011&field-lbr_brands_browse-bin

It might be good to allow the user to (optionally) set parameters for their sensor if it differs from the common types.

Haven't had a chance to dig into the code yet but things are looking good so far.

[DaveSprague]

I'll do a check to see if I'm seeing the same issue on the switch from simulated to Ard.-serial and figure out what's causing it.

That's a great idea to support parameters as an alternate way to set up a specific flow meter. I'll put that on the list of features to add soon. It should be fairly easy.

[DaveSprague]

Another feature I'm planning to add soon is to be able to associate flow sensors with valves in a fairly flexible way, e.g. for each valve you can specify the number of the flow sensor that should run when it's activated. The current setup would be [0, 1, 2, 3, 4, 5, 6, 7] and a system with a single flow_sensor on the master line to all the valves would be [0, 0, 0, 0, 0, 0, 0, 0]. That will probably give us enough flexibility for now?

[Dan-in-CA]

Regarding parameters for different sensors; It might be good to include a "calibration" value. For example a user could run a valve long enough to fill an e.g. 5gal bucket and use the sensor reading from that to set parameters for the sensor. That should make it fairly easy to set up a sensor without knowing it's specs.

[DaveSprague]

Yes, I agree. I've never actually tried testing on of the flow sensors yet to make sure the gallons I'm reporting are fairly accurate so it would be interesting to have someone try that at some point too.

I just tried the Simulated to Arduino-Serial and back to Simulated again and both cases seem to work as expected for me when I kept the flow_sensor type as the Seeed 1/2". I did find one bug in the flow_sensor plugin that occurs when you select the 3/4" flow sensor. In that case, you get an invalid key error when it starts reading the sensors because I had the name for it wrong in the flow_sensor.html file (I spelled it as Seed rather than Seeed three times on line 41.

Are you selecting a 3/4" flow sensor? If so, that may fix the problem. I just pushed a corrected flow_sensor.html up to my fork of sip_plugins so if you reload it that will fix that problem.

If you are using a 1/2" flow sensor, then we'll need to do some more debugging. I still have some print statements in place from testing the savings of the settings. When you save them, you should see something like this in the standard output:

1. settings are:  {'interface': 'Simulated', 'units': 'Gallons', 'sensor_type': 'Seeed 1/2 inch', 'rate_units': 'GpH'}
2. GET method in settings class
3. No flow_sensors.json file
4. my settings here are: {'interface': 'Simulated', 'units': 'Gallons', 'sensor_type': 'Seeed 1/2 inch', 'rate_units': 'GpH'}
5. 192.168.1.118:55701 - - [25/Apr/2017 11:25:53] "HTTP/1.1 GET /flow_sensors-sp" - 200 OK
6. 192.168.1.118:55701 - - [25/Apr/2017 11:25:53] "HTTP/1.1 GET /api/log" - 200 OK
7. qdict = <Storage {'interface': u'Arduino-Serial', 'units': u'Gallons', 'sensor_type': u'Seeed 1/2 inch'}>
8. settings : {'interface': 'Simulated', 'units': 'Gallons', 'sensor_type': 'Seeed 1/2 inch', 'rate_units': 'GpH'}
9. resetting flow sensors
10. values from Arduino on establishing serial port
11. 0,0,0,0,0,0,0,0
12. 
13. after update from qdict, settings = {'interface': u'Arduino-Serial', 'units': u'Gallons', 'sensor_type': u'Seeed 1/2 inch', 'rate_units': 'GpH'}

That last line -- line 13 above, that starts with "after update from qdict, settings =" should show the settings properly being set. In this case, it's showing the settings being Simulated in earlier print statements (lines 1, 4, and 8). In line 7 we see qdict returned from the setting Save action with the new settings and then the settings variable being updated to Arduino-Serial in that last line, line 13.

[DaveSprague]

I also update the data/flow_sensors.json file with the new setting when they're saved.

[Dan-in-CA]

I de-installed and reinstalled the plugin to get the updates. It seems to be working as expected now. The sensor had been connected to a prototyping Arduino Micro on a breadboard with a bunch of other stuff connected to it. That may have contributed to the problem. Now I have the sensor connected to an Arduino Uno without anything else connected.

[Dan-in-CA]

Hmm... After rebooting the Pi it is back to running in simulation mode with the plugin set to Arduino-serial. The flow_sensors.json file has Arduino-Serial as the interface so the setting is saved properly.

[DaveSprague]

Hmm...let me try that and see if I can replicate the problem

[DaveSprague]

Actually thinking about for a second, I believe the problem is that I only access the json settings file for the addon when I go to the settings page. I need to do that when the plugin is initialized, otherwise, it's switching to the default values with is Simulated. Should be a fairly quick fix

[Dan-in-CA]

I think that's it. I went to the settings page and resaved then it started working.