Feature request - flow meter add in

[mattip]

Thanks for this wonderful application.

I have a Seed Studio flow meter http://www.seeedstudio.com/depot/G12-Water-Flow-Sensor-p-635.html hooked into my raspberry pi.

I am using wiringpi like the code posted here https://www.raspberrypi.org/forums/viewtopic.php?f=37&t=94656 to set up an interrupt handler and increment a counter. I see how I can write this as a so and read the counter from Python. I would like to get feedback in SIP about overall flow rate and expected vs actual flow rate (to detect leaks). What would be a strategy to get this into SIP? I see the way the clock time is updated, should I hook into that and update another div under it?

Note that I have not calibrated the flow meter, from some preliminary tests it seems the meter peaks out at about 100 pulses per second, so this might not be an accurate way to measure water quantity rather a qualitative go-no go check of the system, an arduino and maybe a better meter may be needed to get accurate measurements, but that is a different issue.

[DaveSprague]

Yes, I think so too. I fixed the startup code in flow_sensors.py and also noticed a case where the quantity and rates units (e.g. gallons and gallons per hour) could get out of sync (e.g. Liters and Gallons per hour) so added a bit of code that I believe fixes that too.

I've uploaded the changes to sip_plugins and tested it some and it seems to be working properly now. The very first time I reloaded the plugin and tried to run a Run Once program, I got this error:

Some Stations has been scheduled: [[0, 0, 0, 0], [1493139588, 1493139600, 12, 98], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
192.168.1.118:55287 - - [25/Apr/2017 16:59:49] "HTTP/1.1 GET /cr" - 303 See Other
192.168.1.118:55287 - - [25/Apr/2017 16:59:49] "HTTP/1.1 GET /" - 200 OK
192.168.1.118:55287 - - [25/Apr/2017 16:59:49] "HTTP/1.1 GET /api/log" - 200 OK
Traceback (most recent call last):
  File "/home/pi/SIPdev/web/application.py", line 237, in process
    return p(lambda: process(processors))
  File "/home/pi/SIPdev/web/session.py", line 85, in _processor
    self._load()
  File "/home/pi/SIPdev/web/session.py", line 106, in _load
    d = self.store[self.session_id]
  File "/home/pi/SIPdev/web/session.py", line 250, in __getitem__
    return self.decode(pickled)
  File "/home/pi/SIPdev/web/session.py", line 210, in decode
    return pickle.loads(pickled)
EOFError

But I haven't been able to replicate it so keep an eye out for it.

[DaveSprague]

Just noticed that this "pickle EOFError" error is similar or the same as the one noted here: https://github.com/Dan-in-CA/SIP/issues/37

[Dan-in-CA]

I see that error on rare occasions. I have never been able to find the cause or a solution.

[DaveSprague]

Some thoughts on what I'd work on next. Let me know which would be best to prioritize.

1) Take a pass at cleaning up / refactoring the flow_sensors.py code so it's more readable. 2) Start adding support for data gathering that will eventually be used for charting. 3) Adding support for custom flow_sensor parameters to permit using a wider range of sensors and supporting calibration. 4) Adding some documentation linked to the Help button that would describe how to set up flow sensors and use the plugin.

Some initial thoughts on the data gathering (#2 above). I think we should add a Signal for End of Program (which I don't think exists currently?) and also for End of Day. This would allow us to capture the water usage on each station and accumulate it to get Daily, Weekly, and Monthly totals. We'd then maintain a separate log via a json file that stores the current day's usage, and the last 7-14 days, last 10-15 weeks, and last 12-24 months of water usage. As part of this we can also track any water usage that occurs when a program is not running on a station to check for a valve that's stuck on.

[Dan-in-CA]

Hi Dave,

As far as priorities, I think #3 would be good to work on. The calibration and parameters for different sensors are closely related. Doing a calibration should provide the values needed (volume / ticks) to use the sensor without even needing to know the published specs.

Of course documentation in important but if there might be changes to how things work it might be a good idea to start an outline and develop toe details when things are more complete.

On #2, Agreed that signals that mark time intervals like days, weeks, etc should be added to the core code. In fact another user is working on a plugin that will send notifications to e.g. a smart phone when selected events occur. See the SIP forum: http://nosack.com/sipforum/showthread.php?tid=72&pid=275#pid275

He has used a signal that marks the end of a station's run. It might be good to include some signals in the flow_sensors plugin that could be used to notify a user of unusual flow rates. There is also the telegramBot Plugin that could notify a user of abnormal operation.

1 cleaning up the code should make further development easier. I guess how you prioritize things depends on your particular way of doing things.

I have been trying to get communication working over RS485 so I can connect to sensors that are not physically close to my Raspi. I have 7 valves that are in 4 locations around the house. The closest ones to the SIP controller are about 30 feet away (beyond the 16' max for USB). I think this situation is probably fairly typical. So far I am able to write from an Arduino micro to the Pi but can't seem to get it working in the other direction. Probably just not doing something right. This should not involve much change to the plugin code. The RS485 adapter plugs into a USB port on the Pi and converts RS232 to RS485. On the Arduino side there is an adapter the translates from RS485 to RS232 so it is just basic serial communication. There would also need to be some sort of addressing method if there are 2 or more arduinos in the system.

[DaveSprague]

Yes that sounds like a good priority for me to address items. I'll start work on items #3 and #4 for now.

As for the end of day, week, month type signals one thing we'll have to think about is what order those should be generated in the case where the end of day and end of week and possibly end of month occur at the same time. My first thought is that the end of day should be signaled first and then the end of week happens maybe a few seconds later, and then when necessary, the end of month would be send a few seconds after that??

The other option would be to just send an end of day signal and let the code/plugin that uses the signal go through and check if it's also the end of week and end of month if it needs to. That way it can handle those situations however it chooses. An example would be where at the end of each day I compute the daily water usage and then for the end of week I use that day's total plus the previous six day totals to compute the weekly water usage.

On the RS485 serial interface, you should check that you're flushing the output buffer on the Pi side after you write something? Here's the code I'm using (edited slightly for clarity) in the flow_sensors plugin to write to the arduino and then receive data back. I'm not completely happy with having those time.sleep calls in there but they are working for now.

ser = serial.Serial('/dev/ttyACM0', 9600, timeout=1)
time.sleep(0.1)
ser.write("RS\n")
ser.flush()
time.sleep(0.1)
line = ser.readline()

[mattip]

Hi. I would like to try this out, it would be great just to get feedback that water is flowing in the system when a program is running, no need for calibration, just a go/nogo.

What should I do to try out the code?

[DaveSprague]

Hi Matti,

If you'd like, you can use my modified clone of SIP here: https://github.com/DaveSprague/SIPdev

It has some changes to the base SIP code that allows the real-time gallons or liters per hour to display, on the home page, while a valve is running and it modifies the log code to permit plugins to add columns to the log. It also points to my own version of the plugins repository so that you can add the flow sensors plugin that handles the interface to the flow sensors.

The only configuration that's currently supported is one or more flow sensors that interface to an Arudino and then the Raspberry Pi that's running SIP reads the flow data from the Arduino via a USB serial interface. The Arduino code is located here: https://gist.github.com/DaveSprague/88c2380297e0ae53605ac360299f1b6b

The Arduino I use is an Uno so if you use a different type, there might be some debugging to get the connection working and you might need to attach the signals from the flow sensors to different input pins but I can work with you on that.

Since you only want to see whether water is running or not, you can just keep the flow sensor plugin default for the type of valve.

Do you expect to use one flow sensor per valve or a single flow sensors on the main supply? I haven't yet added the code to the flow sensor plugin options to support the later yet but I can do that in the next couple of days if that's how you expect to use it.

Thanks,

Dave Sprague

[mattip]

Cool, I will try it out. I only have one flow meter, so no extra
code needed.
Thanks for putting in the effort
Matti

[mattip]

@DaveSprague I have finally gotten around to testing this. Thanks for all the work, it certainly made it easy to get started. Here are some thoughts:

• I did use the GPIO module + multiprocess.Process to set up a callback in another python process to increment a counter
• I only have one flowmeter and am returning [value]*8 from read_flow_counters. Won't that give strange values for the watering amount on each of the 4 stations?

I will let it run for a few days to see if the reporting is useful. Already it has achieved my goal which was to verify that a running program actually opened the valve, my whole setup is a bit unreliable.

[mattip]

Wow, it actually works! Here is a log for the past few days, with a cycle of each of my 4 programs. The numbers are indicative, but need calibration. Note that since I only have one flow meter, the numbers simply increment across all channels

Date	Start	Station	Duration	Program	Usage
2017-08-16	06:50:00	1: Flowerpots & vegies (pin 7)	29:00	4	2225.72
2017-08-16	06:00:00	2: Center (pin 11)	45:00	1	2203.62
2017-08-16	04:45:00	3? trees? (pin 22)	70:00	3	1996.42
2017-08-15	07:20:00	4Top: Terrace, sidepots (pin12)	60:00	2	1254.33
2017-08-14	19:07:48	4Top: Terrace, sidepots (pin12)	40:00	Run_once	642.81
2017-08-14	18:28:35	1: Flowerpots & vegies (pin 7)	10:00	Run_once	237.91
2017-08-14	18:15:27	1: Flowerpots & vegies (pin 7)	10:00	Run_once	229.26
2017-08-14	06:50:00	1: Flowerpots & vegies (pin 7)	29:00	4	217.08
2017-08-14	06:00:00	2: Center (pin 11)	45:00	1	205.20
2017-08-13	18:08:04	2: Center (pin 11)	45:00	1	0.00
2017-08-13	16:51:20	1: Flowerpots & vegies (pin 7)	29:00	4	0.00
2017-08-13	16:18:19	2: Center (pin 11)	20:00	Run_once	0.00

I am trying to work my changes (using GPIO on the RPI) to @DaveSprague 's code into a pull request

[mattip]

here are the very rough changes to support GPIO flowmeter reading with no arduino. https://github.com/DaveSprague/sip_plugins/pull/1

[Dan-in-CA]

Cleaning up old issues.

[mattip]

Should I resubmit the water meter plugin against the new SIP plugins repo? Unless @DaveSprague wants to do it.

The plugin has been working well for me. It supplies needed feedback about the proper functioning of the system.

[Dan-in-CA]

The latest version of SIP requires some changes to the plugins in order to function under Python3. It would be best for @DaveSprague to do the updates to his plugin.

Otherwise you could stay with the older version.

[DaveSprague]

I should be able to do that in the next week or so.

Dave

On Mon, Dec 30, 2019 at 2:11 PM Dan notifications@github.com wrote:

The latest version of SIP requires some changes to the plugins in order to function under Python3. It would be best for @DaveSprague https://github.com/DaveSprague to do the updates to his plugin.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Dan-in-CA/SIP/issues/143?email_source=notifications&email_token=AALBSCUDDSKPWJANEYCIP6LQ3JBXPA5CNFSM4BZLR25KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH27OLY#issuecomment-569767727, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALBSCQZQO4PG3UA4OQDFXLQ3JBXPANCNFSM4BZLR25A .

[DaveSprague]

Are the required changes "just" for language compatibility with python 3 or are there some other changes that are required for compatibility with the new version of SIP?

On Tue, Dec 31, 2019 at 7:52 AM Dave Sprague david.sprague@gmail.com wrote:

I should be able to do that in the next week or so.

Dave

On Mon, Dec 30, 2019 at 2:11 PM Dan notifications@github.com wrote:

The latest version of SIP requires some changes to the plugins in order to function under Python3. It would be best for @DaveSprague https://github.com/DaveSprague to do the updates to his plugin.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Dan-in-CA/SIP/issues/143?email_source=notifications&email_token=AALBSCUDDSKPWJANEYCIP6LQ3JBXPA5CNFSM4BZLR25KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH27OLY#issuecomment-569767727, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALBSCQZQO4PG3UA4OQDFXLQ3JBXPANCNFSM4BZLR25A .

[Dan-in-CA]

Most of the changes were to enable the program to run under Python 2 and 3. Another big change was a new irrigation program file format. That may not affect your plugin.

[DaveSprague]

I have completed the necessary changes made to the flow_sensors plugin, I believe. I'm not able to test it on my functioning system at this point but it runs properly using the simulated flow meters.

This new version is in my fork of the plugins repository: https://github.com/DaveSprague/sip_plugins . (flow_sensors is the only plugin that's been modified).

Next step is to (re)make my modifications to SIP 4 code base that allows the flow_sensors program to display real-time flow data on the Home page and also add water usage amounts to the program log file.

Dave

On Tue, Dec 31, 2019 at 1:40 PM Dan notifications@github.com wrote:

Most of the changes were to enable the program to run under Python 2 and 3. Another big change was a new irrigation program file format. That may not affect your plugin.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Dan-in-CA/SIP/issues/143?email_source=notifications&email_token=AALBSCWI2PSG6K4DGINZMCLQ3OGZFA5CNFSM4BZLR25KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH4RLIQ#issuecomment-569972130, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALBSCXLW77BPRBHEQOQELTQ3OGZFANCNFSM4BZLR25A .

[mattip]

ping. Anything I can do to get this merged to SIP4 ?

[DaveSprague]

Thanks @mattip. As I recall, the main thing that's missing is a way, on the plugin settings page, to calibrate a flow sensor -- to be able to directly enter a gallons per minute or hour number that someone has calculated. That should be fairly easy to do but I am at a different location for the next several months so I don't have direct access to my irrigation system to do the necessary testing.

The other more complicated issue, that I believe is still open, is to come up with a way for plugin's like this one to display live info on the home screen while a program is running and a way for a plugin to add data to the logs. I haven't kept up enough with the project lately to know if a way to do this now or not.

Dave

[DaveSprague]

One other item that's important to add soon, I believe, is a way to flexibly describe which flow sensor are connected to which valves to permit applications where there's just a single flow sensor that monitors the water going to all the valves or cases where a single flow sensor might be monitoring the supply to a subset of the valves.