PID setpoint tracking

[isabido]

Hi @colinl

The first to thank you for your excellent job!

I am designing a system at home where I need all photovoltaic production to be used to heat water.

What I need is, if the solar panels are generating 1kw (this measurement is extracted by mqtt from the inverter and it would be the setpoint remote) I need to divert all that power to a resistor controlled by a dimmer with 0-10v signal input through a Shelly Dimmer 0-10v

Regulator 25A https://es.aliexpress.com/item/20000003997748.html?spm=a2g0o.cart.0.0.eec37a9dSt4vWf&mp=1&gatewayAdapt=glo2esp

Shelly Dimmer 0-10V https://shelly-api-docs.shelly.cloud/gen2/Devices/ShellyPlus10V

SP is the instantaneous solar power extracted by MQTT from the solar inverter, logically the SP will be continuously oscillating according to the solar generation.

PV will be a ShellyEM with a clamp measuring the consumption of the dimmer. (This value can be obtained by Json HTTP)

OUT .Output will be sent to the Shelly Dimmer (this expects a range of 0-100%) and then in proportion it will act with 0-10v on the regulator where the resistance that heats the water is connected.

The ultimate goal is that the PV = SP

Do you think could do it with your node?

At the moment I'm trying this initial modified version of your example, but I'm getting the Integral Locked error. I imagine it is because I am not entering the PV value correctly in the PID block.

flows.json

Greetings from Spain and sorry for my terrible English.

[colinl]

Taking into account that solar production will never exceed 2000W, it would be good to be able to tell the PID to never exceed 60%, I understand that this way it will be able to work more precisely.

That is easily done, for the linear case. Simply reconfigure the Range node on the pid output so that instead of scaling the range 0:1 to the range 0:100, change it to scale 0:1 to 0:60. Then you will never get more than 60% out. Tweak the 60 figure to whatever value gives you 2kW.

If you feel the need to put the non-linear scaling in (it should work ok without it but will take a few seconds longer to home in on the correct output for large changes in setpoint) then have a look at https://flows.nodered.org/node/node-red-contrib-spline-curve. You can use that to take out the non-linearity. Personally I would not bother unless you find that it does not work well enough with the linear approximation. PID is good at compensating for non-linearity, that is what it is designed for.

[colinl]

Another important thing is with your 2sec RC FILTER OFF, the PID is not able to regulate.

That is because the current reacts to changes in the requested power in milliseconds. It is much too fast to control with a sample rate of 0.5 seconds. The filter slows it down to make it controllable at that sample rate.

[isabido]

Thanks for your explanations...by the way, before you were retired, what was your profession? I work as an Instrumentalist. And from the way you speak I think something similar.

Regarding the linearity of the load, indeed now that you mention it, a PID works very well with ISO PERCENTUAL outputs (such as control valves), so for the moment I am not going to be distracted by the spline node.

The truth is that your "2sec RC" filter has been essential, without it it is impossible to work. You could integrate some option into the node to configure it internally or add it to the documentation. There may be more users who encounter this practical case and cannot solve it.

For this reason I was telling you, that I see it as more efficient to execute the PID within the ESPHOME itself, and use the ADC input for the current clamp and the DAC for the output to the regulator. It should work much more efficiently and accurately and without having to worry about guaranteeing communication between nodered, Shellys and esphome dimmer.

[colinl]

I worked on the software in industrial instrumentation, such as temperature controllers.

The truth is that your "2sec RC" filter has been essential, without it it is impossible to work. You could integrate some option into the node to configure it internally or add it to the documentation. There may be more users who encounter this practical case and cannot solve it.

This is actually the first time I have had to suggest it, as far as I can remember.

For this reason I was telling you, that I see it as more efficient to execute the PID within the ESPHOME itself, and use the ADC input for the current clamp and the DAC for the output to the regulator. It should work much more efficiently and accurately and without having to worry about guaranteeing communication between nodered, Shellys and esphome dimmer.

Yes, there is logic in that, in particular the fact that it does not rely on the comms being available. I doubt if you would be able to sample fast enough to avoid having to use a filter though. I have written a version of the algorithm for use in the Tasmota s/w (https://tasmota.github.io/docs/PID-Control/) which could no doubt be modified to make it available in ESPEasy. I have no information on the one which is apparently already available there.

[isabido]

Hi, although the error no longer appears in the DIMMER node, the 2secRC output continues to drift, can you think of how to solve it?

I had already seen Tasmota's PID but I didn't know that it was based on your development. I was helping Arendst with the phase calibration adjustment on the Shelly EM. ( https://github.com/arendst/Tasmota/discussions/16486 )

Do you think Tasmota could take the reading of the GPIO ADC and GPIO DAC, to execute the PID?

[colinl]

2secRC output continues to drift

Can you add debug nodes showing what is going in and it of the node please and post a selection showing the problem please.

Do you think Tasmota could take the reading of the GPIO ADC and GPIO DAC, to execute the PID?

If you provide it with the data it will run the algorithm.

[isabido]

I think it's not that it overflows below 0, it's that it has a large number of decimals. There are times when the nodered service even restarts. It could also be because it is running on a poor Raspberry 3.

[isabido]

here with a debug before and after 2secRC

[colinl]

I think it's not that it overflows below 0, it's that it has a large number of decimals.

That is not a problem, it is just the value dropping exponentially towards zero.

There are times when the nodered service even restarts.

You need to look at the node-red (and/or system) log and find why it restarts. I will be very surprised if it is anything to do with the PID node or the filter.

[isabido]

That is not a problem, it is just the value dropping exponentially towards zero.

Even so, it seems like a bit of a waste of resources for this limited raspberry3, is it difficult to limit the resolution of that node to 3 decimal places?

Today in Spain we will have a good day of solar production. I have set the PID controlling for surpluses and looking for a setpoint of 0W, to achieve a 0W delivery to the grid. As you can see, it is regulating very effectively.

In this graph you will see the solar power curve, with the power delivered to the thermos. You can see how the curve follows perfectly, the difference between them is the residual consumption that the home has right now about 200-300W. And it can also be seen that at no time does it exceed the "solar curve". Let's see how the rest of the day develops, but it seems like a promising result.

Here you can see how the water temperature evolves. I estimate that to heat the water from about 30 to 65º I will need approximately 4-5kwh.

[colinl]

it seems like a bit of a waste of resources for this limited raspberry3, i

All floating point numbers take the same amount of space and processing power, whether the value is 1.0 or 1.12345667E-30. There is no difference as far as the processor is concerned, it is just a arrangement of bits.

The pid stuff would run perfectly happily on a Pi zero.

[isabido]

The pid stuff would run perfectly happily on a Pi zero.

I wish it were just the PID, in the end this raspberry is running the entire homeassistant system, nodered, esphome, mqtt, etc. I have to transfer it to another machine, because the poor thing is very tight.

On the other hand although the PID seems to work quite well with your initial values. I would like to be able to find the limit and see how to improve if possible, where would you recommend I start?

[colinl]

I wish it were just the PID, in the end this raspberry is running the entire homeassistant system, nodered, esphome, mqtt, etc.

Exactly, so whether the precision of numbers in the PID loop has an impact (which it doesn't) is irrelevant.

I would like to be able to find the limit and see how to improve if possible

What aspect do you want to improve?

[isabido]

What aspect do you want to improve?

If we look at the graph, during the morning when no one was at home, the consumption of the house has been stable at 200-300W, so the PID has controlled all the solar surplus perfectly.

Around 2:20 p.m., we enter the house and begin to "live our normal life" and see how he no longer responds so well.

The first thing is that all the yellow circles are consumption of the thermos above the generated power, it should never occur.

And then I also can't understand what could have happened in the area where I indicate in red, it takes almost 5 minutes with the output at 60% (maximum scaled), when it has a consumption of +800W. The output should have been lowered to reach 0W.

[colinl]

Is the PID_OP line the value after the RC filter?

Is the shelleyem line the measured power going into the heater? If so then at the red arrow, with the pid output at 58 continuously, why is the blue line not showing a steady 2000W?

And then I also can't understand what could have happened in the area where I indicate in red, it takes almost 5 minutes with the output at 60% (maximum scaled), when it has a consumption of +800W. The output should have been lowered to reach 0W.

Is the Potentia Termo line the setpoint going into the PID node? If so, should that be at 2000 at that time?

[isabido]

Is the PID_OP line the value after the RC filter? Yes.....

Here I detail what each thing means.

Is the Potentia Termo line the setpoint going into the PID node? If so, should that be at 2000 at that time? No...

It is only informative to see how much the PID output is actually consuming. The setpoint is always 0W.

It is controlling for zero discharge to the grid. All the solar surplus that is not consumed by the home is directed to the thermos.

[colinl]

Which line is the PV going into the PID node, and which line is the SP going into the PID node? If they are not any of those it is them (and the PID output) that I need to see.

[isabido]

SP is not shown on the graph, it is ALWAYS 0W PV is Shelly EM. Thermo Power is only informative, to see how much energy the resistor consumes, it should never exceed 2000%, which approximately coincides with 60% of the PID output.

[isabido]

Let's see if this makes it clearer... as you see the yellow circles, it is as if the PID had responded in an "inverse" way, while the abrupt change indicated in red responds as expected.

[isabido]

Analyzing in more detail, and focusing on that exact moment, it is very rare that the PV value is completely fixed for 4 minutes without oscillating. This smells to me like communication problems or a raspberry with a very high load.

[colinl]

SP is not shown on the graph, it is ALWAYS 0W PV is Shelly EM.

The pid node tried to control the pv to the sp value, so that means that the pid node is trying to control the Shelly value at 0. Is that correct?

[isabido]

SP is not shown on the graph, it is ALWAYS 0W PV is Shelly EM.

The pid node tried to control the pv to the sp value, so that means that the pid node is trying to control the Shelly value at 0. Is that correct?

That statement is correct

In that section that I indicated in red, the OP would have to have gone down for the PV to approach SP=0W.

I think this is a specific failure in the data acquisition or a "collapsed" raspberry

Sometimes I also encounter this error in nodered, where the shelly does not respond in time.

[isabido]

Here you can clearly see how NODE-RED "blocks" and does not take the reading from the ShellyEM.

Homeassistant at the same time is able to read the power data from the ShellyEM, although at a lower rate obviously.

[colinl]

So are you saying that the problem is in reading the shelly values? I don't know anything about those nodes.

It seems you have not filled in a value for Disabled Output. Set that to 0 and set Max Sample Interval to 5, so that if 5 seconds elapses without a PV value then the output will go to zero.

If that doesn't make a difference, connect a debug node direct on the output of the PID node, set to Output Complete Message please. Wait till it appears to be malfunctioning and then capture a couple of messages from the debug node, and post them here fully expanded. I want to see what the pid node is actually getting.

[isabido]

So are you saying that the problem is in reading the shelly values? I don't know anything about those nodes.

Yes, indeed I have doubts, whether the shellys deny requests as fast as 500ms, or is it the Raspberry itself that crashes. This weekend I am going to migrate the HA installation to a PC to rule out this problem.

I have already placed the debug, let's see if I can see the messages just when it fails.

[colinl]

When it does, make sure that you expand some of the messages so we can see all the contents.

If you have difficulty catching the messages at the right time then set the debug node to output to the Console (which means the node-red log) then when it fails you can look back to see what it says in the log.

[isabido]

Would it be okay like that? Is that what you expect to see, when you say expanded?

This is where you recommend I look, right?

[isabido]

This is what seems to be happening restarting node-red with the "Service Node-RED exited with code 256 (by signal 9)" seems to be a memory overflow problem.

https://community.home-assistant.io/t/service-node-red-exited-with-code-256-by-signal-6/597317/16

[colinl]

What does top show when everything is running?

[isabido]

[isabido]

In case it gives you any more clues... the raspberry definitely cannot, nothing more than the ssh session, the keyboard works with a significant lag. It is very overloaded although the CPU values do not indicate it. With this hardware we will not be able to draw more conclusions or adjust the PID tuning.

[isabido]

I have already prepared this "new" machine, which I hope will be somewhat more extensive, it is a mini HP Pavilion dm1 Notebook PC where I will start running all the processes, but I will not be able to connect it until Wednesday/Thursday.

[colinl]

Very odd, the load average figures show it is clogged up but there is little cpu usage. I think that means there is lots of IO going on, and the sd card access is limiting it.

[isabido]

Well... let's let it go, there is no point in wasting any more time with this poor raspberry. When we move to the new machine we see that it is so and we will be able to draw better conclusions.

On the other hand, does your PID node implement ANTI-WINDUP?

[colinl]

does your PID node implement ANTI-WINDUP?

Yes, that is what is active when the status shows Integral Locked.

[isabido]

does your PID node implement ANTI-WINDUP?

Yes, that is what is active when the status shows Integral Locked.

How wonderful, you didn't miss any detail when developing it!

I'm curious about running Tasmota, with an analog input with the clamp and the direct output to the 0-3.3v/4-20mA converter, I want to think that it has to be much "finer".

By the way, I think I already know what is happening on the Raspberry, the PID-PV sensor saves a significant amount of data (I think it saves a value every 500ms) in the HA DB and it doesn't seem to feel good at all when I ask show me a "day"

I have noticed because on the new PC where I have already installed HAOS, when I recover one day there is no problem. but if I ask for a week of that variable, it takes a long time to show it and then when you move through the graph it appears very lazy when showing values.

[colinl]

it doesn't seem to feel good at all when I ask show me a "day"

I should think not, that is 172800 samples. The browser will never cope with that. That is not going to work on your new system either. That is probably why the pi is IO bound, trying to write all that data to the db. I suggest removing HA from this part of your system if you can and do it all in node-red. If you can't then only send 1 sample in 10 or more to HA for logging.

[colinl]

I'm curious about running Tasmota, with an analog input with the clamp and the direct output to the 0-3.3v/4-20mA converter, I want to think that it has to be much "finer".

You would still have to do the scaling and add the 2 second RC filter of course.

[isabido]

Tengo curiosidad por ejecutar Tasmota, con una entrada analógica con la pinza y la salida directa al convertidor 0-3.3v/4-20mA, quiero pensar que tiene que ser mucho más "fino".

Aún tendrías que hacer el escalado y agregar el filtro RC de 2 segundos, por supuesto.

That's going to be too much for my programming knowledge. A shame!!

[isabido]

it doesn't seem to feel good at all when I ask show me a "day"

I should think not, that is 172800 samples. The browser will never cope with that. That is not going to work on your new system either. That is probably why the pi is IO bound, trying to write all that data to the db. I suggest removing HA from this part of your system if you can and do it all in node-red. If you can't then only send 1 sample in 10 or more to HA for logging.

I have already eliminated the sensors towards HA, if I want to monitor the PID it would be better to do it in a NODE-RED graph as you had left me in your initial flow.

This is how it finally turned out.... Also, if you look at the output, I do it directly with a node that has recently been published and sends the data directly to the ESP, without going through HA.

[colinl]

if I want to monitor the PID it would be better to do it in a NODE-RED graph as you had left me in your initial flow.

I don't agree there, I just used the node-red graph in the example as it is a simple way of seeing what is going on. The problem is that if you want to go back and see what happened yesterday, or zoom in on a particular time period, it is not possible. On my real systems I use Influxdb, but write data at slowest rate that will give the results I need, and I use Grafana for viewing the data. Using grafana with influxdb, when you want to look at a months worth of data, for example, it runs queries on Influxdb that aggregate data points (taking the mean for example) into the number of points that is useful to show on the graph. Also influx has built in methods for automatically deleting data when it is older than a certain period, and for downscaling data so that recent data can be viewed at full time resolution, but when it is, for example, older than four weeks the data are aggregated and stored at a lower time resolution. Influx does not do well on an SD card though, as it does a lot of moving stuff about so needs a hard disk or SSD. I run my home control system in a number of Pis and ESP systems, but run influx and grafana on an old (i5) laptop running Ubuntu, which picks up the data via MQTT from the Pis.

[isabido]

Yes, I think my terrible English was unable to explain that I agreed with you.

Here is the graph

I have decided to graph the values that come out of your PID node, instead of taking the PV and SETPOINT of the input, so I can make sure what the node is interpreting, does this seem correct to you?

I was also using grafana/influxdb for a long time, but for my needs the HA graphics were sufficient, but of course this option that you mention is the best without a doubt.

[colinl]

I have decided to graph the values that come out of your PID node, instead of taking the PV and SETPOINT of the input, so I can make sure what the node is interpreting, does this seem correct to you?

Yes, that is good.

You said earlier

SP is not shown on the graph, it is ALWAYS 0W which did confuse me, and now seems not to be the case.

[isabido]

Yes, sorry to bother you, but since it is night and there is no solar production, I am with an SP a little higher than the real consumption of the house to see how it regulates.

Under normal conditions it will always be 0W when you want to make zero discharge, the other possibility is to allocate all the solar production to send to the thermos, and here the SETPOINT will be the same as the solar production.

[isabido]

Let's see what you think of these values, I see it as very bad but I don't know where to attack.

And after a while it doesn't get better but gets worse....

More later.....

[colinl]

Is this still on the Pi. I don't know much about Docker or Hass IO but it looks as if you are using lots of Swap which probably means you have not got enough RAM.

[isabido]

If it is still running on Raspberry, and I see the same thing as you... the SWAP is at full capacity, and continuously moving data, which is even more accentuated with access to the SD. This raspberry 3 has only 1gb. Well, on to something else...... the PID seems to be responding very well this early in the morning, it is also in very favorable conditions since there is no one at home and the consumption is stable at about 300W, You just have to increase the PID output as the SUN rises!

[isabido]

I have created a new graph, with the Solar Power and the power consumed by the thermos, and one hour of recorded data.

[colinl]

You could probably reduce the prop band a bit which would pull it in a bit quicker. Try 2/3 of whatever it is at the moment. If you bring it down too m much it will start bouncing about.

[isabido]

Right now P=2000 (everything is as you indicated at the beginning)

Here you can see how it responds to an instantaneous PEAK of 2000W, it is just the START of the refrigerator.