Kiln too hot, catching up, but still outputs power.

[captaincaden]

Alright, I touched on this in my last post in here, but I'm going to make it an actual issue now that it's happened twice during glaze firings to see what is going on or what I may be doing wrong.

So, the first bisque firing I did yesterday went great, and followed the profile perfectly until the kiln sitter shut the kiln off (the current way I'm using it is using the kiln controller to perform the ramping, but using the kiln sitter to shut itself off.

During the first test firing to cone 6 that I did, I noticed that it followed the profile perfectly (after I used the auto tuner) until I got into the last segment. The kiln had gone too cold prior to the last segment, and then when it slowed down in the last segment, the kiln caught up, and the controller basically continued to output power constantly, which continued at a constant rate until the kiln sitter dropped at cone 6.

In the case of the glaze firings, it really doesn't matter all that much to me, simply because I used to use the switches on high the whole time anyways, so it's not much of a difference, but I'd like to understand what's going on and why it's happening.

I altered my cone 6 firing schedule a little bit, and this time, the kiln was behind by 10-11 degrees in the last part of the first segment (up to about 1050 degrees f), and then once my segment to hold at 1050 for 10 minutes approached, the kiln caught up, and then instead of holding, or stopping, it just continued to fire, and likely will continue output like this for the rest of the firing ( it's still ongoing at the moment, so I don't know that for a fact, but just an assumption based on how it's going right now.)

Again, this doesn't matter, because the kiln sitter is doing the actual temperature shut off, not the controller, but just not understanding. I've attached a photo of what the ramp looks like at the moment.

The kiln is still outputting full power to the relay, despite logging that the kiln is too hot.

So to summarize the issue, we started with a log of kiln is too cold, it continued to ramp constant power output, but once it caught up, and the schedule changed, the kiln continued to output full power, while it logged kiln is too hot.

[captaincaden]

When we got to about 1175 degrees, the controller stopped constant output, and held it at 1175 for a while. I eventually switched the kiln back over to it's own control and stopped the controller since this is a fully loaded kiln. This wouldn't have changed anything though, because the scheduled target temp was still 1050 and wasn't rising, and the temperature wasn't dropping.

Still, would like to understand the issue here.

[jbruce12000]

Are you using the latest version of the software? I've made a lot of changes over the past week which will likely fix this problem. The kiln should NEVER send heat to the elements if it is too hot. Now the software uses PID control only within a window of N degrees. Outside that window it either turns the elements 100% on [if too cold] or 100% off [if too hot].

I've also changed the logging so it is better organized. I created a reporting tool that will help me troubleshoot. Here is an example from a cone 05 bisque fire I did yesterday.

final.pdf

Please update to the latest version, try a glaze firing, and post the logs if you have this problem again.

My guess is that you have an Integral component that is accumulating A LOT of integral over time. That integral is causing you to overshoot the set point by A LOT.

[jbruce12000]

I want to reiterate... Your PID parameters are not yet well tuned. During the kiln run shown above, my average error was .86 degrees F for the entire schedule. The max error was less than 10 degrees F (and this was an overshoot at low temps).

I'll help you with tuning. Update to the latest code. Run another glaze fire. Post logs.

If you need help with any of those steps, lmk.

[captaincaden]

I will figure out how to update once this firing completes. Got two or so more hours until it's finished.

I'm not super familiar with the concept of PID, so I just ran the auto tuner, and inputted the numbers it gave, and then tested it, and was surprised at how accurate it was during the test, besides this issue. I add this part of my response just because the different terminology are things I know, but don't fully understand how they play together. I believe error refers to the differences in actual temp, and target temp, if that's the case, when I was watching it, it was changing temps in that faster first segment pretty much as soon as the target temp changed. As we got into the thousands though, the kiln struggled a bit, and was behind about 8-9 degrees, and then eventually around 11 once we got near the 1050 degree hold. During yesterdays bisque (a much much slower profile than my ideal glaze profile (ideal glaze profile based on what I know the kiln can do with just it's switches set to high) though, it was well within 1 degree the entire time, if not less.

Which logs is it that you want to see? The daemon logs? I would have all of those that happened during this incident, but can confirm that during the incident happening, the logs were all 'kiln too hot' errors, but the output at the relay was on 100%. I would expect that when this catching up would happen, it would just stop heating until it got back to in this case 1050 degrees, and then resumed once it was within the 10 degree spec, but it continued to output.

I will do some more research here on merging/updating and see what I can do once this firing completes with the kiln running manually. I'm still using the kiln-controller as a pyrometer at the moment even without it controlling.

[captaincaden]

What would you say the best way to update is, when there have been local changes to some of the project? What would you say the best way is if there weren't local changes? While it'd be nice to just have it all swapped over, if that's not a simple task, then it'd be easier to copy over my changes into the updated files, and then work from there, but am unfamiliar with the different options available to update

[jbruce12000]

For fast ramps where the kiln has trouble keeping up with your profile, you can do a few things:

1. increase Kp which is the proportional to a higher value. this will cause overshoot at low temps
2. decrease Ki which is the integral. This gives you MORE integral, and you probably do not want to do this. It looks like you have more than enough.
3. check the age of your elements... you may just be asking the kiln to heat more quickly than it can.

[captaincaden]

I think the issue in this case, that caused offset, is that the kiln was just not able to keep up with what the profile was given. The elements are all within resistance spec, and are relatively new. Still though, I think the controller was telling it to output, but the kiln was unable to keep up perfectly. But I would think that the controller would be able to do the catch up, and would just pause the profile until the kiln was back in time spec. In this case, we were 10 minutes away from getting on to the slower ramp segment. Before I redid this profile and during the first glaze test fire, this section was perfectly on point even the 1050 degree 10 minute hold. The issue seems to only happen when the kiln had first got off being too cold, then it starts to heat up, but surpasses the target temperature, then continues to heat for some reason.

[jbruce12000]

Here are the best ways to update.

without local changes:

cd kiln-controller
git status
git pull

with local changes, it gets tougher. you have to stash your local changes and then go a git pull, then merge your changes with mine. If there are conflicts [say we changed the same line], you'll have to manually resolve them and merge. See this doc for instructions...

https://www.atlassian.com/git/tutorials/saving-changes/git-stash

[captaincaden]

Here are the best ways to update.

without local changes:

cd kiln-controller
git status
git pull

with local changes, it gets tougher. you have to stash your local changes and then go a git pull, then merge your changes with mine. If there are conflicts [say we changed the same line], you'll have to manually resolve them and merge. See this doc for instructions...

https://www.atlassian.com/git/tutorials/saving-changes/git-stash

I have all of my changes backedup in .txt files on my computer, so I'll probably just git pull, then go in and paste in the changes I made. There's only 2-3 sections I'd need to copy and paste, and then make sure it doesn't conflict anyway else. Thank you!

[jbruce12000]

So you understand what was happening with the old version of the software you're currently running:

1. the kiln is unable to keep up with your profile
2. Integral builds up because the temperature is constantly under the set point
3. once the set point is reached, it overshoots and continues heating above the set point until the integral component is zeroed out. This overshoot can be large and it totally depends on how much integral is accumulated. You see this as being out of control.

What changes with the new version [if you have kiln_must_catch_up=True]?

1. the kiln is unable to keep up with your profile
2. it struggles and falls outside the PID control window [for example 5 degrees below the set point]
3. the kiln pauses the profile and applies 100% power to the elements until it gets back inside the window
4. once inside the window, PID control takes over

While outside the PID control window, no Integral accumulates. If the kiln goes outside the window, you'll see this stair step pattern form on the web client as the profile pauses [vertical part] until it gets back in the window [horizontal part]. This may repeat over-and-over if the kiln cannot keep up.

[jbruce12000]

There are changes to config.py. You'll need to take your changes and add them to the newest version the same way you're planning to update other files.

Good luck, Jason

[captaincaden]

Thank you again. The kiln finished, and I updated, and transferred all of the changes I've made and ran some quick tests in simulation, and everything seems to be working well. Still got a long way to cool, but will try the glaze firing again with the update and see how things go once the kiln is cooled and unloaded.

Thank you again for your help. I'm still processing your first response, but will edit this once I understand more

[captaincaden]

I believe I understand the concept. Are you saying that the updates (probably) would solve this issue? Without needing to make any changes to the current PID parameters?

[jbruce12000]

The updates will likely mask the mis-tuning of the PID. Don't make any changes [except for the code update] for now. Let me see /var/logs/daemon.log from your next run. This file will be big. Some folks have used google drive or Dropbox. whatever.

[jbruce12000]

once I have the log, I'll run some stats and get back to you with suggestions.

[captaincaden]

Sounds good, thank you again!

[captaincaden]

I did the update and transferred everything. Everything otherwise seems to behave normally, including the changes I made, but it seems like every profile I run shows the target temperature as 0. The kiln is still cooling at 340 degrees right now(I've never tried to do anything like this (this being starting a profile when the kiln isn't at room temp), just was testing to make sure things were working, so I'm not sure if that's normal or not, but I don't think it is), but I've made some test profiles that ramp around 340 and other temps, and still am seeing a 0 degree target temperature.

I rebooted the pi, and then tried again on the first profile, and it showed an expected target temperature, but when I switched to any of the other profiles, the target temperature went back to staying at zero. It also won't show the first profile as anything but zero anymore either.

[captaincaden]

This is what the error relative to this time frame is showing in the daemon logs. The first starting is when it showed an appropriate target temp. Then after the error all the other attempts showed the zero target temp, with no output.

Doing some tests showed that target temperature behaves normally when the kiln-controller is restarted, but fails out with that same error on any profile, once the stop command is logged, and then from that point further, the target temperature and any of the other profiles fail to function. When running the kiln-controller from the console, after that error, it will log starting when a profile is started, but does nothing after that.

In case it makes any difference, just because I see 'ispoint' referenced in a few spots, and since the line number will be different since my oven.py file included extra code, the 'ispoint' the keyerror is referencing is in the heat_then_cool in Real Oven class.

It also doesn't throw any errors or have any issues when using the simulated oven. I updated the code in a virtual machine running raspbian but can't test it with a real oven since there's no actual pi being used with the VM.

[jbruce12000]

that's a bug. I made lots of code changes over the past week so there will be some bugs. keep an eye out.

I think that one is fixed, but don't have time to test on my real kiln until this afternoon. update and try again.

[captaincaden]

That seemed to fix it. Thanks! Will keep an eye out.

Heavy storms here today, so I'm going to wait to minimize the risk of a power outage interrupting to do the test firing to cone 6 again, but will update here once I do. Thank you again for all of your help and work on this project.

[captaincaden]

I'm currently running a test firing to cone 6, with just the shelves and vent running. It would seem like the controller from the graph didn't run into the same issue at the 1050 degree hold, and the ramp up to, the hold, and the ramp out of that 1050 degree hold went smoothly. During the hold the error was max 4, but average 1-2, and prior to the hold the error was less than 0.20 the whole time , and now as we go out of the hold we were erroring around 2, and now at 1072 we're back to erroring around .5 and dropping.

So for all intents and purposes of the original issue described prior, I'd say that updating has fixed the issue stated. Let me know what you think of those numbers, or what other data you wanted to see from that time period. I'm going to let the kiln fire out to it's max temperature and will update the comment if anything happens, or if anything else happens when it's a fully packed glaze load again. But for now, I'm going to close this issue.

A screenshot of the graph during that hold.

[jbruce12000]

Looks great so far. Those numbers are pretty good. If your temps are always low, you can adjust the Integral to a lower (more aggressive) number and it will zero out faster when transitioning from one slope to another in the schedule.

So this was a totally empty kiln except for shelves?

I have a new script for posting logs in the latest master... called ziplogs. It's intended for folks to use it when they need my help troubleshooting. I still need for that to be tested. If you post your logs here after running this test, I'll take a close look.

See the end of https://github.com/jbruce12000/kiln-controller/blob/master/docs/logs.md for instructions. simple, run, post generated kiln.logs.gz here.

[captaincaden]

I had looked at that added script last night. Once it completes I'll mess around with it.

We're still headed upwards, at 1755 degrees, at the moment running error of .09 on average currently. And yes, I won't have a full glaze load until probably next weekend, so it's just all of my posts (for some added thermal mass) and all of my shelves.

My only concern regarding it is that I'm not sure how to get them to you from the pi. What does the output of the .gz file look like? is it something I can copy paste from remote or ssh? For some reason my pi really doesn't work well when trying to use chromium. It takes forever to load anything.

[jbruce12000]

you'll want to install sshd on the pi. if you're on linux, use scp to copy the file from the pi to your linux desktop. mac would be the same. if you're on windows, use putty which includes pscp.

The file is a compressed text file using gzip for the compression.

[captaincaden]

When I run ./ziplogs in cd kiln-controller, I get: -bash: ./ziplogs: No such file or directory

I know I had updated the branch prior to you adding the instructions in the log for ziplog, so I'm not sure where the script lives to create the ziplogs but I assume it's probably not something I have currently in the version I'm running? I looked through the master on here and cant seem to locate anything relative either.

[jbruce12000]

It's located in the same directory as kiln-controller.py and is available on the latest master. That was just created yesterday and that is probably why you don't have it.

[captaincaden]

Yep, was looking in the wrong spots lol.

Is ziplogs a .py file? and is it the only thing that's needed for that to work? I only ask because other than that everything else on my end is up to date, and I'd like to not re pull the whole master and have to go in and re add all of my additional codes and such if it's just that one file added to the kiln-controller directory.

Disregard that, it's an executable file. I got it working and am getting the logs over to my local computer now.

[jbruce12000]

it's just a shell script. should work fine without having to install anything. you should just be able to copy it over (like you said you did already).

[jbruce12000]

oh, btw, average error of .09 just makes me smile.

[captaincaden]

I think I did this correctly, haha. And the average error of .09 was just at the point I was watching it over the course of a few minutes. There was a short section towards the end of the firing where I noticed it got off a little bit, in the 1 or 2 I believe, but if I remember correctly it corrected itself, and got back on track until the kiln sitter eventually shut the kiln off. There was a point at the end of the firing as well where the kiln was changing a degree ahead of the target roughly half a second or so, but that also corrected itself, and really doesn't bother me much. I don't believe I ever saw the err above 3, but I wasn't watching it the entire time. Overall, even though the kiln was empty, and I'll of course update when I run my next glaze firing to see how it performs fully loaded, but I'm extremely happy with the precision I watched during the test firing yesterday. Watching both the graph, temps, and the err value every so often.

Perfect cones on all 4 shelves, almost even as well and a perfect bend in the sitter cone if that adds anything, haha.

[jbruce12000]

final.pdf

Let me know if the graphs accurately reflect the kiln run.

started = May 18 14:26:10 ended = May 18 22:15:37 average_error = 0.595527728085885 elements_on_percent = 76.6698032200358 max_temperature = 2137.4

The Integral really spiked at the end as your kiln was trying to keep up with the schedule but could not. The schedule only paused at the very end when the kiln fell behind by more than 10 degrees. I think you could prevent this by making the slope a bit more mellow. You can see the elements were at 100% near the end.

The error jittered more as the kiln got hotter. Not sure the cause. My kiln does not do this.

This was a good test of the new code and proved that both kiln_must_catch_up and pid_control_window are working as expected. This kiln is ready for some wares. :-)

[captaincaden]

Yes, I had noticed that the elements were being held on in when we were over 2050 degrees or so. I know that when it was running without the controller present, it struggled a bit in the last few hundred degrees, if I remember right, it was struggling to be able to keep around 100 degrees per hour in the last hundred degrees roughly, and my intention when making the schedule was to almost try to make it stay on 100% at the end, just because I want it to remain near that 100f/hr rate, and I know that it struggles to do so on it's own when it's manually switched to high without the controller. This is more than likely just the result of slightly underpowered and slightly unbalanced elements, that I intend to balance a little bit better on my next element change (this struggle in the last 200 degrees was way worse 4 months ago before I did some other unrelated projects to the kiln, and originally would take almost 12-14 hours with all switches on high to achieve cone 6, and after that project of reconfiguring elements and such I got it down to be able to get there in 5-6 hours on high (empty). That being said, the top of the kiln is still a bit underpowered. I intend to reconfigure when I change elements next, but these are all brand new, so I'm going to keep it going until they're worn.)

The time, and all of the data seems like it reflects what I was seeing as I checked in on it, and what I recorded in regards to starting and stopping.

I agree that it seemed like it worked well, at least on the spectrum of seeing it graphed out without any other data. I don't think the white line really ever left the green line in the graph on the websocket. And everytime i looked at the temps throughout it was following very closely as the target changed, except maybe in the last 100 degrees roughly.

I will update once I run my glaze firing with it loaded next week. My bisque firing runs a much slower schedule, and when I ran it last week totally full (before any updates) it didn't have any issues at all keeping a close track with the target temps, and also graph was perfect as well, so I expect that will also run the same or better with the updates in place. The API scheduling also worked great last week when I used it for the first time. My vent is on a smart receptacle, so I could also schedule that and the kiln to start an hour or two before I woke up to get a head start, and it worked flawlessly.

Thank you again for all of the information and all of the work you do and have done on this project. I love that you can pull this much data out of a firing and see it all graphed out. I have another digital kiln that runs along side this much older kiln, and I'd almost prefer that that one ran this setup as well. I'm incredibly happy that I decided to go even further with this project than I originally intended.

[captaincaden]

Just a follow up here:

Running a glaze firing with a 75% loaded kiln today, and thus far, did not have the issue that this issue referenced, and has been running smoothly up to now, so updating seemed to fix this issue.

Firing went well and as expected to completion in regards to this issue.