search

DrAlexLiu / OctoPrint-PiNozCam

AI model running on RPi for failure detection
GNU Affero General Public License v3.0
39 stars 4 forks source link

100% CPU on Pi 4 8Gb leads to bulbs while printing #11

Closed jofland closed 5 months ago

jofland commented 5 months ago

I installed the plugin on my Pi 4 8 Gb running

Linux octopi 6.1.21-v7l+ #1642 SMP Mon Apr  3 17:22:30 BST 2023 armv7l
OctoPrint version : 1.9.3
OctoPi version    : 1.0.0
PiNozCam (1.0.0)

I used the recomended settings for Pi 4 but the system gets to 100% CPU each time the snapshot is taken. This leads to pauses in the print head´s movement. At each pause the suface of the print gets a bulb.

When I disable the plugin the surface of the print is smooth again.

DrAlexLiu commented 5 months ago

if in this case, would you try to run with 50% CPU speed?

DrAlexLiu commented 5 months ago

Would you please also share your camera specs? Like resolution, frequency etc?

Also, do you use a cooling fan in your pi4?

I use my prusa mini+ and octoprint 1.9.3, I don't experience pause and blob.

Plus, my CPU do not run 100% all the time or the time when it gets the snapshot.

Would you please try reinstall this plugin and run with 50% cpu speed, to see whether there is an issue still exist.

jofland commented 5 months ago

Thanks for your reply. Here the additional infos:

Camera: Lenovo 500 FHD Webcam; 1920x1080; the webcam stream within octoprint seems to be 640x480; don´t know the frame rate

I don´t use a cooling fan as I have an all metal case from https://geekworm.com/products/raspberry-pi-4-heavy-duty-aluminum-passive-cooling-metal-case.

I already used 50% CPU speed.

Now I reinstalled the plugin.

Plugin config: AI Status: ON CPU Temperature: 65°C Failure Count: 0 Action after Detection: Notification Boxes Display Threshold 0.5 Image Sensitivity 0.04 Failure Scores Threshold 0.7 Max Failure Count 2 Failure Consider Time (s) 300 CPU Speed Control: Half

htop shows: load average 2.5 2 of 4 CPUs nearly the whole time at 100% CPU temperature 67 °C

==> I have blobs again.

Then I disabled the plugin: load average 0.7 no blobs CPU temperature 54 °C

DrAlexLiu commented 5 months ago

when you install, would you use the zip from there: https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/master.zip leave the custom snapshot URL blank. and 50% CPU to check blob/pausing printing happens or not.

If the problem still exist,

DO a test: put this in the custom snapshot URL:

https://raw.githubusercontent.com/DrAlexLiu/OctoPrint-PiNozCam/master/octoprint_pinozcam/static/no_camera.jpg

It will get a picture from the website, not from your printer. In this way, it will use CPU for inference, but source is that picture, not your camera. Do a print, and see blob.

Yes: If the problem still exist, is your CPU too hot? Could you try a cooling fan?

No: If the problem is gone, that means the "calling the snapshot" has issue. Because you get internet picture with no issue, but getting snapshot from your camera has issue. I have to contact the octoprint to see what is going on. My code is a standard code to call octoprint to provide the snapshot picture. https://github.com/DrAlexLiu/OctoPrint-PiNozCam/blob/master/octoprint_pinozcam/__init__.py#L501

DrAlexLiu commented 5 months ago

One more thing:

https://community.octoprint.org/t/how-can-i-change-mjpg-streamer-parameters-on-octopi/203

Here is a guideline to change the frequence and resolution. I would say changing it to 30Hz would be recommended.

Also, FYI:

my Pi4 temperature is. 50-55°C under full 100% load. 38°C under idle.

jofland commented 5 months ago

Thanks. I did some tests but could´nt fix the issue.

First I installed https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/master.zip

Test 1: custom snapshot URL blank CPU load 2.8 CPU temp 60 °C ==> blobs

Test 2: Used the static picture URL https://raw.githubusercontent.com/DrAlexLiu/OctoPrint-PiNozCam/master/octoprint_pinozcam/static/no_camera.jpg CPU load 2.3 CPU temp 61 °C ==> blobs ==> At this point it should be clear that it isn´t a camera issue. But I´ll do test 3.

Test 3:

Sorry, I cannot install a CPU fan because I don´t have any an im using the metal case. 60 °C isn´t much higher than your 50-55 °C either.

PS: The plugin did a good job in Test 3 and identified a stringing as a spaghetti error and notified me via telegram.

DrAlexLiu commented 5 months ago

Thank you for your testing.

You are right, it is not camera or snapshot issue.

May I ask you what is your printer model? Is it a 400mm/s high speed printing mode? or a slower/old one?

Would you run it with opening your metal case? or when the job finish, and open the metal case immediately and see what the hot air/heat is trapped inside the case.

To be honest, I will do a test over the weekend, run it without fan and see what is going on.

If we run out of options, there is only one solution, I write a python script to run AI on a desktop server, and the Pi is using the desktop CPU to do the AI inference, not on Arm CPU. The desktop and Pi must be on the same local network. It is OK that Pi use wifi to connect local network. I don't know this would be OK for you. Also, I probably need one month to design the desktop server.

jofland commented 5 months ago

May I ask you what is your printer model? Is it a 400mm/s high speed printing mode? or a slower/old one?

I am using an Elegoo Neptune 3 Pro running Marlin mostly 60 mm/s.

Would you run it with opening your metal case? or when the job finish, and open the metal case immediately and see what the hot air/heat is trapped inside the case.

No I won't. I don't think that it's a temperature issue. I run this pi in this case for several years as a home server with much higher loads and cpu temperatures above 80 °C without throttling.

To be honest, I will do a test over the weekend, run it without fan and see what is going on.

Just try it.

If we run out of options, there is only one solution, I write a python script to run AI on a desktop server, and the Pi is using the desktop CPU to do the AI inference, not on Arm CPU. The desktop and Pi must be on the same local network.

That's fine for me. I wouldn't run it in a VM on my local proxmox node.

It is OK that Pi use wifi to connect local network. I don't know this would be OK for you. Also, I probably need one month to design the desktop server.

If it's too much work maybe it's better to identify the current problem?

Here are some additional infos about my setup:

As there are only 2 of 4 cpu kernels are under full load I would consider a timing problem rather than a temperature problem. My printer pauses after every 5 to 10 second produsing the blobs. I had a similar problem when using the printer's power loss protection which writes the printing state to sd card every some seconds. When I switched this off the blobs had gone. But with octoprint I don't use the printer's sd card and print directly from octoprint.

DrAlexLiu commented 5 months ago

OK. I will try to run without fan. And I try to figure out which thing cause the pause. It seems running a specific script/function to cause 100% CPU load and pause the printer.

Also, my code does not write anything or image to sd card. It only write data on memory, and at end sending to telegram/discord.

DrAlexLiu commented 5 months ago

Hi @jofland

I want to confirm, 1. does your pi4 use SD card or SSD? (system on ssd, attached to usb 3 port, no sd card)

  1. If yes, I believe you also put a lot into pi4, will octoprint and its plugins directly run on arm cpu or you use a docker on pi to run the octoprint?

Also, I may think it is because of USB congestion?! You know, everytime I have to load the AI inference model from desk (35MB file). It causes slow, and may also a lot of communication between your PI and SSD. At the same time, there is no time for Pi to talk with your printer, and it causes pause for a few ms.

so later I will optimize my code, like loading everything into the memory and to do the AI inference. I will let you know when I am done. It won't even read any local files during AI inference, to avoid any I/O issue.

I should have a USB3 SSD hard drive enclosure. Let me also test it and see whether there is an issue.

Btw, personally, I would recommend to connect printer through USB2, while put camera attached to usb3. printer attached to usb 3 port camera attached to usb 2 port

DrAlexLiu commented 5 months ago

hi @jofland would you please download https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/test.zip

It is a test version. and use this zip to reinstall the plugin.

If possible, switch the ports. printer attached to usb 2 port camera attached to usb 3 port

Could you try it whether there is a blob?! running at 50% CPU.

This test version is the one that I avoid reading the model file during each inference. It would load the model in the starting plugin. In this way, the USB3 will be free and not heavy data transfer load during AI inference. Your pi will talk with your printer without delay.

Thanks.

jofland commented 5 months ago

I want to confirm, 1. does your pi4 use SD card or SSD? (system on ssd, attached to usb 3 port, no sd card)

Yes, I have everything on the ssd. I don't use a sd card. Disk IO is much faster from ssd than sd card. But maybe it's an interference on usb?

  1. If yes, I believe you also put a lot into pi4, will octoprint and its plugins directly run on arm cpu or you use a docker on pi to run the octoprint?

No, it's the octopi image from https://octoprint.org/download running octoprint directly on system. No docker. No other services.

Also, I may think it is because of USB congestion?! You know, everytime I have to load the AI inference model from desk (35MB file). It causes slow, and may also a lot of communication between your PI and SSD. At the same time, there is no time for Pi to talk with your printer, and it causes pause for a few ms.

That could be a possibility.

so later I will optimize my code, like loading everything into the memory and to do the AI inference. I will let you know when I am done. It won't even read any local files during AI inference, to avoid any I/O issue.

Since there should be plenty of RAM even on the smallest 2 Gb pi this should be a good idea. Mine has 8 Gb of RAM.

I should have a USB3 SSD hard drive enclosure. Let me also test it and see whether there is an issue.

Just try it.

Btw, personally, I would recommend to connect printer through USB2, while put camera attached to usb3. printer attached to usb 3 port camera attached to usb 2 port

Printer on usb 3 was recommended by the official documentation because of real time IO, but I can switch for testing purposes.

jofland commented 5 months ago

hi @jofland would you please download https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/test.zip

It is a test version. and use this zip to reinstall the plugin.

If possible, switch the ports. printer attached to usb 2 port camera attached to usb 3 port

Could you try it whether there is a blob?! running at 50% CPU.

This test version is the one that I avoid reading the model file during each inference. It would load the model in the starting plugin. In this way, the USB3 will be free and not heavy data transfer load during AI inference. Your pi will talk with your printer without delay.

Thanks.

Thanks, I'll try tomorrow evening.

jofland commented 5 months ago

Here are the test results:

Test 1: Version from https://github.com/DrAlexLiu/OctoPrint-PiNozCam/issues/11#issuecomment-2004202453 USB 3: SSD and camera USB 2: printer CPU Speed Control: Half ==> Result: failed Load Average 2.8 - 3.0 Printer pauses and prints blobs

Test 2: Version from yesterday https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/test.zip USB 3: SSD and camera USB 2: printer CPU Speed Control: Half ==> Result: successful Load Average 2.8 - 3.0 Printer doesn´t pause and no blobs

Test 3: Version from yesterday https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/test.zip USB 3: SSD and printer USB 2: camera CPU Speed Control: Half ==> Result: successful Load Average 2.6 - 3.0 Printer doesn´t pause and no blobs

Test 4: Version from yesterday https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/test.zip USB 3: SSD and printer USB 2: camera CPU Speed Control: Full ==> Result: successful Load Average 3.6 - 4.0 Printer doesn´t pause and no blobs

Conclusion:

DrAlexLiu commented 5 months ago

https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/test.zip

So this test.zip will work, so I will merge the test into the mater branch over the weekend, after QA test on other platforms etc.

test.zip is the one avoiding repeated loading the large data from hard drive. So, I am glad it works on your end. Besides, you can continue to use test.zip, it is tested and working on pi4, I must make sure it can run on other platforms.

I will leave this ticket open until I merge everything into the master branch. I will let you know when to use the latest stable version. Then you can use master version.

Besides, leave the CPU at 50%. I find out the speed is quite same for both 50% cpu and 100% cpu.

DrAlexLiu commented 5 months ago

Now you can use the latest stable version.

https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/master.zip

if you have other issue or this issue happens again, please feel free to open a ticket

jofland commented 5 months ago

Thanks, I'll try the other week.

jofland commented 5 months ago

Now you can use the latest stable version.

https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/master.zip

if you have other issue or this issue happens again, please feel free to open a ticket

Version ist working for me.

DrAlexLiu commented 5 months ago

@jofland

I am still new to the Octoprint plugin and version control. Usually, you can download the

https://github.com/DrAlexLiu/OctoPrint-PiNozCam/archive/master.zip

and every several weeks, I will update to a new version number.