100 % cpu load

[n800sau]

It fully loads CPU even while idle.

[MrYsLab]

Could you please provide:

1. A sample program demonstrating your issue.
2. The version number of Python you are using.
3. The Operating system and version number you are using.
4. The CPU Model and version number you are using.
5. The amount of RAM your system has.
6. Are you concurrently running any other programs?

To see typical performance results, please go to http://mryslab.blogspot.com/ and search for the article entitled "Asyncio is Coming to PyMata!" for a sample program that was run on both PyMata and pymata_aio. The code and CPU utilization charts are provided for both verisons of pymata.

Please be aware that the more pins you have configured as INPUT or ANALOG pins with data that is changing, the more data that Firmata (on the Arduino) will generate, which will affect CPU utilization.

[n800sau]

1. pymata_blink.py after it starts blinking
2. 2.7.3
3. raspbian 4.BCM2708 5.256M 6 No

I've found some solution. There are problems in pymata_serial.py inside of run loop as well as in run in pymata_command_handler.py. I changed the first to have

if self.arduino.inWaiting():

c = self.arduino.read() if len(c) > 0: self.command_deque.append(ord(c))

And in the end of the second I have added else: time.sleep(0.1)

[MrYsLab]

@n800sau Thanks for the fix. It was added and is available in v2.10.

[gengshenghong]

@MrYsLab ,

I have got this issue on my environment and I am confused that "can this sleep resolve the high CPU load issue?".

Basically, my environment is a Ubuntu 14.04 with python 2.7.6 by default, I used the standard firmata on Ardunio. I got very high CPU for the Python process, the code is really simple, do nothing after reset the board (I got the issue in my project and trying to isolate the issue and find even such test case can reproduce). Later, I find this thread and trying to understand what the issue is, and now I am clear why it consumes 100% CPU, as the 2 loop will keep running always, which will take full use of CPU, even you added a sleep with 0.1, I do not think it can reduce much (from about 110% to 100% maybe). To prove my understanding, I changed both sleep to time.sleep(100), and now, CPU will not be high, but definitely, this is not a fix as it means I cannot get timely response.

Is this by design and do you have any idea to avoid it? If taking such high CPU (even I remove my serial cable, it will keep running in the loop), it seems really bad.

Appreciate your reply (sorry for my bad English, hope it is clear...)

Thanks, Shenghong

The code:

! /usr/bin/evn python

''' debug the high CPU issue.... '''

from PyMata.pymata import PyMata import time

def test(): board = PyMata("/dev/ttyUSB0")

print "start..."
print board.get_firmata_version()
print board.get_firmata_firmware_version()
print board.get_pymata_version()

print "reset..."
board.reset()
print "reset done..."

while True:
    print "test..debug why CPU is high..."
    time.sleep(100)

test()

[MrYsLab]

@gengshenghong If you are willing to install and use Python 3.5, I would suggest using pymata-aio, that is much more CPU efficient than PyMata.

If that is not acceptable, I will make the following comments.

1. If you are running a multi-core processor (I think you are, since you mention that you saw CPU utilization of 110%. The percentage is total across all cores.), you should be able to run other applications simultaneously without issue. The high CPU utilization should not limit the functionality of your PyMata application.
2. PyMata needs to continuously monitor the serial port for data changes. This is to allow for immediate updates to its database and to call any registered callbacks. Pyserial does not provide an event driven interface, and so it must be polled. Hence the high CPU utilization.
3. Python cannot run multiple threads across multiple cores. This is a limitation due to the Python GIL. PyMata uses a threading concurrency model.

pymata-aio on the other hand, does not use a threading concurrency model, but uses asyncio and does not suffer from this limitation.

If you have any other questions or comments, please let me know.

[gengshenghong]

@MrYsLab Thank you for your confirmation. You are correct that it does not affect much for my current machine I am testing, but I expect the code to run on more low level machines like Raspberry pi in final project. :)

Regarding the 2nd comment you mentioned, I've think a little bit more your code yesterday, it seems like a possible solution is that, for serial run loop, you may invoke "read" without "inWatiing()", as "read" will blocked until data comes, and then, regarding the loop to handle the commands, you may use "Queue" instead of "queue" you are using now, with the "get" method of "Queue", it will also block until data comes. This is just my rough idea regarding the issue. But I guess Queue will have worse performance than queue, I think it has some internal lock/sync mechanism and that's why it can be used in blocking mode (yes, another solution may be using lock/sync mechanism, I am not sure what is provided in Python for threading stuff).

Upgrading to Python3.5 may be best, but I need some evaluation as I do not know whether all my other code (dependencies) is ok to run on Python3.5. It is still a pain to decide between Python2.x and Python3.x.

Thanks, Shenghong

[gengshenghong]

@MrYsLab Sorry, there are something wrong in my last test....Ignore that (adding sleep to 100 will fail to detect board, not sure why it passed yesterday! maybe because I am too sleepy and something with my eye!!!).

I've carefully tested today, and I do find an issue in your master code for the fix, in "pymata_command_handler.py", the sleep is inside "if" part, you need to move it out of "if" and make it parallel with "if" statement, which seems like a minor mistake when you submit the code. With this correct sleep adde, it do resolve the issue (I originally thought sleep(.1) means sleep 0.1ms so get the conclusion that it should not resolve it, but in fact it is 100ms, and I agree it can lower the CPU usage now)

From (your current code):

while not self.is_stopped():
            if len(self.pymata.command_deque):
                 ...other code....
                else:
                    time.sleep(.01)

To:

while not self.is_stopped():
            if len(self.pymata.command_deque):
                  ...other code....
            else:
                time.sleep(.01)

Also, if I change both 2 sleep in pymata_serial.py and pymata_command_handler.py to 0.001s, the CPU load is about 10% to 15% only (Note: just the simple code, no analog reporting, but I guess even with reporting, the load will not be higher as the sleep is still there). For people who need a little higher real-time, using 0.001s may be enough. Or, use 0.01s or 0.1s accordingly (less than 2% CPU in my system).

Hope it is helpful for other users reading this thread.

Thanks, Shenghong

[MrYsLab]

Thanks for your suggestions. I will look at possibly making your suggested changes some time in the future.

[MrYsLab]

@gengshenghong I implemented your suggestion for release 2.11. Thanks for your input

[gengshenghong]

:+1: