search

celery / billiard

Multiprocessing Pool Extensions
Other
419 stars 252 forks source link

worker unable to determine memory usage #228

Closed listingmirror closed 6 years ago

listingmirror commented 7 years ago

Python 3.6.1 Billiard 3.5.0.2

In docker on an image based on 3.6-alpine

2017-06-26T21:43:35.854019324Z [2017-06-26 21:43:35,853: ERROR/PoolWorker-7] worker unable to determine worker memory usage

Any ideas why I am hitting this? https://github.com/celery/billiard/blob/bfe2dc6387853595ef8877809c1404fe4959519d/billiard/pool.py#L377

Edit - Investigated on the alpine docker image directly.

I can run both compat.mem_rss()and getrusage(RUSAGE_SELF).ru_maxrss without a problem. So very weird that they work directly in docker on the same image I run, but not when actually running via celery..

>>> getrusage(RUSAGE_SELF).ru_maxrss
64520

Edit 2:

I even added the following line to a celery task

logger.info('Memory: %s', mem_rss())

And it works. So clearly mem_rss() is good.. so how the heck do I get this error then,...

listingmirror commented 7 years ago

Grrr this is just https://github.com/celery/billiard/pull/207

Can we get a release please ;)

arizhakov commented 5 years ago

@auvipy , do you have a recommendation of which billiard version to use to avoid this?

From https://github.com/celery/billiard/pull/207/commits/8f8e958915e062c356697c171c6f4ec3213b02e6, I see that this was fixed in master on Dec 15, 2016. However, billiard 3.5 (the latest release) looks like it still has the bug: https://github.com/celery/billiard/blob/40d449d20264925b89b84e6f1425de83babccc5a/billiard/pool.py#L383-L384

Is there a ballpark approximation for when a new release of billiard will happen that has this bug fix?

arizhakov commented 5 years ago

I have answered my own question. v3.5.0.5 was released 2 months ago, and has the buggy else clause removed. Also, it helps if one follows the installation docs properly :) (i.e., install celery with pip).