search

graphite-project / carbon

Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
http://graphite.readthedocs.org/
Apache License 2.0
1.5k stars 490 forks source link

carbon-cache cpu usage 100% when sent metric with too big name #89

Closed kaarelk closed 9 years ago

kaarelk commented 11 years ago

Carbon-cache will eat up all cpu when following error happens:

exceptions.IOError: [Errno 36] File name too long:

It should discard the metric instead.

rowanu commented 11 years ago

Please add some more details, like:

If there's an actual stack-trace message, please include it.

Thanks!

kaarelk commented 11 years ago

Running Centos 5.8, python 2.4 with latest graphite 0.9.10 with whisper backend.

Error message in console.log:

28/03/2013 12:41:23 :: Sorted 8771 cache queues in 0.002869 seconds 28/03/2013 12:41:23 :: Unhandled Error Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 422, in run self.target(_self.__args, *_self.kwargs) File "/usr/lib64/python2.4/site-packages/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, _args, _kwargs) File "/usr/lib64/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, _args, _kw) File "/usr/lib64/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(args,*kw) --- --- File "/opt/graphite/lib/carbon/writer.py", line 158, in writeForever writeCachedDataPoints() File "/opt/graphite/lib/carbon/writer.py", line 118, in writeCachedDataPoints whisper.create(dbFilePath, archiveConfig, xFilesFactor, aggregationMethod, settings.WHISPER_SPARSE_CREATE) File "/usr/lib/python2.4/site-packages/whisper.py", line 327, in create fh = open(path,'wb') exceptions.IOError: [Errno 36] File name too long: /opt/local/graphite.......

jbrinegar commented 11 years ago

I use a lengthy namespace. @kaarelk might you share the full file name? Feel free to obfuscate when needed.

kaarelk commented 11 years ago

It's easy to reproduce. e.g:

File "/usr/lib/python2.4/site-packages/whisper.py", line 327, in create fh = open(path,'wb') exceptions.IOError: [Errno 36] File name too long: '/opt/local/graphite/storage/whisper/test/test-asdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf-asdfadfasdfasdfasdf/0/24HourRate.wsp'

arron-green commented 10 years ago

I am dealing with the same issue as well:

09/10/2013 02:25:19 :: Sorted 122859 cache queues in 0.115664 seconds 09/10/2013 02:25:19 :: /etc/carbon/storage-aggregation.conf not found, ignoring. 09/10/2013 02:26:19 :: /etc/carbon/storage-aggregation.conf not found, ignoring. 09/10/2013 02:26:22 :: Unhandled Error Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 484, in run self.target(_self.__args, *_self.kwargs) File "/usr/lib64/python2.6/site-packages/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, _args, _kwargs) File "/usr/lib64/python2.6/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, _args, _kw) File "/usr/lib64/python2.6/site-packages/twisted/python/context.py", line 37, in callWithContext return func(args,*kw) --- --- File "/usr/lib/python2.6/site-packages/carbon/writer.py", line 158, in writeForever writeCachedDataPoints() File "/usr/lib/python2.6/site-packages/carbon/writer.py", line 118, in writeCachedDataPoints whisper.create(dbFilePath, archiveConfig, xFilesFactor, aggregationMethod, settings.WHISPER_SPARSE_CREATE) File "/usr/lib/python2.6/site-packages/whisper.py", line 327, in create fh = open(path,'wb') exceptions.IOError: [Errno 36] File name too long: '/var/lib/graphite/storage/whisper/stats/foo/production/routes/_01vp01_01_012_011729158880_012589977024_01_01_011000_011282_01_01_01_01_01_01_01_01_01_01_01_01_01_01Safari_01an_20unknown_20version_01an_20unknown_20OS_01_2Fpic_100740499_2Fxxxx_xxxx_xxxxx_xxxxx_sdsd_xx_x_xxx_x_html_01_01_0113807324433940231133_01popular_0114_194_242_255_01IN_01en_01http_3A_2F_2Fwww_google_com_2Fm_2Fsearch_3Fclient_3Dmobile_samsung_bada_26channel_3Dmm_26q_3Dxxxxxx_2Bxxxxxx_01221_3D886_3B503_3D1700_3B332_3D1160_3B389_3D1382_3B230_3D958_3B218_3D872_3B341_3D1193_3B115_3D466_3B454_3D1558_3B377_3D1334_3B430_3D1486_3B158_3D572_3B212_3D815_3B461_3D1580_3B485_3D1664_01_01_01_012299cae95e368ec528035aa365aadf83_01_01_01_01_01_01_01_png/request.wsp'

kaarelk commented 10 years ago

The pull request is a workaround for occasional invalid metric names. You should always try to send valid names. Note that most of the filesystems have 255 character limit for file names.

arron-green commented 10 years ago

thanks @kaarelk

arron-green commented 10 years ago

for anyone else having this issue we got around it by using a blacklist entry [\w-]{256,}

kaarelk commented 10 years ago

Blacklist wastes a lot of cpu

lazybios commented 9 years ago

I encountered the same problem. And it will cause the graphite webapp crash.

27/09/2014 23:57:28 :: "[Errno 36] File name too long: '/data1/graphite/storage/whisper/stats/openapi/dm_web/host/oweb004/dm/wbapi/yf/sinanode/com/api/-2-direct_messages-user_list_jsoncount25count25count25uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667'" 27/09/2014 23:57:28 :: Unhandled Error Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib64/python2.6/site-packages/twisted/python/threadpool.py", line 207, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib64/python2.6/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib64/python2.6/site-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/opt/graphite/lib/carbon/writer.py", line 149, in writeForever writeCachedDataPoints() File "/opt/graphite/lib/carbon/writer.py", line 121, in writeCachedDataPoints settings.WHISPER_FALLOCATE_CREATE) File "/usr/lib/python2.6/site-packages/whisper.py", line 363, in create fh = open(path,'wb') exceptions.IOError: [Errno 36] File name too long: '/data1/graphite/storage/whisper/stats/openapi/dm_web/host/oweb004/dm/wbapi/yf/sinanode/com/api/-2-direct_messages-user_list_jsoncount25count25count25uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667uid1788775667/400.wsp'