INTERNALSERVER ERROR 500

[Prasadevops0710]

[Prasadevops0710]

please help here @deniszh

[deniszh]

Hi @Prasadevops0710

You need uncomment FETCH_TIMEOUT (and probably FIND_TIMEOUT also) in local_settings.py file and put some sane value there. Default is 6.0, means 6 seconds, you can e.g. double it, so, put 12 seconds there.

[Prasadevops0710]

I've made changes in setting.py file now its working fine but in grafana the dashboard loading slowly it taking 10 sec to load dash board

[Prasadevops0710]

thank you for your response and am new to graphite thing and please let me know can i change in localsetting.py also @deniszh

[Prasadevops0710]

when am checking logs 2022-03-06,12:25:30.221 :: Failed CarbonLink query 'rtb-test.st.aggregated.accounts.54db32e4_4d94_41b5_8152_848cf8b1ab6b.9fb7e69e_f681_4071_b89c_8c11465c78a5.submitted' Traceback (most recent call last): File "/data/graphite/webapp/graphite/readers/utils.py", line 110, in merge_with_carbonlink cached_datapoints = CarbonLink().query(metric) File "/data/graphite/webapp/graphite/carbonlink.py", line 97, in query results = self.send_request(request) File "/data/graphite/webapp/graphite/carbonlink.py", line 130, in send_request conn = self.get_connection(host) File "/data/graphite/webapp/graphite/carbonlink.py", line 87, in get_connection connection = socket.create_connection((server, port), self.timeout) File "/usr/lib/python2.7/socket.py", line 575, in create_connection raise err error: [Errno 111] Connection refused

@deniszh

[deniszh]

@Prasadevops0710

please let me know can i change in localsetting.py also

If you changed settings.py that doesn't matter

error: [Errno 111] Connection refused

what's in your CARBONLINK_HOSTS parameter? Do you use carbon or go-carbon ? Is in installed on same server?

[Prasadevops0710]

I’m using gocarbon installed in same server and any config I need to change in localsetting.py am I right? @deniszh

[Prasadevops0710]

this is my localsetting.py

[Prasadevops0710]

some times some of panels showing no data but when I'm refreshing dashboard the panels are showing metrics

[deniszh]

@Prasadevops0710

Uncomment CARBONLINK_HOSTS in local_settings.py and make sure that carbonlink section is present in go-carbon.conf:

[carbonlink]
listen = "127.0.0.1:7002"
enabled = true
# Close inactive connections after "read-timeout"
read-timeout = "30s"

Also you can try to switch to carbonserver proto. Enable it in go-carbon.conf:

[carbonserver]
listen = "127.0.0.1:8080"
enabled = true
....

and then add

CLUSTER_SERVERS=["127.0.0.1:8080"]

to local_settings.py and restart both go-carbon and graphite-web.

[Prasadevops0710]

● go-carbon.service - Go-carbon Loaded: loaded (/etc/systemd/system/go-carbon.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Wed 2022-03-09 17:27:08 UTC; 4s ago Process: 20402 ExecStart=/bin/sh -c go-carbon -config /data/graphite/conf/go-carbon.conf (code=exited, status=1/FAILURE) Main PID: 20402 (code=exited, status=1/FAILURE)

Mar 09 17:27:08 ip-10-5-102-84 systemd[1]: go-carbon.service: Unit entered failed state. Mar 09 17:27:08 ip-10-5-102-84 systemd[1]: go-carbon.service: Failed with result 'exit-code'.

[Prasadevops0710]

I've enabled carbon server and restarted go-carbon but its always in loaded state as above @deniszh

[Prasadevops0710]

CorruptWhisperFile: Unable to read header (/data/graphite/storage/whisper/rtb-test/st/aggregated/accounts/54db32e4_4d94_41b5_8152_848cf8b1ab6b/9fb7e69e_f681_4071_b89c_8c11465c78a5/bidUnderDealFloor.wsp) 18442
18443
18444 Traceback (most recent call last): 18445 File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 185, in _get_response 18446 response = wrapped_callback(request, *callback_args, *callback_kwargs) 18447 File "/data/graphite/webapp/graphite/render/views.py", line 118, in renderView 18448 data.extend(evaluateTarget(requestContext, targets)) 18449 File "/data/graphite/webapp/graphite/render/evaluator.py", line 15, in evaluateTarget 18450 prefetchData(requestContext, pathExpressions) 18451 File "/data/graphite/webapp/graphite/render/datalib.py", line 263, in prefetchData 18452 for result in STORE.fetch(pathExpressions, startTime, endTime, now, requestContext): 18453 File "/data/graphite/webapp/graphite/storage.py", line 190, in fetch 18454 'fetch for %s' % str(patterns)) 18455 File "/data/graphite/webapp/graphite/storage.py", line 132, in wait_jobs 18456 raise Exception(message) 18457 Exception: All requests failed for fetch for [u'rtb-test.st.aggregated.accounts.54db32e4_4d94_41b5_8152_848cf8b1ab6b.9fb7e69e_f681_4071_b89c_8c11465c78a5.'] (1) 18458
18459 fetch for [u'rtb-test.st.aggregated.accounts.54db32e4_4d94_41b5_8152_848cf8b1ab6b.9fb7e69e_f681_4071_b89c_8c11465c78a5.']: Unable to read header (/data/graphite/storage/whisper/rtb-test/st/aggregated/accounts/54db32e4_4d94_41b5_8152_848cf8b1ab6b/9fb7e69e_f681_4071_b89c_8c11465c78a5/bidUnderDealFloor.wsp): Traceback (most recent call last): 18460
18461 File "/data/graphite/webapp/graphite/worker_pool/pool.py", line 43, in run 18462 self.result = self.func(self.args, **self.kwargs) 18463
18464 File "/data/graphite/webapp/graphite/finders/utils.py", line 140, in fetch 18465 now=now, requestContext=requestContext 18466
18467 File "/data/graphite/webapp/graphite/node.py", line 30, in fetch 18468 result = self.reader.fetch(startTime, endTime, now, requestContext) 18469
18470 File "/data/graphite/webapp/graphite/readers/whisper.py", line 59, in fetch 18471 data = self.fetch_data(startTime, endTime, now=now) 18472
18473 File "/data/graphite/webapp/graphite/readers/whisper.py", line 55, in fetch_data 18474 return whisper.fetch(self.fs_path, startTime, endTime, now=now) 18475
18476 File "/usr/local/lib/python2.7/dist-packages/whisper-1.2.0-py2.7.egg/whisper.py", line 906, in fetch 18477 return file_fetch(fh, fromTime, untilTime, now, archiveToSelect) 18478
18479 File "/usr/local/lib/python2.7/dist-packages/whisper-1.2.0-py2.7.egg/whisper.py", line 910, in file_fetch 18480 header = readHeader(fh) 18481
18482 File "/usr/local/lib/python2.7/dist-packages/whisper-1.2.0-py2.7.egg/whisper.py", line 290, in readHeader 18483 raise CorruptWhisperFile("Unable to read header", fh.name) 18484
@deniszh can you help here

[deniszh]

@Prasadevops0710 : File /data/graphite/storage/whisper/rtb-test/st/aggregated/accounts/54db32e4_4d94_41b5_8152_848cf8b1ab6b/9fb7e69e_f681_4071_b89c_8c11465c78a5/bidUnderDealFloor.wsp is corrupted and can't be read. Please delete it.

[Prasadevops0710]

@deniszh but it seems that there are so many corrupted file around 1000 can I delete all or is there any way to prevent that in future. thank you

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.