search

graphite-project / graphite-web

A highly scalable real-time graphing system
http://graphite.readthedocs.org/
Apache License 2.0
5.89k stars 1.26k forks source link

Graphite Not showing data from all nodes #2715

Closed AshishVerma7690 closed 2 years ago

AshishVerma7690 commented 3 years ago

I am using graphite with a 5 node cluster and Replication factor 2. I am using grafana on top of it. Below are my cluster settings.

REMOTE_EXCLUDE_LOCAL = True CLUSTER_SERVERS = ['xxx.xx.xx.71:8081', 'xxx.xx.xx.61:8081', 'xxx.xx.xx.72:8081', 'xxx.xx.xx.73:8081', 'xxx.xx.xx75:8081'] REMOTE_RETRY_DELAY = 0 REMOTE_STORE_MERGE_RESULTS=True

While fetching data I am getting random data.. for the first time I am getting 8 data points and if I refresh it then I am getting only 2 data points. Again if I refresh the dashboard I am getting 8 data points and after that again 2 data points.

Is there any other setting I need to do for getting consistent data?

image

After refresh image

Also, it is showing the below error in logs.

HTTPConnectionPool(host='172.16.2.52', port=8081): Max retries exceeded with url: /render/?format=pickle&local=1&noCache=1&from=1630305919&until=1630309519&target=

 (Caused by ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))

Expected behavior The data should be consistent and merge from all nodes

Environment

deniszh commented 3 years ago

Hi @AshishVerma7690

What's installed on these hosts on port 8081? Graphite-web or something else? How data is distributed, by carbon relay, carboc-c-relay? What's the config of carbon?

AshishVerma7690 commented 3 years ago

Hi @deniszh 8081 is used for graphite-web. Data is distributed by carbon relay. Below is my carbon conf.

[aggregator]
DESTINATIONS = 172.16.2.52:2014,172.16.1.72:2014
FORWARD_ALL = False
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2023
LOG_LISTENER_CONNECTIONS = False
MAX_AGGREGATION_INTERVALS = 5
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2024
REPLICATION_FACTOR = 1
USE_FLOW_CONTROL = True

[cache]
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002
CACHE_WRITE_STRATEGY = sorted
ENABLE_LOGROTATION = True
ENABLE_TAGS = False
ENABLE_UDP_LISTENER = True
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013
LOCAL_DATA_DIR = /bp2/data/graphite/storage/whisper/
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = False
LOG_CREATES = False
LOG_DIR = /bp2/data/graphite/storage/log/
LOG_LISTENER_CONNECTIONS = False
LOG_UPDATES = False
MAX_CACHE_SIZE = inf
MAX_CREATES_PER_MINUTE = 1000
MAX_UPDATES_PER_SECOND = 1000
MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = 1000
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
PID_DIR = /dev/shm
STORAGE_DIR = /bp2/data/graphite/storage/
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2013
USER =
USE_FLOW_CONTROL = True
USE_INSECURE_UNPICKLER = False
WHISPER_AUTOFLUSH = False
WHISPER_FALLOCATE_CREATE = True
WHITELISTS_DIR = /bp2/data/graphite/storage/lists/

[relay]
DESTINATIONS = 172.16.2.52:2014,172.16.1.72:2014
ENABLE_UDP_LISTENER = True
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
LOG_LISTENER_CONN_SUCCESS = False
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 30000
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
QUEUE_LOW_WATERMARK_PCT = 0.8
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 2
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003
USE_FLOW_CONTROL = True
USE_WHITELIST = True
deniszh commented 3 years ago

But why relay have only 2 destinations but you have 5 servers in Graphite cluster?

AshishVerma7690 commented 3 years ago

@deniszh Sorry, we have downgraded the graphite to 2 instances now. I have given you that setting. Below are the carbon conf with 5 instances

[aggregator]
DESTINATIONS = 172.16.2.52:2014,172.16.0.54:2014,172.16.1.72:2014,172.16.2.73:2014,172.16.0.75:2014
FORWARD_ALL = False
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2023
LOG_LISTENER_CONNECTIONS = False
MAX_AGGREGATION_INTERVALS = 5
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 10000
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2024
REPLICATION_FACTOR = 1
USE_FLOW_CONTROL = True

[cache]
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002
CACHE_WRITE_STRATEGY = sorted
ENABLE_LOGROTATION = True
ENABLE_TAGS = False
ENABLE_UDP_LISTENER = True
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013
LOCAL_DATA_DIR = /bp2/data/graphite/storage/whisper/
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = False
LOG_CREATES = False
LOG_DIR = /bp2/data/graphite/storage/log/
LOG_LISTENER_CONNECTIONS = False
LOG_UPDATES = False
MAX_CACHE_SIZE = inf
MAX_CREATES_PER_MINUTE = 1000
MAX_UPDATES_PER_SECOND = 1000
MAX_UPDATES_PER_SECOND_ON_SHUTDOWN = 1000
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
PID_DIR = /dev/shm
STORAGE_DIR = /bp2/data/graphite/storage/
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2013
USER =
USE_FLOW_CONTROL = True
USE_INSECURE_UNPICKLER = False
WHISPER_AUTOFLUSH = False
WHISPER_FALLOCATE_CREATE = True
WHITELISTS_DIR = /bp2/data/graphite/storage/lists/

[relay]
DESTINATIONS = 172.16.2.52:2014,172.16.0.54:2014,172.16.1.72:2014,172.16.2.73:2014,172.16.0.75:2014
ENABLE_UDP_LISTENER = True
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
LOG_LISTENER_CONN_SUCCESS = False
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_QUEUE_SIZE = 30000
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
QUEUE_LOW_WATERMARK_PCT = 0.8
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 2
UDP_RECEIVER_INTERFACE = 0.0.0.0
UDP_RECEIVER_PORT = 2003
USE_FLOW_CONTROL = True
USE_WHITELIST = True
deniszh commented 3 years ago

Ah. Sorry then, I have no ideas why data could be inconsistent, or why do you have errors in logs as above.

AshishVerma7690 commented 3 years ago

Some error logs:

2021-08-31,01:58:14.047 :: Exception during fetch for ['*.kafka.*.kafka.topicOffsets.{BPCORRELATOR,ConfigMgmt,Discovery,IFD,NSI,PINPOINT,RaProviderEventManager-0,RaProviderEventManager-1,RaProviderEventManager-2,TDC,aeprocessor,aeprocessor0,aeprocessor1,api-gw_0,api-gw_1,asset-area-ansible,asset-area-bpmn,asset-area-generic,asset-area-model-definitions,asset-area-terraform,bp_commissioning,chronos,com_cyaninc_bpocore_auth_troneventmonitor,crinoidapi_kompressor,equipmenttopologyplanning,fregraph,invhist_username_cache_0,invhist_username_cache_1,kompressor,mcpviewalarmcorrelator,perfg-monitor-servicetrail,perfg-monitor-viability,perfg-oms,perfg-plannedfiberloss,perfg-scheduler-ppg,perfg_equipment_v1,perfg_equipmentstaticspec_v1,perfg_fre_v1,perfg_nc_v1,perfg_tpe_v1,planningintent,pm-discovery,pm-ingestion,pm-metrics_34741333,pm-metrics_666918844,pm-nsi,pm-scheduler,pm-thresholds,pmp-derived,pmprocessor-0,pmprocessor-1,pmprocessor_derived_943239074,pmprocessor_derived_979700869,pmprocessor_threshold_43780691,pmprocessor_threshold_57930108,policy,projectmanagement,session-event-monitors,svchist_pipeline,tpegraph,viability,watcher}.{bp_aeprocessor_v1_alarmsforcorrelation,bp_aeprocessor_v1_filters_pool,bp_aeprocessor_v1_notifyhosts,bp_aeprocessor_v1_notifytapi,bp_aeprocessor_v1_operations,bp_aeprocessor_v2_0_alarms,bp_aeprocessor_v2_geo_sync,bp_bpoifd_v1_changes_services,bp_ciena_watcher_v2_geo,bp_ciena_watcher_v2_geo_in,bp_config_api_v1_backup,bp_config_api_v1_restore,bp_config_api_v1_upgrade,bp_correlation_v1_changes_alarms,bp_correlation_v1_changes_device,bp_correlation_v1_changes_inventoryRelationship,bp_crinoid_v1_kompressor,bp_crinoid_v1_kompressor_in,bp_discovery_v1_alarms,bp_discovery_v2_0_changes_discoverySyncRequest,bp_discovery_v2_0_changes_sync,bp_discovery_v2_geo,bp_discovery_v2_geo_in,bp_discovery_v3_changes_managementSession,bp_gcs_v2_kompressor,bp_hpmra_v2_pm,bp_ifd_v3_0_srlgJobs,bp_kafkacomet_v1_channel_administration,bp_licensing_v1_geo_sync,bp_nms_notreplicated_v1_alarms,bp_nms_replicated_v1_alarms,bp_nsi_v1_alarms,bp_nsi_v1_changes_equipmentStaticSpec,bp_nsi_v1_changes_sync,bp_nsi_v1_internal_changes_inventory,bp_nsi_v1_internal_group,bp_nsi_v1_internal_stitch,bp_nsi_v1_internal_utilization,bp_nsi_v2_0_changes_nc,bp_nsi_v2_0_validation,bp_nsi_v3_changes_equipment,bp_nsi_v3_changes_fre,bp_nsi_v3_changes_tpe,bp_nsi_v3_validation,bp_nsi_v5_changes_geo_in,bp_perfg_v1_scheduler_ppg,bp_perfg_v2_monitor_servicetrail,bp_perfg_v2_monitor_viability,bp_pinpoint_v1_otdr_state,bp_pm_v1_geo_sync,bp_pmp_v1_pm,bp_pmprocessor_v1_datastore,bp_projectmanagement_v2_cancelled,bp_projectmanagement_v2_commit,bp_projectmanagement_v2_deleted,bp_projectmanagement_v2_list_deleted,bp_projectmanagement_v2_list_projectadded,bp_projectmanagement_v2_list_statechange,bp_projectmanagement_v2_networkchanged,bp_projectmanagement_v2_projectvalidate,bp_projectmanagement_v2_statechange,bp_ra_v1_EroEvent,bp_ra_v1_SelectiveSync,bp_ra_v1_alarms,bp_ra_v1_diagnostics_loopback,bp_ra_v1_nodecommissioning,bp_ra_v1_pm,bp_ra_v1_pm_thresholds,bp_ra_v1_poke,bp_ra_v2_0_sync,bp_rasdk_v1_changes,bp_stitcher_v1_changes_iplinkfre,bp_stitcher_v2_changes_tpe,bp_stitcher_v2_patch_fre,bp_stitcher_v3_changes_fre,bp_tdc_v1_tests,bp_topologyplanning_v1_planneddata_fiberloss_changed,bp_topologyplanning_v1_planneddata_omssnr_changed,bp_tron_v1_changes,bp_tron_v1_kompressor,bp_view_alarmcorrelation_publish,chronos_dispatch_queue,chronos_scheduling_queue,chronos_sharding_queues_0,chronos_sharding_queues_1,com_cyaninc_bp_market,cr,freg2svchist,pm_scheduling}.committed'] after 0.801729s: Error requesting http://172.16.0.54:8081/render/?format=pickle&local=1&noCache=1&from=1630367893&until=1630389493&target=%2A.kafka.%2A.kafka.topicOffsets.%7BBPCORRELATOR%2CConfigMgmt%2CDiscovery%2CIFD%2CNSI%2CPINPOINT%2CRaProviderEventManager-0%2CRaProviderEventManager-1%2CRaProviderEventManager-2%2CTDC%2Caeprocessor%2Caeprocessor0%2Caeprocessor1%2Capi-gw_0%2Capi-gw_1%2Casset-area-ansible%2Casset-area-bpmn%2Casset-area-generic%2Casset-area-model-definitions%2Casset-area-terraform%2Cbp_commissioning%2Cchronos%2Ccom_cyaninc_bpocore_auth_troneventmonitor%2Ccrinoidapi_kompressor%2Cequipmenttopologyplanning%2Cfregraph%2Cinvhist_username_cache_0%2Cinvhist_username_cache_1%2Ckompressor%2Cmcpviewalarmcorrelator%2Cperfg-monitor-servicetrail%2Cperfg-monitor-viability%2Cperfg-oms%2Cperfg-plannedfiberloss%2Cperfg-scheduler-ppg%2Cperfg_equipment_v1%2Cperfg_equipmentstaticspec_v1%2Cperfg_fre_v1%2Cperfg_nc_v1%2Cperfg_tpe_v1%2Cplanningintent%2Cpm-discovery%2Cpm-ingestion%2Cpm-metrics_34741333%2Cpm-metrics_666918844%2Cpm-nsi%2Cpm-scheduler%2Cpm-thresholds%2Cpmp-derived%2Cpmprocessor-0%2Cpmprocessor-1%2Cpmprocessor_derived_943239074%2Cpmprocessor_derived_979700869%2Cpmprocessor_threshold_43780691%2Cpmprocessor_threshold_57930108%2Cpolicy%2Cprojectmanagement%2Csession-event-monitors%2Csvchist_pipeline%2Ctpegraph%2Cviability%2Cwatcher%7D.%7Bbp_aeprocessor_v1_alarmsforcorrelation%2Cbp_aeprocessor_v1_filters_pool%2Cbp_aeprocessor_v1_notifyhosts%2Cbp_aeprocessor_v1_notifytapi%2Cbp_aeprocessor_v1_operations%2Cbp_aeprocessor_v2_0_alarms%2Cbp_aeprocessor_v2_geo_sync%2Cbp_bpoifd_v1_changes_services%2Cbp_ciena_watcher_v2_geo%2Cbp_ciena_watcher_v2_geo_in%2Cbp_config_api_v1_backup%2Cbp_config_api_v1_restore%2Cbp_config_api_v1_upgrade%2Cbp_correlation_v1_changes_alarms%2Cbp_correlation_v1_changes_device%2Cbp_correlation_v1_changes_inventoryRelationship%2Cbp_crinoid_v1_kompressor%2Cbp_crinoid_v1_kompressor_in%2Cbp_discovery_v1_alarms%2Cbp_discovery_v2_0_changes_discoverySyncRequest%2Cbp_discovery_v2_0_changes_sync%2Cbp_discovery_v2_geo%2Cbp_discovery_v2_geo_in%2Cbp_discovery_v3_changes_managementSession%2Cbp_gcs_v2_kompressor%2Cbp_hpmra_v2_pm%2Cbp_ifd_v3_0_srlgJobs%2Cbp_kafkacomet_v1_channel_administration%2Cbp_licensing_v1_geo_sync%2Cbp_nms_notreplicated_v1_alarms%2Cbp_nms_replicated_v1_alarms%2Cbp_nsi_v1_alarms%2Cbp_nsi_v1_changes_equipmentStaticSpec%2Cbp_nsi_v1_changes_sync%2Cbp_nsi_v1_internal_changes_inventory%2Cbp_nsi_v1_internal_group%2Cbp_nsi_v1_internal_stitch%2Cbp_nsi_v1_internal_utilization%2Cbp_nsi_v2_0_changes_nc%2Cbp_nsi_v2_0_validation%2Cbp_nsi_v3_changes_equipment%2Cbp_nsi_v3_changes_fre%2Cbp_nsi_v3_changes_tpe%2Cbp_nsi_v3_validation%2Cbp_nsi_v5_changes_geo_in%2Cbp_perfg_v1_scheduler_ppg%2Cbp_perfg_v2_monitor_servicetrail%2Cbp_perfg_v2_monitor_viability%2Cbp_pinpoint_v1_otdr_state%2Cbp_pm_v1_geo_sync%2Cbp_pmp_v1_pm%2Cbp_pmprocessor_v1_datastore%2Cbp_projectmanagement_v2_cancelled%2Cbp_projectmanagement_v2_commit%2Cbp_projectmanagement_v2_deleted%2Cbp_projectmanagement_v2_list_deleted%2Cbp_projectmanagement_v2_list_projectadded%2Cbp_projectmanagement_v2_list_statechange%2Cbp_projectmanagement_v2_networkchanged%2Cbp_projectmanagement_v2_projectvalidate%2Cbp_projectmanagement_v2_statechange%2Cbp_ra_v1_EroEvent%2Cbp_ra_v1_SelectiveSync%2Cbp_ra_v1_alarms%2Cbp_ra_v1_diagnostics_loopback%2Cbp_ra_v1_nodecommissioning%2Cbp_ra_v1_pm%2Cbp_ra_v1_pm_thresholds%2Cbp_ra_v1_poke%2Cbp_ra_v2_0_sync%2Cbp_rasdk_v1_changes%2Cbp_stitcher_v1_changes_iplinkfre%2Cbp_stitcher_v2_changes_tpe%2Cbp_stitcher_v2_patch_fre%2Cbp_stitcher_v3_changes_fre%2Cbp_tdc_v1_tests%2Cbp_topologyplanning_v1_planneddata_fiberloss_changed%2Cbp_topologyplanning_v1_planneddata_omssnr_changed%2Cbp_tron_v1_changes%2Cbp_tron_v1_kompressor%2Cbp_view_alarmcorrelation_publish%2Cchronos_dispatch_queue%2Cchronos_scheduling_queue%2Cchronos_sharding_queues_0%2Cchronos_sharding_queues_1%2Ccom_cyaninc_bp_market%2Ccr%2Cfreg2svchist%2Cpm_scheduling%7D.committed&now=1630389493: HTTPConnectionPool(host='172.16.0.54', port=8081): Max retries exceeded with url: /render/?format=pickle&local=1&noCache=1&from=1630367893&until=1630389493&target=%2A.kafka.%2A.kafka.topicOffsets.%7BBPCORRELATOR%2CConfigMgmt%2CDiscovery%2CIFD%2CNSI%2CPINPOINT%2CRaProviderEventManager-0%2CRaProviderEventManager-1%2CRaProviderEventManager-2%2CTDC%2Caeprocessor%2Caeprocessor0%2Caeprocessor1%2Capi-gw_0%2Capi-gw_1%2Casset-area-ansible%2Casset-area-bpmn%2Casset-area-generic%2Casset-area-model-definitions%2Casset-area-terraform%2Cbp_commissioning%2Cchronos%2Ccom_cyaninc_bpocore_auth_troneventmonitor%2Ccrinoidapi_kompressor%2Cequipmenttopologyplanning%2Cfregraph%2Cinvhist_username_cache_0%2Cinvhist_username_cache_1%2Ckompressor%2Cmcpviewalarmcorrelator%2Cperfg-monitor-servicetrail%2Cperfg-monitor-viability%2Cperfg-oms%2Cperfg-plannedfiberloss%2Cperfg-scheduler-ppg%2Cperfg_equipment_v1%2Cperfg_equipmentstaticspec_v1%2Cperfg_fre_v1%2Cperfg_nc_v1%2Cperfg_tpe_v1%2Cplanningintent%2Cpm-discovery%2Cpm-ingestion%2Cpm-metrics_34741333%2Cpm-metrics_666918844%2Cpm-nsi%2Cpm-scheduler%2Cpm-thresholds%2Cpmp-derived%2Cpmprocessor-0%2Cpmprocessor-1%2Cpmprocessor_derived_943239074%2Cpmprocessor_derived_979700869%2Cpmprocessor_threshold_43780691%2Cpmprocessor_threshold_57930108%2Cpolicy%2Cprojectmanagement%2Csession-event-monitors%2Csvchist_pipeline%2Ctpegraph%2Cviability%2Cwatcher%7D.%7Bbp_aeprocessor_v1_alarmsforcorrelation%2Cbp_aeprocessor_v1_filters_pool%2Cbp_aeprocessor_v1_notifyhosts%2Cbp_aeprocessor_v1_notifytapi%2Cbp_aeprocessor_v1_operations%2Cbp_aeprocessor_v2_0_alarms%2Cbp_aeprocessor_v2_geo_sync%2Cbp_bpoifd_v1_changes_services%2Cbp_ciena_watcher_v2_geo%2Cbp_ciena_watcher_v2_geo_in%2Cbp_config_api_v1_backup%2Cbp_config_api_v1_restore%2Cbp_config_api_v1_upgrade%2Cbp_correlation_v1_changes_alarms%2Cbp_correlation_v1_changes_device%2Cbp_correlation_v1_changes_inventoryRelationship%2Cbp_crinoid_v1_kompressor%2Cbp_crinoid_v1_kompressor_in%2Cbp_discovery_v1_alarms%2Cbp_discovery_v2_0_changes_discoverySyncRequest%2Cbp_discovery_v2_0_changes_sync%2Cbp_discovery_v2_geo%2Cbp_discovery_v2_geo_in%2Cbp_discovery_v3_changes_managementSession%2Cbp_gcs_v2_kompressor%2Cbp_hpmra_v2_pm%2Cbp_ifd_v3_0_srlgJobs%2Cbp_kafkacomet_v1_channel_administration%2Cbp_licensing_v1_geo_sync%2Cbp_nms_notreplicated_v1_alarms%2Cbp_nms_replicated_v1_alarms%2Cbp_nsi_v1_alarms%2Cbp_nsi_v1_changes_equipmentStaticSpec%2Cbp_nsi_v1_changes_sync%2Cbp_nsi_v1_internal_changes_inventory%2Cbp_nsi_v1_internal_group%2Cbp_nsi_v1_internal_stitch%2Cbp_nsi_v1_internal_utilization%2Cbp_nsi_v2_0_changes_nc%2Cbp_nsi_v2_0_validation%2Cbp_nsi_v3_changes_equipment%2Cbp_nsi_v3_changes_fre%2Cbp_nsi_v3_changes_tpe%2Cbp_nsi_v3_validation%2Cbp_nsi_v5_changes_geo_in%2Cbp_perfg_v1_scheduler_ppg%2Cbp_perfg_v2_monitor_servicetrail%2Cbp_perfg_v2_monitor_viability%2Cbp_pinpoint_v1_otdr_state%2Cbp_pm_v1_geo_sync%2Cbp_pmp_v1_pm%2Cbp_pmprocessor_v1_datastore%2Cbp_projectmanagement_v2_cancelled%2Cbp_projectmanagement_v2_commit%2Cbp_projectmanagement_v2_deleted%2Cbp_projectmanagement_v2_list_deleted%2Cbp_projectmanagement_v2_list_projectadded%2Cbp_projectmanagement_v2_list_statechange%2Cbp_projectmanagement_v2_networkchanged%2Cbp_projectmanagement_v2_projectvalidate%2Cbp_projectmanagement_v2_statechange%2Cbp_ra_v1_EroEvent%2Cbp_ra_v1_SelectiveSync%2Cbp_ra_v1_alarms%2Cbp_ra_v1_diagnostics_loopback%2Cbp_ra_v1_nodecommissioning%2Cbp_ra_v1_pm%2Cbp_ra_v1_pm_thresholds%2Cbp_ra_v1_poke%2Cbp_ra_v2_0_sync%2Cbp_rasdk_v1_changes%2Cbp_stitcher_v1_changes_iplinkfre%2Cbp_stitcher_v2_changes_tpe%2Cbp_stitcher_v2_patch_fre%2Cbp_stitcher_v3_changes_fre%2Cbp_tdc_v1_tests%2Cbp_topologyplanning_v1_planneddata_fiberloss_changed%2Cbp_topologyplanning_v1_planneddata_omssnr_changed%2Cbp_tron_v1_changes%2Cbp_tron_v1_kompressor%2Cbp_view_alarmcorrelation_publish%2Cchronos_dispatch_queue%2Cchronos_scheduling_queue%2Cchronos_sharding_queues_0%2Cchronos_sharding_queues_1%2Ccom_cyaninc_bp_market%2Ccr%2Cfreg2svchist%2Cpm_scheduling%7D.committed&now=1630389493 (Caused by ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
2021-08-31,01:58:14.074 :: Exception during fetch for ['*.kafka.*.kafka.topicOffsets.{BPCORRELATOR,ConfigMgmt,Discovery,IFD,NSI,PINPOINT,RaProviderEventManager-0,RaProviderEventManager-1,RaProviderEventManager-2,TDC,aeprocessor,aeprocessor0,aeprocessor1,api-gw_0,api-gw_1,asset-area-ansible,asset-area-bpmn,asset-area-generic,asset-area-model-definitions,asset-area-terraform,bp_commissioning,chronos,com_cyaninc_bpocore_auth_troneventmonitor,crinoidapi_kompressor,equipmenttopologyplanning,fregraph,invhist_username_cache_0,invhist_username_cache_1,kompressor,mcpviewalarmcorrelator,perfg-monitor-servicetrail,perfg-monitor-viability,perfg-oms,perfg-plannedfiberloss,perfg-scheduler-ppg,perfg_equipment_v1,perfg_equipmentstaticspec_v1,perfg_fre_v1,perfg_nc_v1,perfg_tpe_v1,planningintent,pm-discovery,pm-ingestion,pm-metrics_34741333,pm-metrics_666918844,pm-nsi,pm-scheduler,pm-thresholds,pmp-derived,pmprocessor-0,pmprocessor-1,pmprocessor_derived_943239074,pmprocessor_derived_979700869,pmprocessor_threshold_43780691,pmprocessor_threshold_57930108,policy,projectmanagement,session-event-monitors,svchist_pipeline,tpegraph,viability,watcher}.{bp_aeprocessor_v1_alarmsforcorrelation,bp_aeprocessor_v1_filters_pool,bp_aeprocessor_v1_notifyhosts,bp_aeprocessor_v1_notifytapi,bp_aeprocessor_v1_operations,bp_aeprocessor_v2_0_alarms,bp_aeprocessor_v2_geo_sync,bp_bpoifd_v1_changes_services,bp_ciena_watcher_v2_geo,bp_ciena_watcher_v2_geo_in,bp_config_api_v1_backup,bp_config_api_v1_restore,bp_config_api_v1_upgrade,bp_correlation_v1_changes_alarms,bp_correlation_v1_changes_device,bp_correlation_v1_changes_inventoryRelationship,bp_crinoid_v1_kompressor,bp_crinoid_v1_kompressor_in,bp_discovery_v1_alarms,bp_discovery_v2_0_changes_discoverySyncRequest,bp_discovery_v2_0_changes_sync,bp_discovery_v2_geo,bp_discovery_v2_geo_in,bp_discovery_v3_changes_managementSession,bp_gcs_v2_kompressor,bp_hpmra_v2_pm,bp_ifd_v3_0_srlgJobs,bp_kafkacomet_v1_channel_administration,bp_licensing_v1_geo_sync,bp_nms_notreplicated_v1_alarms,bp_nms_replicated_v1_alarms,bp_nsi_v1_alarms,bp_nsi_v1_changes_equipmentStaticSpec,bp_nsi_v1_changes_sync,bp_nsi_v1_internal_changes_inventory,bp_nsi_v1_internal_group,bp_nsi_v1_internal_stitch,bp_nsi_v1_internal_utilization,bp_nsi_v2_0_changes_nc,bp_nsi_v2_0_validation,bp_nsi_v3_changes_equipment,bp_nsi_v3_changes_fre,bp_nsi_v3_changes_tpe,bp_nsi_v3_validation,bp_nsi_v5_changes_geo_in,bp_perfg_v1_scheduler_ppg,bp_perfg_v2_monitor_servicetrail,bp_perfg_v2_monitor_viability,bp_pinpoint_v1_otdr_state,bp_pm_v1_geo_sync,bp_pmp_v1_pm,bp_pmprocessor_v1_datastore,bp_projectmanagement_v2_cancelled,bp_projectmanagement_v2_commit,bp_projectmanagement_v2_deleted,bp_projectmanagement_v2_list_deleted,bp_projectmanagement_v2_list_projectadded,bp_projectmanagement_v2_list_statechange,bp_projectmanagement_v2_networkchanged,bp_projectmanagement_v2_projectvalidate,bp_projectmanagement_v2_statechange,bp_ra_v1_EroEvent,bp_ra_v1_SelectiveSync,bp_ra_v1_alarms,bp_ra_v1_diagnostics_loopback,bp_ra_v1_nodecommissioning,bp_ra_v1_pm,bp_ra_v1_pm_thresholds,bp_ra_v1_poke,bp_ra_v2_0_sync,bp_rasdk_v1_changes,bp_stitcher_v1_changes_iplinkfre,bp_stitcher_v2_changes_tpe,bp_stitcher_v2_patch_fre,bp_stitcher_v3_changes_fre,bp_tdc_v1_tests,bp_topologyplanning_v1_planneddata_fiberloss_changed,bp_topologyplanning_v1_planneddata_omssnr_changed,bp_tron_v1_changes,bp_tron_v1_kompressor,bp_view_alarmcorrelation_publish,chronos_dispatch_queue,chronos_scheduling_queue,chronos_sharding_queues_0,chronos_sharding_queues_1,com_cyaninc_bp_market,cr,freg2svchist,pm_scheduling}.committed'] after 0.828657s: Error requesting http://172.16.1.72:8081/render/?format=pickle&local=1&noCache=1&from=1630367893&until=1630389493&target=%2A.kafka.%2A.kafka.topicOffsets.%7BBPCORRELATOR%2CConfigMgmt%2CDiscovery%2CIFD%2CNSI%2CPINPOINT%2CRaProviderEventManager-0%2CRaProviderEventManager-1%2CRaProviderEventManager-2%2CTDC%2Caeprocessor%2Caeprocessor0%2Caeprocessor1%2Capi-gw_0%2Capi-gw_1%2Casset-area-ansible%2Casset-area-bpmn%2Casset-area-generic%2Casset-area-model-definitions%2Casset-area-terraform%2Cbp_commissioning%2Cchronos%2Ccom_cyaninc_bpocore_auth_troneventmonitor%2Ccrinoidapi_kompressor%2Cequipmenttopologyplanning%2Cfregraph%2Cinvhist_username_cache_0%2Cinvhist_username_cache_1%2Ckompressor%2Cmcpviewalarmcorrelator%2Cperfg-monitor-servicetrail%2Cperfg-monitor-viability%2Cperfg-oms%2Cperfg-plannedfiberloss%2Cperfg-scheduler-ppg%2Cperfg_equipment_v1%2Cperfg_equipmentstaticspec_v1%2Cperfg_fre_v1%2Cperfg_nc_v1%2Cperfg_tpe_v1%2Cplanningintent%2Cpm-discovery%2Cpm-ingestion%2Cpm-metrics_34741333%2Cpm-metrics_666918844%2Cpm-nsi%2Cpm-scheduler%2Cpm-thresholds%2Cpmp-derived%2Cpmprocessor-0%2Cpmprocessor-1%2Cpmprocessor_derived_943239074%2Cpmprocessor_derived_979700869%2Cpmprocessor_threshold_43780691%2Cpmprocessor_threshold_57930108%2Cpolicy%2Cprojectmanagement%2Csession-event-monitors%2Csvchist_pipeline%2Ctpegraph%2Cviability%2Cwatcher%7D.%7Bbp_aeprocessor_v1_alarmsforcorrelation%2Cbp_aeprocessor_v1_filters_pool%2Cbp_aeprocessor_v1_notifyhosts%2Cbp_aeprocessor_v1_notifytapi%2Cbp_aeprocessor_v1_operations%2Cbp_aeprocessor_v2_0_alarms%2Cbp_aeprocessor_v2_geo_sync%2Cbp_bpoifd_v1_changes_services%2Cbp_ciena_watcher_v2_geo%2Cbp_ciena_watcher_v2_geo_in%2Cbp_config_api_v1_backup%2Cbp_config_api_v1_restore%2Cbp_config_api_v1_upgrade%2Cbp_correlation_v1_changes_alarms%2Cbp_correlation_v1_changes_device%2Cbp_correlation_v1_changes_inventoryRelationship%2Cbp_crinoid_v1_kompressor%2Cbp_crinoid_v1_kompressor_in%2Cbp_discovery_v1_alarms%2Cbp_discovery_v2_0_changes_discoverySyncRequest%2Cbp_discovery_v2_0_changes_sync%2Cbp_discovery_v2_geo%2Cbp_discovery_v2_geo_in%2Cbp_discovery_v3_changes_managementSession%2Cbp_gcs_v2_kompressor%2Cbp_hpmra_v2_pm%2Cbp_ifd_v3_0_srlgJobs%2Cbp_kafkacomet_v1_channel_administration%2Cbp_licensing_v1_geo_sync%2Cbp_nms_notreplicated_v1_alarms%2Cbp_nms_replicated_v1_alarms%2Cbp_nsi_v1_alarms%2Cbp_nsi_v1_changes_equipmentStaticSpec%2Cbp_nsi_v1_changes_sync%2Cbp_nsi_v1_internal_changes_inventory%2Cbp_nsi_v1_internal_group%2Cbp_nsi_v1_internal_stitch%2Cbp_nsi_v1_internal_utilization%2Cbp_nsi_v2_0_changes_nc%2Cbp_nsi_v2_0_validation%2Cbp_nsi_v3_changes_equipment%2Cbp_nsi_v3_changes_fre%2Cbp_nsi_v3_changes_tpe%2Cbp_nsi_v3_validation%2Cbp_nsi_v5_changes_geo_in%2Cbp_perfg_v1_scheduler_ppg%2Cbp_perfg_v2_monitor_servicetrail%2Cbp_perfg_v2_monitor_viability%2Cbp_pinpoint_v1_otdr_state%2Cbp_pm_v1_geo_sync%2Cbp_pmp_v1_pm%2Cbp_pmprocessor_v1_datastore%2Cbp_projectmanagement_v2_cancelled%2Cbp_projectmanagement_v2_commit%2Cbp_projectmanagement_v2_deleted%2Cbp_projectmanagement_v2_list_deleted%2Cbp_projectmanagement_v2_list_projectadded%2Cbp_projectmanagement_v2_list_statechange%2Cbp_projectmanagement_v2_networkchanged%2Cbp_projectmanagement_v2_projectvalidate%2Cbp_projectmanagement_v2_statechange%2Cbp_ra_v1_EroEvent%2Cbp_ra_v1_SelectiveSync%2Cbp_ra_v1_alarms%2Cbp_ra_v1_diagnostics_loopback%2Cbp_ra_v1_nodecommissioning%2Cbp_ra_v1_pm%2Cbp_ra_v1_pm_thresholds%2Cbp_ra_v1_poke%2Cbp_ra_v2_0_sync%2Cbp_rasdk_v1_changes%2Cbp_stitcher_v1_changes_iplinkfre%2Cbp_stitcher_v2_changes_tpe%2Cbp_stitcher_v2_patch_fre%2Cbp_stitcher_v3_changes_fre%2Cbp_tdc_v1_tests%2Cbp_topologyplanning_v1_planneddata_fiberloss_changed%2Cbp_topologyplanning_v1_planneddata_omssnr_changed%2Cbp_tron_v1_changes%2Cbp_tron_v1_kompressor%2Cbp_view_alarmcorrelation_publish%2Cchronos_dispatch_queue%2Cchronos_scheduling_queue%2Cchronos_sharding_queues_0%2Cchronos_sharding_queues_1%2Ccom_cyaninc_bp_market%2Ccr%2Cfreg2svchist%2Cpm_scheduling%7D.committed&now=1630389493: HTTPConnectionPool(host='172.16.1.72', port=8081): Max retries exceeded with url: /render/?format=pickle&local=1&noCache=1&from=1630367893&until=1630389493&target=%2A.kafka.%2A.kafka.topicOffsets.%7BBPCORRELATOR%2CConfigMgmt%2CDiscovery%2CIFD%2CNSI%2CPINPOINT%2CRaProviderEventManager-0%2CRaProviderEventManager-1%2CRaProviderEventManager-2%2CTDC%2Caeprocessor%2Caeprocessor0%2Caeprocessor1%2Capi-gw_0%2Capi-gw_1%2Casset-area-ansible%2Casset-area-bpmn%2Casset-area-generic%2Casset-area-model-definitions%2Casset-area-terraform%2Cbp_commissioning%2Cchronos%2Ccom_cyaninc_bpocore_auth_troneventmonitor%2Ccrinoidapi_kompressor%2Cequipmenttopologyplanning%2Cfregraph%2Cinvhist_username_cache_0%2Cinvhist_username_cache_1%2Ckompressor%2Cmcpviewalarmcorrelator%2Cperfg-monitor-servicetrail%2Cperfg-monitor-viability%2Cperfg-oms%2Cperfg-plannedfiberloss%2Cperfg-scheduler-ppg%2Cperfg_equipment_v1%2Cperfg_equipmentstaticspec_v1%2Cperfg_fre_v1%2Cperfg_nc_v1%2Cperfg_tpe_v1%2Cplanningintent%2Cpm-discovery%2Cpm-ingestion%2Cpm-metrics_34741333%2Cpm-metrics_666918844%2Cpm-nsi%2Cpm-scheduler%2Cpm-thresholds%2Cpmp-derived%2Cpmprocessor-0%2Cpmprocessor-1%2Cpmprocessor_derived_943239074%2Cpmprocessor_derived_979700869%2Cpmprocessor_threshold_43780691%2Cpmprocessor_threshold_57930108%2Cpolicy%2Cprojectmanagement%2Csession-event-monitors%2Csvchist_pipeline%2Ctpegraph%2Cviability%2Cwatcher%7D.%7Bbp_aeprocessor_v1_alarmsforcorrelation%2Cbp_aeprocessor_v1_filters_pool%2Cbp_aeprocessor_v1_notifyhosts%2Cbp_aeprocessor_v1_notifytapi%2Cbp_aeprocessor_v1_operations%2Cbp_aeprocessor_v2_0_alarms%2Cbp_aeprocessor_v2_geo_sync%2Cbp_bpoifd_v1_changes_services%2Cbp_ciena_watcher_v2_geo%2Cbp_ciena_watcher_v2_geo_in%2Cbp_config_api_v1_backup%2Cbp_config_api_v1_restore%2Cbp_config_api_v1_upgrade%2Cbp_correlation_v1_changes_alarms%2Cbp_correlation_v1_changes_device%2Cbp_correlation_v1_changes_inventoryRelationship%2Cbp_crinoid_v1_kompressor%2Cbp_crinoid_v1_kompressor_in%2Cbp_discovery_v1_alarms%2Cbp_discovery_v2_0_changes_discoverySyncRequest%2Cbp_discovery_v2_0_changes_sync%2Cbp_discovery_v2_geo%2Cbp_discovery_v2_geo_in%2Cbp_discovery_v3_changes_managementSession%2Cbp_gcs_v2_kompressor%2Cbp_hpmra_v2_pm%2Cbp_ifd_v3_0_srlgJobs%2Cbp_kafkacomet_v1_channel_administration%2Cbp_licensing_v1_geo_sync%2Cbp_nms_notreplicated_v1_alarms%2Cbp_nms_replicated_v1_alarms%2Cbp_nsi_v1_alarms%2Cbp_nsi_v1_changes_equipmentStaticSpec%2Cbp_nsi_v1_changes_sync%2Cbp_nsi_v1_internal_changes_inventory%2Cbp_nsi_v1_internal_group%2Cbp_nsi_v1_internal_stitch%2Cbp_nsi_v1_internal_utilization%2Cbp_nsi_v2_0_changes_nc%2Cbp_nsi_v2_0_validation%2Cbp_nsi_v3_changes_equipment%2Cbp_nsi_v3_changes_fre%2Cbp_nsi_v3_changes_tpe%2Cbp_nsi_v3_validation%2Cbp_nsi_v5_changes_geo_in%2Cbp_perfg_v1_scheduler_ppg%2Cbp_perfg_v2_monitor_servicetrail%2Cbp_perfg_v2_monitor_viability%2Cbp_pinpoint_v1_otdr_state%2Cbp_pm_v1_geo_sync%2Cbp_pmp_v1_pm%2Cbp_pmprocessor_v1_datastore%2Cbp_projectmanagement_v2_cancelled%2Cbp_projectmanagement_v2_commit%2Cbp_projectmanagement_v2_deleted%2Cbp_projectmanagement_v2_list_deleted%2Cbp_projectmanagement_v2_list_projectadded%2Cbp_projectmanagement_v2_list_statechange%2Cbp_projectmanagement_v2_networkchanged%2Cbp_projectmanagement_v2_projectvalidate%2Cbp_projectmanagement_v2_statechange%2Cbp_ra_v1_EroEvent%2Cbp_ra_v1_SelectiveSync%2Cbp_ra_v1_alarms%2Cbp_ra_v1_diagnostics_loopback%2Cbp_ra_v1_nodecommissioning%2Cbp_ra_v1_pm%2Cbp_ra_v1_pm_thresholds%2Cbp_ra_v1_poke%2Cbp_ra_v2_0_sync%2Cbp_rasdk_v1_changes%2Cbp_stitcher_v1_changes_iplinkfre%2Cbp_stitcher_v2_changes_tpe%2Cbp_stitcher_v2_patch_fre%2Cbp_stitcher_v3_changes_fre%2Cbp_tdc_v1_tests%2Cbp_topologyplanning_v1_planneddata_fiberloss_changed%2Cbp_topologyplanning_v1_planneddata_omssnr_changed%2Cbp_tron_v1_changes%2Cbp_tron_v1_kompressor%2Cbp_view_alarmcorrelation_publish%2Cchronos_dispatch_queue%2Cchronos_scheduling_queue%2Cchronos_sharding_queues_0%2Cchronos_sharding_queues_1%2Ccom_cyaninc_bp_market%2Ccr%2Cfreg2svchist%2Cpm_scheduling%7D.committed&now=1630389493 (Caused by ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
2021-08-31,01:58:14.297 :: graphite.render.datalib.fetchData :: lookup and merge of "*.kafka.*.kafka.topicOffsets.{BPCORRELATOR,ConfigMgmt,Discovery,IFD,NSI,PINPOINT,RaProviderEventManager-0,RaProviderEventManager-1,RaProviderEventManager-2,TDC,aeprocessor,aeprocessor0,aeprocessor1,api-gw_0,api-gw_1,asset-area-ansible,asset-area-bpmn,asset-area-generic,asset-area-model-definitions,asset-area-terraform,bp_commissioning,chronos,com_cyaninc_bpocore_auth_troneventmonitor,crinoidapi_kompressor,equipmenttopologyplanning,fregraph,invhist_username_cache_0,invhist_username_cache_1,kompressor,mcpviewalarmcorrelator,perfg-monitor-servicetrail,perfg-monitor-viability,perfg-oms,perfg-plannedfiberloss,perfg-scheduler-ppg,perfg_equipment_v1,perfg_equipmentstaticspec_v1,perfg_fre_v1,perfg_nc_v1,perfg_tpe_v1,planningintent,pm-discovery,pm-ingestion,pm-metrics_34741333,pm-metrics_666918844,pm-nsi,pm-scheduler,pm-thresholds,pmp-derived,pmprocessor-0,pmprocessor-1,pmprocessor_derived_943239074,pmprocessor_derived_979700869,pmprocessor_threshold_43780691,pmprocessor_threshold_57930108,policy,projectmanagement,session-event-monitors,svchist_pipeline,tpegraph,viability,watcher}.{bp_aeprocessor_v1_alarmsforcorrelation,bp_aeprocessor_v1_filters_pool,bp_aeprocessor_v1_notifyhosts,bp_aeprocessor_v1_notifytapi,bp_aeprocessor_v1_operations,bp_aeprocessor_v2_0_alarms,bp_aeprocessor_v2_geo_sync,bp_bpoifd_v1_changes_services,bp_ciena_watcher_v2_geo,bp_ciena_watcher_v2_geo_in,bp_config_api_v1_backup,bp_config_api_v1_restore,bp_config_api_v1_upgrade,bp_correlation_v1_changes_alarms,bp_correlation_v1_changes_device,bp_correlation_v1_changes_inventoryRelationship,bp_crinoid_v1_kompressor,bp_crinoid_v1_kompressor_in,bp_discovery_v1_alarms,bp_discovery_v2_0_changes_discoverySyncRequest,bp_discovery_v2_0_changes_sync,bp_discovery_v2_geo,bp_discovery_v2_geo_in,bp_discovery_v3_changes_managementSession,bp_gcs_v2_kompressor,bp_hpmra_v2_pm,bp_ifd_v3_0_srlgJobs,bp_kafkacomet_v1_channel_administration,bp_licensing_v1_geo_sync,bp_nms_notreplicated_v1_alarms,bp_nms_replicated_v1_alarms,bp_nsi_v1_alarms,bp_nsi_v1_changes_equipmentStaticSpec,bp_nsi_v1_changes_sync,bp_nsi_v1_internal_changes_inventory,bp_nsi_v1_internal_group,bp_nsi_v1_internal_stitch,bp_nsi_v1_internal_utilization,bp_nsi_v2_0_changes_nc,bp_nsi_v2_0_validation,bp_nsi_v3_changes_equipment,bp_nsi_v3_changes_fre,bp_nsi_v3_changes_tpe,bp_nsi_v3_validation,bp_nsi_v5_changes_geo_in,bp_perfg_v1_scheduler_ppg,bp_perfg_v2_monitor_servicetrail,bp_perfg_v2_monitor_viability,bp_pinpoint_v1_otdr_state,bp_pm_v1_geo_sync,bp_pmp_v1_pm,bp_pmprocessor_v1_datastore,bp_projectmanagement_v2_cancelled,bp_projectmanagement_v2_commit,bp_projectmanagement_v2_deleted,bp_projectmanagement_v2_list_deleted,bp_projectmanagement_v2_list_projectadded,bp_projectmanagement_v2_list_statechange,bp_projectmanagement_v2_networkchanged,bp_projectmanagement_v2_projectvalidate,bp_projectmanagement_v2_statechange,bp_ra_v1_EroEvent,bp_ra_v1_SelectiveSync,bp_ra_v1_alarms,bp_ra_v1_diagnostics_loopback,bp_ra_v1_nodecommissioning,bp_ra_v1_pm,bp_ra_v1_pm_thresholds,bp_ra_v1_poke,bp_ra_v2_0_sync,bp_rasdk_v1_changes,bp_stitcher_v1_changes_iplinkfre,bp_stitcher_v2_changes_tpe,bp_stitcher_v2_patch_fre,bp_stitcher_v3_changes_fre,bp_tdc_v1_tests,bp_topologyplanning_v1_planneddata_fiberloss_changed,bp_topologyplanning_v1_planneddata_omssnr_changed,bp_tron_v1_changes,bp_tron_v1_kompressor,bp_view_alarmcorrelation_publish,chronos_dispatch_queue,chronos_scheduling_queue,chronos_sharding_queues_0,chronos_sharding_queues_1,com_cyaninc_bp_market,cr,freg2svchist,pm_scheduling}.committed" took 0.00185609s
deniszh commented 3 years ago

as you can see remote graphite-web just closing connection and not responding with anything - but doing that after 0.001 s - so, it's not a timeout. Can be firewall or something like this, but looks like something beyond Graphite itself.

AshishVerma7690 commented 3 years ago

It is showing this issue for some of the dashboards. For the dashboard with less no. of endpoints, it is working fine and shows combined data on the dashboard.

AshishVerma7690 commented 3 years ago

So Tried multiple scenarios and found that after a number of data points the graphite web starts giving issues.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.