search

graphprotocol / mission-control-indexer

Technical indexer documentation and infrastructure templates for the Mission Control testnet
21 stars 4 forks source link

Failed messages: Can't manage allocations (223 over 10 hrs) #253

Open davaymne opened 3 years ago

davaymne commented 3 years ago

[2020-12-11 00:01:51.485 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 00:01:51.485 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 00:02:49.591 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 00:11:14.347 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 00:11:46.617 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 00:50:44.509 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 01:00:15.978 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 01:00:15.979 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 01:01:14.347 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 01:02:14.340 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 01:03:14.341 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 01:34:44.638 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 01:37:49.425 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 01:37:49.425 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:02:14.350 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:02:15.745 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:02:15.745 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:03:14.371 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:03:48.737 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:03:48.738 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:04:14.345 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:05:14.346 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:06:14.346 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:06:15.745 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:06:15.746 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:07:14.352 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:08:14.342 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:08:15.760 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:08:15.761 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:09:14.345 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:10:14.349 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:10:15.763 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:10:15.763 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:11:14.342 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:12:14.355 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:12:44.627 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 02:12:44.627 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:13:14.348 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:14:17.391 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 02:14:17.391 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:15:14.345 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:16:15.753 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:16:15.753 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:18:14.380 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:18:15.753 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:18:15.753 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:38:17.790 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 02:38:17.790 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:40:14.426 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:40:15.826 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:40:15.826 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:41:14.424 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 02:42:15.827 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:42:15.828 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 02:44:15.826 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 02:44:15.826 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:00:14.344 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:00:17.343 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 03:00:17.344 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:04:14.345 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:05:14.350 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:08:25.157 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 03:08:25.157 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:12:14.364 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:12:15.764 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 03:12:15.764 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:13:14.373 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:13:53.703 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 03:13:53.704 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:14:14.354 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:15:14.350 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:18:16.983 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 03:18:16.983 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:19:14.351 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:20:14.361 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:20:42.472 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 03:20:42.472 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:21:14.351 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:22:14.356 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:22:15.802 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 03:22:15.802 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:24:14.360 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:24:42.826 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 03:24:42.826 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:26:14.364 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:26:38.426 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 03:26:38.426 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:28:14.364 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:28:15.760 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 03:28:15.761 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:29:14.370 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:34:41.162 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 03:34:41.163 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:36:41.976 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 03:36:41.976 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:38:14.363 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 03:40:20.115 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 03:40:20.115 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:44:19.896 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 03:44:19.897 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:58:16.807 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 03:58:16.808 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 03:59:56.771 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 03:59:56.772 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 04:01:14.351 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 04:02:21.839 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 04:02:21.839 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 04:04:17.315 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 04:04:17.316 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 04:06:14.352 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 04:12:15.775 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 04:12:15.776 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 04:20:15.765 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 04:20:15.766 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 04:21:14.386 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 04:22:14.384 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 04:22:31.632 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 04:22:31.633 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 04:24:14.386 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:10:14.378 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:10:15.778 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 05:10:15.778 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:12:56.455 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:14:14.391 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:15:14.404 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:18:14.428 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:18:17.198 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 05:18:17.198 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:20:15.784 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 05:20:15.784 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:22:14.387 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:22:15.794 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 05:22:15.794 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:26:44.623 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:30:15.804 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 05:30:15.804 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:32:23.457 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 05:32:23.457 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:34:14.372 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:34:18.550 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 05:34:18.550 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:40:15.803 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 05:40:15.803 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:41:14.369 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:42:14.381 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:42:15.762 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 05:42:15.762 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:44:14.382 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:44:15.773 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 05:44:15.773 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:46:18.539 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 05:46:18.539 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:47:45.915 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 05:47:45.915 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:50:38.248 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 05:50:38.249 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 05:51:14.387 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:53:14.387 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 05:54:15.789 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 05:54:15.789 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 06:00:14.406 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 06:00:15.785 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 06:00:15.785 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 06:21:55.414 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 06:21:55.415 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 06:30:45.704 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 06:33:14.431 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 06:36:15.804 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 06:36:15.804 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 06:38:21.732 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 06:38:21.732 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 06:40:18.506 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 06:40:18.506 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 06:43:44.610 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 07:03:45.774 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 07:27:45.715 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 07:28:15.848 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 07:28:15.848 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 07:32:14.574 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 07:32:43.333 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 07:32:43.333 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 07:57:56.017 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 07:57:56.018 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 08:08:15.886 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 08:08:15.886 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 08:09:14.414 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 08:10:14.436 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 08:10:15.863 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 08:10:15.864 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 08:12:14.411 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 08:12:15.800 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 08:12:15.800 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 08:14:16.935 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 08:14:16.935 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 08:15:14.438 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 08:16:14.412 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 08:16:15.847 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 08:16:15.847 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 08:30:15.772 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 08:30:15.772 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 08:59:34.247 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 08:59:34.248 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 09:05:24.027 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 09:11:21.148 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 09:11:21.148 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 09:15:20.965 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query indexer allocations [2020-12-11 09:15:20.965 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 09:17:21.869 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 09:17:21.869 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 09:21:55.709 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 09:23:23.993 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 09:45:04.703 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 09:45:04.703 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 09:46:50.385 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 09:46:50.386 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 09:49:20.260 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 09:49:20.261 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 09:55:24.088 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to check for network pause, assuming it has not changed [2020-12-11 09:57:20.250 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query subgraph deployments worth indexing [2020-12-11 09:57:20.251 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network [2020-12-11 09:59:24.718 +0000] ERROR (IndexerAgent/1 on 18e74966ee2b): Failed to query claimable indexer allocations [2020-12-11 09:59:24.719 +0000] WARN (IndexerAgent/1 on 18e74966ee2b): Failed to synchronize with network

Jannis commented 3 years ago

/network request stats (last 6 hours)

| Status | Requests (total) | Percent |
|--------+------------------+---------|
|        |           778000 |  100.00 |
|    200 |           679700 |   87.37 |
|    204 |            24500 |    3.15 |
|    404 |            23000 |    2.96 |
|    499 |              300 |    0.04 |
|    502 |            50000 |    6.43 |

About 10% of the overall requests to =/network= are failing with a 404, 499 or 502 error. That is neither great nor terrible, since the agent and service both try again after a minute or two.

Jannis commented 3 years ago

The total number of requests I'd expect in 6 hours is something like:

total_requests = n_indexers * (3600 * 6 / 120) * (n_services + 1) * c

where

n_indexers ~ 100
n_services ~ 4
c          ~ 5 (the number of queries we make in each syncing cycle)

That adds up to 450000 requests made by indexers alone (the gateways also make some).

Jannis commented 3 years ago

I've seen about 200 IE010 errors over 5 minutes on average, over the last 6 hours. That's 0.6/s, which adds up to 14,400 failed IE010s over 6 hours. Compared to about ~ 90,000 expected requests over that period.

That makes it an estimated 16% error rate on the IE010 requests alone. Too high, for sure.

davaymne commented 3 years ago

Full log for day 2020-12-11 is here

agent.b5b624ffdfc41715fff3935284ef42a6a.log

davaymne commented 3 years ago

Number of Errors per minute:

Screenshot 2020-12-11 at 11 57 10
WayneWayner commented 3 years ago

Same issue on my side. https://gist.github.com/WayneWayner/7b9765209f1c69fc629a378afa4a5a0e

davaymne commented 3 years ago

Below you can find number of Failed messages by day: 20201209 - 128 20201210 - 266 20201211 - 418 20201212 - 194 20201213 - 207

Please, let me know if you need more details.

davaymne commented 3 years ago

Allocations are not getting updated, hence incorrect Indexer Reword is displayed on Account Card.

Screenshot 2020-12-14 at 13 31 35