search

Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.42k stars 115 forks source link

[Argus] `heap out of memory` crash #5080

Open kdembler opened 8 months ago

kdembler commented 8 months ago

Full log with crash at the end:

2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101959
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101959",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7e2922ab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-5ccdb8e7a7da8b8a17c9639cdfa34827-d6262f86b925a15e-01",
                "x-forwarded-for": "172.69.35.139",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101959",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-YoCNAtC7v+WLZhCrBKuH2VsJUis\""
            },
            "statusCode": 500
        },
        "responseTime": 27719,
        "prematurelyClosed": false
    },
    "trace_id": "5ccdb8e7a7da8b8a17c9639cdfa34827",
    "span_id": "b9b56f20f279f540",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "objectId": "2101915",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "objectId": "2101915",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "objectId": "2101915",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "objectId": "2101915",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
    "err": {
        "message": "Failed to download object 2101915 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101915 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
    "err": {
        "message": "Failed to download object 2101915 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101915 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "req": {
        "url": "/api/v1/assets/2101915",
        "headers": {
            "host": "dist1.joyutils.org",
            "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
            "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
            "accept-encoding": "gzip",
            "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
            "cdn-loop": "cloudflare",
            "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
            "cf-ipcountry": "AR",
            "cf-ray": "8590cd7e2925ab32-EZE",
            "cf-visitor": "{\"scheme\":\"https\"}",
            "referer": "https://gleev.xyz/",
            "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
            "sec-ch-ua-mobile": "?1",
            "sec-ch-ua-platform": "\"Android\"",
            "sec-fetch-dest": "image",
            "sec-fetch-mode": "no-cors",
            "sec-fetch-site": "cross-site",
            "traceparent": "00-a93216e4db2d4bb4c93708c9b31c5b71-0bd49bf08deb91aa-01",
            "x-forwarded-for": "172.69.35.138",
            "x-forwarded-host": "dist1.joyutils.org",
            "x-forwarded-proto": "https"
        },
        "method": "GET",
        "httpVersion": "1.1",
        "originalUrl": "/api/v1/assets/2101915",
        "query": {}
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101915
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101915",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7e2925ab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-a93216e4db2d4bb4c93708c9b31c5b71-0bd49bf08deb91aa-01",
                "x-forwarded-for": "172.69.35.138",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101915",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-17VO0ULVLc30wSWRZevqEyq7CAk\""
            },
            "statusCode": 500
        },
        "responseTime": 27720,
        "prematurelyClosed": false
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "objectId": "2101979",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "objectId": "2101979",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "objectId": "2101979",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "objectId": "2101979",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
    "err": {
        "message": "Failed to download object 2101979 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101979 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
    "err": {
        "message": "Failed to download object 2101979 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101979 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "req": {
        "url": "/api/v1/assets/2101979",
        "headers": {
            "host": "dist1.joyutils.org",
            "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
            "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
            "accept-encoding": "gzip",
            "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
            "cdn-loop": "cloudflare",
            "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
            "cf-ipcountry": "AR",
            "cf-ray": "8590cd7e2912ab32-EZE",
            "cf-visitor": "{\"scheme\":\"https\"}",
            "referer": "https://gleev.xyz/",
            "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
            "sec-ch-ua-mobile": "?1",
            "sec-ch-ua-platform": "\"Android\"",
            "sec-fetch-dest": "image",
            "sec-fetch-mode": "no-cors",
            "sec-fetch-site": "cross-site",
            "traceparent": "00-d1fcecc237c7860294550a8740333cf0-91e764aa10290763-01",
            "x-forwarded-for": "172.69.35.139",
            "x-forwarded-host": "dist1.joyutils.org",
            "x-forwarded-proto": "https"
        },
        "method": "GET",
        "httpVersion": "1.1",
        "originalUrl": "/api/v1/assets/2101979",
        "query": {}
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101979
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101979",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7e2912ab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-d1fcecc237c7860294550a8740333cf0-91e764aa10290763-01",
                "x-forwarded-for": "172.69.35.139",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101979",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-GuyLFrfFSeAf+cgtCn08QrrAopo\""
            },
            "statusCode": 500
        },
        "responseTime": 27726,
        "prematurelyClosed": false
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "objectId": "2101965",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "objectId": "2101965",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "objectId": "2101965",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "objectId": "2101965",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
    "err": {
        "message": "Failed to download object 2101965 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101965 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
    "err": {
        "message": "Failed to download object 2101965 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101965 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "req": {
        "url": "/api/v1/assets/2101965",
        "headers": {
            "host": "dist1.joyutils.org",
            "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
            "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
            "accept-encoding": "gzip",
            "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
            "cdn-loop": "cloudflare",
            "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
            "cf-ipcountry": "AR",
            "cf-ray": "8590cd7e292dab32-EZE",
            "cf-visitor": "{\"scheme\":\"https\"}",
            "referer": "https://gleev.xyz/",
            "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
            "sec-ch-ua-mobile": "?1",
            "sec-ch-ua-platform": "\"Android\"",
            "sec-fetch-dest": "image",
            "sec-fetch-mode": "no-cors",
            "sec-fetch-site": "cross-site",
            "traceparent": "00-967d1636c501a5f625c64943e74227dc-74de9b66e6252cb1-01",
            "x-forwarded-for": "172.69.35.139",
            "x-forwarded-host": "dist1.joyutils.org",
            "x-forwarded-proto": "https"
        },
        "method": "GET",
        "httpVersion": "1.1",
        "originalUrl": "/api/v1/assets/2101965",
        "query": {}
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101965
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101965",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7e292dab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-967d1636c501a5f625c64943e74227dc-74de9b66e6252cb1-01",
                "x-forwarded-for": "172.69.35.139",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101965",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-e0TQIzyfoufzzJX6e7Fyy/VdXdA\""
            },
            "statusCode": 500
        },
        "responseTime": 27739,
        "prematurelyClosed": false
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "objectId": "2101963",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "objectId": "2101963",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "objectId": "2101963",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "objectId": "2101963",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
    "err": {
        "message": "Failed to download object 2101963 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101963 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
    "err": {
        "message": "Failed to download object 2101963 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101963 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "req": {
        "url": "/api/v1/assets/2101963",
        "headers": {
            "host": "dist1.joyutils.org",
            "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
            "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
            "accept-encoding": "gzip",
            "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
            "cdn-loop": "cloudflare",
            "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
            "cf-ipcountry": "AR",
            "cf-ray": "8590cd7c5e85ab32-EZE",
            "cf-visitor": "{\"scheme\":\"https\"}",
            "referer": "https://gleev.xyz/",
            "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
            "sec-ch-ua-mobile": "?1",
            "sec-ch-ua-platform": "\"Android\"",
            "sec-fetch-dest": "image",
            "sec-fetch-mode": "no-cors",
            "sec-fetch-site": "cross-site",
            "traceparent": "00-06efab4bfed59b61dade00ff424919e7-b880249753ee0cfd-01",
            "x-forwarded-for": "172.69.35.138",
            "x-forwarded-host": "dist1.joyutils.org",
            "x-forwarded-proto": "https"
        },
        "method": "GET",
        "httpVersion": "1.1",
        "originalUrl": "/api/v1/assets/2101963",
        "query": {}
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101963
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101963",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7c5e85ab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-06efab4bfed59b61dade00ff424919e7-b880249753ee0cfd-01",
                "x-forwarded-for": "172.69.35.138",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101963",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-tiUt7lt6aYy57dMAQTIHVELHQO4\""
            },
            "statusCode": 500
        },
        "responseTime": 27752,
        "prematurelyClosed": false
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}

<--- Last few GCs --->

[7:0x55f3870] 106762850 ms: Scavenge 4039.2 (4114.6) -> 4038.9 (4125.6) MB, 8.6 / 0.0 ms  (average mu = 0.253, current mu = 0.152) allocation failure;
[7:0x55f3870] 106762872 ms: Scavenge 4045.8 (4125.6) -> 4045.6 (4126.4) MB, 14.1 / 0.0 ms  (average mu = 0.253, current mu = 0.152) allocation failure;
[7:0x55f3870] 106764336 ms: Scavenge 4046.6 (4126.4) -> 4045.8 (4148.4) MB, 1463.1 / 0.0 ms  (average mu = 0.253, current mu = 0.152) allocation failure;

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb95b60 node::Abort() [node]
 2: 0xa9a7f8  [node]
 3: 0xd6f2f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd6f697 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf4cba5  [node]
 6: 0xf5f08d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xf3978e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xf3ab57 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xf1bd2a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x12e114f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x170deb9  [node]
/joystream/distributor-node/runner.sh: line 8:     7 Aborted                 (core dumped) node --require @joystream/opentelemetry ./bin/run $*
Loaded Application Instrumentation: "Distributor Node"
Starting tracing...
2024-02-21 17:46:38:4638 App info: Starting the app

All of the above logs claim to be at the exact same timestamp which is a bit weird.

CleanShot 2024-02-21 at 19 54 13@2x

Looks like a memory leak, not sure what happened on 19th

kdembler commented 8 months ago

Seems to still be happening on 2.0 CleanShot 2024-02-23 at 11 05 00@2x