search

threefoldtech / tfchain_graphql

Graphql for TFchain
Apache License 2.0
2 stars 3 forks source link

[all nets] Processor crashes regularly #191

Closed coesensbert closed 1 month ago

coesensbert commented 2 months ago

Did not find a pattern yet, but seems to happen on all nets. If we compare two stacks of the same net, both have this issue but not always at the same block.

Mainnet: dashboard.grid.tf

{"level":2,"time":1723610115263,"ns":"sqd:processor","msg":"13789842 / 13789842, rate: 0 blocks/sec, mapping: 8 blocks/sec, 83 items/sec, ingest: 47 blocks/sec, eta: 0s"}
{"level":2,"time":1723610125306,"ns":"sqd:processor","msg":"13789843 / 13789843, rate: 0 blocks/sec, mapping: 8 blocks/sec, 91 items/sec, ingest: 46 blocks/sec, eta: 0s"}
{"level":2,"time":1723610130999,"ns":"sqd:processor","msg":"13789844 / 13789844, rate: 0 blocks/sec, mapping: 4 blocks/sec, 55 items/sec, ingest: 45 blocks/sec, eta: 0s"}
{"level":2,"time":1723610139527,"ns":"sqd:processor","msg":"13789845 / 13789845, rate: 0 blocks/sec, mapping: 1 blocks/sec, 25 items/sec, ingest: 46 blocks/sec, eta: 0s"}
{"level":5,"time":1723610164513,"ns":"sqd:processor","err":{"query":"INSERT INTO \"node_contract\"(\"id\", \"grid_version\", \"contract_id\", \"twin_id\", \"node_id\", \"deployment_data\", \"deployment_hash\", \"number_of_public_i_ps\", \"state\", \"created_at\", \"solution_provider_id\", \"resources_used_id\") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, DEFAULT) ON CONFLICT ( \"id\" ) DO UPDATE SET \"id\" = EXCLUDED.\"id\", \"grid_version\" = EXCLUDED.\"grid_version\", \"contract_id\" = EXCLUDED.\"contract_id\", \"twin_id\" = EXCLUDED.\"twin_id\", \"node_id\" = EXCLUDED.\"node_id\", \"deployment_data\" = EXCLUDED.\"deployment_data\", \"deployment_hash\" = EXCLUDED.\"deployment_hash\", \"number_of_public_i_ps\" = EXCLUDED.\"number_of_public_i_ps\", \"state\" = EXCLUDED.\"state\", \"created_at\" = EXCLUDED.\"created_at\", \"solution_provider_id\" = EXCLUDED.\"solution_provider_id\"","parameters":["0013789691-000713-fab74",4,"611406",10972,888,"{\"version\":3,\"type\":\"network\",\"name\":\"example_c124_network\",\"projectName\":\"vm/group_c\"}","0e549bca93f361471fcd7a22595fc3a2",0,"Deleted","1723609194",0],"driverError":{"length":221,"name":"error","severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 27 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"error: failed reflecting node_contract updates could not serialize access due to concurrent update\n    at /squid/node_modules/pg/lib/client.js:526:17\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async nodeContractCanceled (/squid/lib/mappings/contracts.js:239:5)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"},"length":221,"severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 27 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"QueryFailedError: failed reflecting node_contract updates could not serialize access due to concurrent update\n    at PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async nodeContractCanceled (/squid/lib/mappings/contracts.js:239:5)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"}}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Mainnet: dashboard.be.grid.tf

{"level":2,"time":1723264489440,"ns":"sqd:processor","msg":"13732237 / 13732237, rate: 0 blocks/sec, mapping: 2 blocks/sec, 29 items/sec, ingest: 39 blocks/sec, eta: 0s"}
{"level":2,"time":1723264499728,"ns":"sqd:processor","msg":"13732238 / 13732239, rate: 0 blocks/sec, mapping: 1 blocks/sec, 17 items/sec, ingest: 37 blocks/sec, eta: 6s"}
{"level":2,"time":1723264583919,"ns":"sqd:processor","msg":"13732239 / 13732241, rate: 0 blocks/sec, mapping: 0 blocks/sec, 10 items/sec, ingest: 23 blocks/sec, eta: 15s"}
{"level":5,"time":1723264584478,"ns":"sqd:processor","err":{"query":"INSERT INTO \"node_contract\"(\"id\", \"grid_version\", \"contract_id\", \"twin_id\", \"node_id\", \"deployment_data\", \"deployment_hash\", \"number_of_public_i_ps\", \"state\", \"created_at\", \"solution_provider_id\", \"resources_used_id\") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, DEFAULT) ON CONFLICT ( \"id\" ) DO UPDATE SET \"id\" = EXCLUDED.\"id\", \"grid_version\" = EXCLUDED.\"grid_version\", \"contract_id\" = EXCLUDED.\"contract_id\", \"twin_id\" = EXCLUDED.\"twin_id\", \"node_id\" = EXCLUDED.\"node_id\", \"deployment_data\" = EXCLUDED.\"deployment_data\", \"deployment_hash\" = EXCLUDED.\"deployment_hash\", \"number_of_public_i_ps\" = EXCLUDED.\"number_of_public_i_ps\", \"state\" = EXCLUDED.\"state\", \"created_at\" = EXCLUDED.\"created_at\", \"solution_provider_id\" = EXCLUDED.\"solution_provider_id\"","parameters":["0013732108-000852-dfc53",4,"606172",10972,2089,"{\"version\":3,\"type\":\"network\",\"name\":\"example_c92_network\",\"projectName\":\"vm/group_c\"}","69626baa73d293ad16bda1ba677b9e6c",0,"Deleted","1723263696",0],"driverError":{"length":221,"name":"error","severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 27 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"error: failed reflecting node_contract updates could not serialize access due to concurrent update\n    at /squid/node_modules/pg/lib/client.js:526:17\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async nodeContractCanceled (/squid/lib/mappings/contracts.js:239:5)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"},"length":221,"severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 27 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"QueryFailedError: failed reflecting node_contract updates could not serialize access due to concurrent update\n    at PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async nodeContractCanceled (/squid/lib/mappings/contracts.js:239:5)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"}}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Testnet: dashboard.test.grid.tf

{"level":2,"time":1724728590910,"ns":"sqd:processor","msg":"14961403 / 14961403, rate: 0 blocks/sec, mapping: 54 blocks/sec, 54 items/sec, ingest: 61 blocks/sec, eta: 0s"}
{"level":2,"time":1724728595947,"ns":"sqd:processor","msg":"14961404 / 14961404, rate: 0 blocks/sec, mapping: 46 blocks/sec, 55 items/sec, ingest: 63 blocks/sec, eta: 0s"}
{"level":2,"time":1724728600973,"ns":"sqd:processor","msg":"14961405 / 14961405, rate: 0 blocks/sec, mapping: 39 blocks/sec, 63 items/sec, ingest: 62 blocks/sec, eta: 0s"}
{"level":5,"time":1724728606001,"ns":"sqd:processor","err":{"query":"INSERT INTO \"node_contract\"(\"id\", \"grid_version\", \"contract_id\", \"twin_id\", \"node_id\", \"deployment_data\", \"deployment_hash\", \"number_of_public_i_ps\", \"state\", \"created_at\", \"solution_provider_id\", \"resources_used_id\") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, DEFAULT) ON CONFLICT ( \"id\" ) DO UPDATE SET \"id\" = EXCLUDED.\"id\", \"grid_version\" = EXCLUDED.\"grid_version\", \"contract_id\" = EXCLUDED.\"contract_id\", \"twin_id\" = EXCLUDED.\"twin_id\", \"node_id\" = EXCLUDED.\"node_id\", \"deployment_data\" = EXCLUDED.\"deployment_data\", \"deployment_hash\" = EXCLUDED.\"deployment_hash\", \"number_of_public_i_ps\" = EXCLUDED.\"number_of_public_i_ps\", \"state\" = EXCLUDED.\"state\", \"created_at\" = EXCLUDED.\"created_at\", \"solution_provider_id\" = EXCLUDED.\"solution_provider_id\"","parameters":["0014961401-000008-6cf88",4,"49058",1149,12,"{\"version\":3,\"type\":\"network\",\"name\":\"net_4c7va2z4pg\",\"projectName\":\"Network\"}","7fbe918104c584d5e90710eb3c21b841",0,"Deleted","1724728560",0],"driverError":{"length":221,"name":"error","severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 27 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"error: failed reflecting node_contract updates could not serialize access due to concurrent update\n    at /squid/node_modules/pg/lib/client.js:526:17\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async nodeContractCanceled (/squid/lib/mappings/contracts.js:239:5)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"},"length":221,"severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 27 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"QueryFailedError: failed reflecting node_contract updates could not serialize access due to concurrent update\n    at PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async nodeContractCanceled (/squid/lib/mappings/contracts.js:239:5)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"}}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

QAnet: dashboard.qa.grid.tf

{"level":1,"time":1723692674105,"ns":"sqd:processor:archive-request","msg":"http body","httpRequestId":2331986,"httpResponseBody":{"data":{"status":{"head":11329240}}}}
{"level":5,"time":1723692674144,"ns":"sqd:processor","err":{"query":"INSERT INTO \"node_contract\"(\"id\", \"grid_version\", \"contract_id\", \"twin_id\", \"node_id\", \"deployment_data\", \"deployment_hash\", \"number_of_public_i_ps\", \"state\", \"created_at\", \"solution_provider_id\", \"resources_used_id\") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, DEFAULT) ON CONFLICT ( \"id\" ) DO UPDATE SET \"id\" = EXCLUDED.\"id\", \"grid_version\" = EXCLUDED.\"grid_version\", \"contract_id\" = EXCLUDED.\"contract_id\", \"twin_id\" = EXCLUDED.\"twin_id\", \"node_id\" = EXCLUDED.\"node_id\", \"deployment_data\" = EXCLUDED.\"deployment_data\", \"deployment_hash\" = EXCLUDED.\"deployment_hash\", \"number_of_public_i_ps\" = EXCLUDED.\"number_of_public_i_ps\", \"state\" = EXCLUDED.\"state\", \"created_at\" = EXCLUDED.\"created_at\", \"solution_provider_id\" = EXCLUDED.\"solution_provider_id\"","parameters":["0011329240-000002-dffb9",4,"39227",465,3,"{\"version\":3,\"type\":\"Gateway Name\",\"name\":\"ivbghiqeus\",\"projectName\":\"ivbghiqeus\"}","992b15a23a77e3b59f612683910b6e13",0,"Created","1723692660",0],"driverError":{"length":214,"name":"error","severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 42 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"error: failed calc node_contracts_count could not serialize access due to concurrent update\n    at /squid/node_modules/pg/lib/client.js:526:17\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async contractCreated (/squid/lib/mappings/contracts.js:106:9)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"},"length":214,"severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 42 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"QueryFailedError: failed calc node_contracts_count could not serialize access due to concurrent update\n    at PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async contractCreated (/squid/lib/mappings/contracts.js:106:9)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"}}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

devnet: dashboard.dev.grid.tf

{"level":1,"time":1724720673634,"ns":"sqd:processor:archive-request","msg":"http body","httpRequestId":721358,"httpResponseBody":{"data":{"status":{"head":12565482}}}}
{"level":5,"time":1724720673834,"ns":"sqd:processor","err":{"query":"INSERT INTO \"node_contract\"(\"id\", \"grid_version\", \"contract_id\", \"twin_id\", \"node_id\", \"deployment_data\", \"deployment_hash\", \"number_of_public_i_ps\", \"state\", \"created_at\", \"solution_provider_id\", \"resources_used_id\") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, DEFAULT) ON CONFLICT ( \"id\" ) DO UPDATE SET \"id\" = EXCLUDED.\"id\", \"grid_version\" = EXCLUDED.\"grid_version\", \"contract_id\" = EXCLUDED.\"contract_id\", \"twin_id\" = EXCLUDED.\"twin_id\", \"node_id\" = EXCLUDED.\"node_id\", \"deployment_data\" = EXCLUDED.\"deployment_data\", \"deployment_hash\" = EXCLUDED.\"deployment_hash\", \"number_of_public_i_ps\" = EXCLUDED.\"number_of_public_i_ps\", \"state\" = EXCLUDED.\"state\", \"created_at\" = EXCLUDED.\"created_at\", \"solution_provider_id\" = EXCLUDED.\"solution_provider_id\"","parameters":["0012565482-000006-b824d",4,"145267",51,176,"{\"version\":3,\"type\":\"network\",\"name\":\"pn6zjc7ud7tbxua\",\"projectName\":\"kubernetes/ugwqj5qzid5zmq4\"}","23087aeda8b4c99740e53643718745ab",0,"Created","1724720658",0],"driverError":{"length":214,"name":"error","severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 42 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"error: failed calc node_contracts_count could not serialize access due to concurrent update\n    at /squid/node_modules/pg/lib/client.js:526:17\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async contractCreated (/squid/lib/mappings/contracts.js:106:9)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"},"length":214,"severity":"ERROR","code":"P0001","where":"PL/pgSQL function reflect_node_contract_changes() line 42 at RAISE","file":"pl_exec.c","line":"3909","routine":"exec_stmt_raise","stack":"QueryFailedError: failed calc node_contracts_count could not serialize access due to concurrent update\n    at PostgresQueryRunner.query (/squid/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async InsertQueryBuilder.execute (/squid/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n    at async Store.upsert (/squid/node_modules/@subsquid/typeorm-store/lib/store.js:34:13)\n    at async contractCreated (/squid/lib/mappings/contracts.js:106:9)\n    at async /squid/lib/processor.js:138:13\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:117:17)"}}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
sameh-farouk commented 2 months ago

The error originating from the processor database is unrelated to the processor stack (GraphQL). It stems from an SQL function (trigger) created in the database by the grid proxy.

@Omarabdul3ziz is checking how this can be resolved from grid-proxy side.

https://github.com/threefoldtech/tfgrid-sdk-go/issues/1172

Omarabdul3ziz commented 2 months ago

update here: https://github.com/threefoldtech/tfgrid-sdk-go/issues/1172#issuecomment-2325771159