Closed eliteprox closed 2 months ago
Runtime error occurred on production orchestrator:
I0516 16:39:30.673213 1 block_watcher.go:454] Polling blocks from=211907963 to=211907982 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2729f64] goroutine 723 [running]: math/big.(*Rat).Num(...) /usr/local/go/src/math/big/rat.go:409 github.com/livepeer/go-livepeer/core.(*orchestrator).priceInfo(0xc000ea9090, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, 0xdf, ...}, ...) /src/core/orchestrator.go:374 +0x204 github.com/livepeer/go-livepeer/core.(*orchestrator).PriceInfo(0xc0009937b8?, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, 0xdf, ...}, ...) /src/core/orchestrator.go:287 +0x5a github.com/livepeer/go-livepeer/server.getPriceInfo({0xb22b40?, 0xc000ea9090?}, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, ...}, ...) /src/server/rpc.go:363 +0xe4 github.com/livepeer/go-livepeer/server.orchestratorInfoWithCaps({0xb22b40, 0xc000ea9090}, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, ...}, ...) /src/server/rpc.go:374 +0xaa github.com/livepeer/go-livepeer/server.orchestratorInfo(...) /src/server/rpc.go:367 github.com/livepeer/go-livepeer/server.getOrchestrator({0xb22b40, 0xc000ea9090}, 0xc000877440) /src/server/rpc.go:341 +0x2b2 github.com/livepeer/go-livepeer/server.(*lphttp).GetOrchestrator(0x5138c0?, {0xc000877440?, 0x18d03c6?}, 0x0?) /src/server/rpc.go:185 +0x25 github.com/livepeer/go-livepeer/net._Orchestrator_GetOrchestrator_Handler({0x57ab20?, 0xc000579500}, {0xb10a50, 0xc0007337d0}, 0xc00078a9a0, 0x0) /src/net/lp_rpc_grpc.pb.go:109 +0x169 google.golang.org/grpc.(*Server).processUnaryRPC(0xc000e4e1e0, {0xb19d20, 0xc0006666e0}, 0xc000a9c120, 0xc000579590, 0x321ddc0, 0x0) /go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:1358 +0xe15 google.golang.org/grpc.(*Server).handleStream(0xc000e4e1e0, {0xb19d20, 0xc0006666e0}, 0xc000a9c120, 0x0) /go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:1735 +0x9e7 google.golang.org/grpc.(*Server).serveStreams.func1.1() /go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:970 +0xbb created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 693 /go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:981 +0x145
Just backlogging this for now, will check more into it shortly
Run production AI Orchestrator with following command
docker run -d --restart unless-stopped -v /livepeer/ai/data:/root/.lpData/ -v /var/run/docker.sock:/var/run/docker.sock --network host --gpus all livepeer/go-livepeer:ai-video -network=arbitrum-one-mainnet -orchestrator -transcoder -serviceAddr 76.215.8.207:8936 -v 6 -nvidia "all" -aiWorker -aiModels /root/.lpData/aiModels.json -aiModelsDir /livepeer/ai/data/models -ticketEV 2999999999999 -ethUrl ***** -ethOrchAddr 0x104a7ca059a35fd4def5ecb16600b2caa1fe1361 -ethPassword *** -ethAcctAddr 0x0074780FefF1FD0277FAD6ccdb5a29908df6051F
aiModels.json config:
[ { "pipeline": "text-to-image", "model_id": "SG161222/RealVisXL_V4.0_Lightning", "price_per_unit": 4768371, "warm": true } ]
Orchestrator should not crash
Major
No response
None
docker pull ghcr.io/livepeer/go-livepeer/build:sha-816920a51e883ac1e690fc15b7837e2189aa130b
816920a51e883ac1e690fc15b7837e2189aa130b
Issue seems to be resolved by adding -pricePerPixel into startup flag #3047
-pricePerPixel
Fixed in https://github.com/livepeer/go-livepeer/commit/b7a9fb72f8d98eef5164a5c7e8b6820c9e9370bc.
Describe the bug
Runtime error occurred on production orchestrator:
Just backlogging this for now, will check more into it shortly
Reproduction steps
Run production AI Orchestrator with following command
docker run -d --restart unless-stopped -v /livepeer/ai/data:/root/.lpData/ -v /var/run/docker.sock:/var/run/docker.sock --network host --gpus all livepeer/go-livepeer:ai-video -network=arbitrum-one-mainnet -orchestrator -transcoder -serviceAddr 76.215.8.207:8936 -v 6 -nvidia "all" -aiWorker -aiModels /root/.lpData/aiModels.json -aiModelsDir /livepeer/ai/data/models -ticketEV 2999999999999 -ethUrl ***** -ethOrchAddr 0x104a7ca059a35fd4def5ecb16600b2caa1fe1361 -ethPassword *** -ethAcctAddr 0x0074780FefF1FD0277FAD6ccdb5a29908df6051F
aiModels.json config:
Expected behaviour
Orchestrator should not crash
Severity
Major
Screenshots / Live demo link
No response
OS
None
Running on
None
AI go-livepeer version
docker pull ghcr.io/livepeer/go-livepeer/build:sha-816920a51e883ac1e690fc15b7837e2189aa130b
AI go-livepeer commit hash
816920a51e883ac1e690fc15b7837e2189aa130b
Additional context
No response