livepeer / go-livepeer

Official Go implementation of the Livepeer protocol
http://livepeer.org
MIT License
537 stars 165 forks source link

Orchestrator crash, possibly related to pricing #3057

Closed eliteprox closed 2 months ago

eliteprox commented 2 months ago

Describe the bug

Runtime error occurred on production orchestrator:

I0516 16:39:30.673213       1 block_watcher.go:454] Polling blocks from=211907963 to=211907982
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2729f64]

goroutine 723 [running]:
math/big.(*Rat).Num(...)
        /usr/local/go/src/math/big/rat.go:409
github.com/livepeer/go-livepeer/core.(*orchestrator).priceInfo(0xc000ea9090, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, 0xdf, ...}, ...)
        /src/core/orchestrator.go:374 +0x204
github.com/livepeer/go-livepeer/core.(*orchestrator).PriceInfo(0xc0009937b8?, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, 0xdf, ...}, ...)
        /src/core/orchestrator.go:287 +0x5a
github.com/livepeer/go-livepeer/server.getPriceInfo({0xb22b40?, 0xc000ea9090?}, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, ...}, ...)
        /src/server/rpc.go:363 +0xe4
github.com/livepeer/go-livepeer/server.orchestratorInfoWithCaps({0xb22b40, 0xc000ea9090}, {0x68, 0xd6, 0xff, 0x39, 0x38, 0xff, 0x63, 0xd2, ...}, ...)
        /src/server/rpc.go:374 +0xaa
github.com/livepeer/go-livepeer/server.orchestratorInfo(...)
        /src/server/rpc.go:367
github.com/livepeer/go-livepeer/server.getOrchestrator({0xb22b40, 0xc000ea9090}, 0xc000877440)
        /src/server/rpc.go:341 +0x2b2
github.com/livepeer/go-livepeer/server.(*lphttp).GetOrchestrator(0x5138c0?, {0xc000877440?, 0x18d03c6?}, 0x0?)
        /src/server/rpc.go:185 +0x25
github.com/livepeer/go-livepeer/net._Orchestrator_GetOrchestrator_Handler({0x57ab20?, 0xc000579500}, {0xb10a50, 0xc0007337d0}, 0xc00078a9a0, 0x0)
        /src/net/lp_rpc_grpc.pb.go:109 +0x169
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000e4e1e0, {0xb19d20, 0xc0006666e0}, 0xc000a9c120, 0xc000579590, 0x321ddc0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:1358 +0xe15
google.golang.org/grpc.(*Server).handleStream(0xc000e4e1e0, {0xb19d20, 0xc0006666e0}, 0xc000a9c120, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:1735 +0x9e7
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:970 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 693
        /go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:981 +0x145

Just backlogging this for now, will check more into it shortly

Reproduction steps

Run production AI Orchestrator with following command

docker run -d --restart unless-stopped -v /livepeer/ai/data:/root/.lpData/ -v /var/run/docker.sock:/var/run/docker.sock --network host --gpus all livepeer/go-livepeer:ai-video -network=arbitrum-one-mainnet -orchestrator -transcoder -serviceAddr 76.215.8.207:8936 -v 6 -nvidia "all" -aiWorker -aiModels /root/.lpData/aiModels.json -aiModelsDir /livepeer/ai/data/models -ticketEV 2999999999999 -ethUrl ***** -ethOrchAddr 0x104a7ca059a35fd4def5ecb16600b2caa1fe1361 -ethPassword *** -ethAcctAddr 0x0074780FefF1FD0277FAD6ccdb5a29908df6051F

aiModels.json config:

[
    {
        "pipeline": "text-to-image",
        "model_id": "SG161222/RealVisXL_V4.0_Lightning",
        "price_per_unit": 4768371,
        "warm": true
    }
]

Expected behaviour

Orchestrator should not crash

Severity

Major

Screenshots / Live demo link

No response

OS

None

Running on

None

AI go-livepeer version

docker pull ghcr.io/livepeer/go-livepeer/build:sha-816920a51e883ac1e690fc15b7837e2189aa130b

AI go-livepeer commit hash

816920a51e883ac1e690fc15b7837e2189aa130b

Additional context

No response

eliteprox commented 2 months ago

Issue seems to be resolved by adding -pricePerPixel into startup flag #3047

rickstaa commented 2 months ago

Fixed in https://github.com/livepeer/go-livepeer/commit/b7a9fb72f8d98eef5164a5c7e8b6820c9e9370bc.