When using the self-host bfgd, got response with 0 keystones.

YukiCoco commented 1 month ago

Confirmation

[X] I have checked for similar issues.

Describe the bug

I am trying to run a bfgd service and then use popmd to connect, but I encountered an issue where the bfgd server returns Got response with 0 keystones. At the same time, bfgd keeps logging these messages repeatedly. Could this be because bfgd is still syncing its height? Do we need a clearer progress indicator for bfgd? popmd log:

2024-09-20 06:03:45 TRACE popm popm.go:896 bfg                                                                                                         
2024-09-20 06:03:45 TRACE popm popm.go:833 connectBFG                                                                                                  
2024-09-20 06:03:45 TRACE auth secp256k1.go:146 HandshakeClient                                                                                        
2024-09-20 06:03:45 TRACE auth secp256k1.go:165 (*auth.Secp256k1HelloChallenge)(0xc00051e020)({                                                        
 Message: (string) (len=234) "c90faf54357617f8a59b80421bad000948656c6c6f3a20303265363132383463653466656434346363363438383963616330376161366361326536646
4306563613063373030666662343938323565383836666466376566330a4d6573736167653a204920616d206e6f74206120726f626f74210a"                                     
})                                                                                                                                                     
2024-09-20 06:03:45 TRACE auth secp256k1.go:76 handleSecp256k1HelloChallenge                                                                           
2024-09-20 06:03:45 TRACE auth secp256k1.go:90 handleSecp256k1HelloChallenge exit                                                                      
2024-09-20 06:03:45 TRACE auth secp256k1.go:186 HandshakeClient complete                                                                               
2024-09-20 06:03:45 TRACE auth secp256k1.go:187 HandshakeClient exit                                                                                   
2024-09-20 06:03:45 DEBUG popm popm.go:876 Connected to BFG: wss://testnet.example.com/v1/ws/public                                                    
2024-09-20 06:03:45 TRACE popm popm.go:820 handleBFGWebsocketCall                                                                                      
2024-09-20 06:03:45 TRACE popm popm.go:773 handleBFGWebsocketRead                                                                                      
2024-09-20 06:04:05 TRACE popm popm.go:691 Checking for new keystone headers...                                                                        
2024-09-20 06:04:05 TRACE popm popm.go:656 callBFG *bfgapi.L2KeystonesRequest                                                                          
2024-09-20 06:04:05 TRACE popm popm.go:745 handleBFGCallCompletion                                                                                     
2024-09-20 06:04:05 TRACE popm popm.go:751 handleBFGCallCompletion: (*bfgapi.L2KeystonesRequest)(0xc0006820a0)({                                       
 NumL2Keystones: (uint64) 3                                                                                                                            
})                                                                                                                                                     

2024-09-20 06:04:05 TRACE popm popm.go:765 handleBFGCallCompletion returned: (*bfgapi.L2KeystonesResponse)(0xc000002060)({                             
 L2Keystones: ([]hemi.L2Keystone) {                                                                                                                    
 },                                                                                                                                                    
 Error: (*protocol.Error)(<nil>)                                                                                                                       
})                                                                                                                                                     

2024-09-20 06:04:05 TRACE popm popm.go:768 handleBFGCallCompletion exit        
2024-09-20 06:04:05 TRACE popm popm.go:684 callBFG exit *bfgapi.L2KeystonesRequest                                                                     
2024-09-20 06:04:05 TRACE popm popm.go:711 Got response with 0 keystones                                                                               
2024-09-20 06:05:05 TRACE popm popm.go:691 Checking for new keystone headers...                                                                        
2024-09-20 06:05:05 TRACE popm popm.go:656 callBFG *bfgapi.L2KeystonesRequest
2024-09-20 06:05:05 TRACE popm popm.go:745 handleBFGCallCompletion
2024-09-20 06:05:05 TRACE popm popm.go:751 handleBFGCallCompletion: (*bfgapi.L2KeystonesRequest)(0xc000514258)({
 NumL2Keystones: (uint64) 3
})

2024-09-20 06:05:05 TRACE popm popm.go:765 handleBFGCallCompletion returned: (*bfgapi.L2KeystonesResponse)(0xc0001763e0)({
 L2Keystones: ([]hemi.L2Keystone) {
 },
 Error: (*protocol.Error)(<nil>)
})

2024-09-20 06:05:05 TRACE popm popm.go:768 handleBFGCallCompletion exit
2024-09-20 06:05:05 TRACE popm popm.go:684 callBFG exit *bfgapi.L2KeystonesRequest
2024-09-20 06:05:05 TRACE popm popm.go:711 Got response with 0 keystones
2024-09-20 06:06:05 TRACE popm popm.go:691 Checking for new keystone headers...
2024-09-20 06:06:05 TRACE popm popm.go:656 callBFG *bfgapi.L2KeystonesRequest
2024-09-20 06:06:05 TRACE popm popm.go:745 handleBFGCallCompletion
2024-09-20 06:06:05 TRACE popm popm.go:751 handleBFGCallCompletion: (*bfgapi.L2KeystonesRequest)(0xc000514038)({
 NumL2Keystones: (uint64) 3
})

2024-09-20 06:06:05 TRACE popm popm.go:765 handleBFGCallCompletion returned: (*bfgapi.L2KeystonesResponse)(0xc000176280)({
 L2Keystones: ([]hemi.L2Keystone) {
 },
 Error: (*protocol.Error)(<nil>)
})

2024-09-20 06:06:05 TRACE popm popm.go:768 handleBFGCallCompletion exit
2024-09-20 06:06:05 TRACE popm popm.go:684 callBFG exit *bfgapi.L2KeystonesRequest
2024-09-20 06:06:05 TRACE popm popm.go:711 Got response with 0 keystones
2024-09-20 06:07:05 TRACE popm popm.go:691 Checking for new keystone headers...
2024-09-20 06:07:05 TRACE popm popm.go:656 callBFG *bfgapi.L2KeystonesRequest
2024-09-20 06:07:05 TRACE popm popm.go:745 handleBFGCallCompletion

...

bfgd log:

Expected behaviour

popmd behaves the same when connected to wss://testnet.rpc.hemi.network/v1/ws/public.

Environment

Version: Hemi Network v0.4.3 Operating System: Debian 12 Architecture: amd64

Additional Information

No response

NorthernWinter commented 1 month ago

same issue BFGD 2024-09-20 07:44:26 DEBUG bfg bfg.go:315 Responding to bfgapi-l2-keystones-request request with (bfgapi.L2KeystonesResponse)(0xc0001f4860)({ L2Keystones: ([]hemi.L2Keystone) { }, Error: (protocol.Error)() })

my popmd 2024-09-20 07:37:45 DEBUG popm popm.go:876 Connected to BFG: ws://localhost:8383/v1/ws/public

ClaytonNorthey92 commented 1 month ago

hey @YukiCoco thanks for opening this issue.

your BFG will have to receive keystones from some BSS hooked up to an op-node. Otherwise, it will just continue to return no keystones because it has none.

long story short: op-node is our l2 chain, when a keystone is reached in op-node, it will "notify" bss, which will in turn notify bfg.

I realize that setting up an op-node in non-sequencing mode to receive keystones is a lot to ask.

@max-sanchez @jcvernaleo thinking this through, asking the community to set up a non-sequencing op-node just to get keystones seems a bit heavy...I wonder if there is some way we could expose this via our l2 rpc calls? 🤔 perhaps adding a way to poll keystones from bss instead of wait for notifications? (giving users the option to do either)

YukiCoco commented 1 month ago

hey @YukiCoco thanks for opening this issue.

your BFG will have to receive keystones from some BSS hooked up to an op-node. Otherwise, it will just continue to return no keystones because it has none.

long story short: op-node is our l2 chain, when a keystone is reached in op-node, it will "notify" bss, which will in turn notify bfg.

I realize that setting up an op-node in non-sequencing mode to receive keystones is a lot to ask.

@max-sanchez @jcvernaleo thinking this through, asking the community to set up a non-sequencing op-node just to get keystones seems a bit heavy...I wonder if there is some way we could expose this via our l2 rpc calls? 🤔 perhaps adding a way to poll keystones from bss instead of wait for notifications? (giving users the option to do either)

Thank you for your reply! This really sounds like a challenge. 😂

ClaytonNorthey92 commented 1 month ago

@max-sanchez thinking this through a little more...

what do you think of having an RPC call in op-node that returns keystones? so, rather than op-node needing to explicitly connect to a BSS instance (i.e. a node runner would have to run an op-node p2p with ours and their own BSS to receive keystones), we could optionally have BSS connect to op-node and make periodic RPC calls to get keystones. That way it would be really easy to run a BSS and connect it to one of our network's op-node rpcs. 🤔

hemilabs / heminetwork