Code 6 when calling `rly transact raw update-client`

[asmodat]

Gaia and Rly checkouts b617e2b & 2282f8b

I have a full "connected" status: chns(✔️) clnts(✔️) conn(✔️) chan(✔️) however all my rly transact raw update-client calls end with code: 6.

I never had any other response, restarts or re-inits with -f do not help. Any ides ?

Here is example output:

{"height":"0","txhash":"29919AA27188D3295FD127B2CB6F6FDD47633F20F0BEF8B0EC3F880C9FB76880","codespace":"client","code":6,"gas_wanted":"200000","gas_used":"50898"}

[asmodat]

If I try to send standard tx I get the "undefined" error

rly tx transfer kira-alpha kira-1 1ukex true cosmos13tyaljtac4l9jfen4uz338qdmklaupktgtwsrl
I[2020-05-06|03:49:18.916] ✔ [kira-alpha]@{1279} - msg(0:transfer) hash(D9ACFE77E53F5A099E97A3E195C988136A7EB679197AD6C9E910F78AC6EB3C58) 
I[2020-05-06|03:49:25.802] codespace for undefined(1) not found in map  
I[2020-05-06|03:49:25.803] ✘ [kira-1]@{0} - msg(0:update_client,1:ics04/opaque) err(undefined: )

re-push also results in the same err

rly q unrelayed goz_alpha_1
{"src":["1"]}
root@kira-alpha-relayer-test-v4-1:/# rly tx rly goz_alpha_1
I[2020-05-06|03:52:32.031] codespace for undefined(1) not found in map  
I[2020-05-06|03:52:32.032] ✘ [kira-1]@{0} - msg(0:update_client,1:ics04/opaque) err(undefined: ) 

[asmodat]

Same with debug:

rly tx transfer kira-alpha kira-1 1ukex true cosmos13tyaljtac4l9jfen4uz338qdmklaupktgtwsrl --debug
I[2020-05-06|04:18:41.058] ✔ [kira-alpha]@{1630} - msg(0:transfer) hash(F8709CD62953B8C387E10EA5CA5D61603B92B2BFB276983B885D7AC5E74874EF) 
I[2020-05-06|04:18:48.000] - [kira-1] -> sending transaction:           
[{"type":"ibc/client/tendermint/MsgUpdateClient","value":{"client_id":"nhzihoslfo","header":{"SignedHeader":{"header":{"version":{"block":"10","app":"0"},"chain_id":"kira-alpha","height":"1631","time":"2020-05-06T04:18:41.042877306Z","last_block_id":{"hash":"26C9CC2D8389CA1574984B6F039CBE03DA81BE8A77C5878F1246914CF07DA8BA","parts":{"total":"1","hash":"C218E572747C53EF5F56F438D8022F24F9F35C194FDAD011A3909BEB439D5A27"}},"last_commit_hash":"61B55958746D295C74588F88CE02850F8092B3EBCC836EFD48CAF4405E150CDC","data_hash":"","validators_hash":"4E79AD00CE564B39429B0BBBE759ADA5D8B2FB8634C9D70BD9AEB1D2092CFA4B","next_validators_hash":"4E79AD00CE564B39429B0BBBE759ADA5D8B2FB8634C9D70BD9AEB1D2092CFA4B","consensus_hash":"048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F","app_hash":"11F31689926DD2CEA3D8170396BA324C095FA27AFFCEEA90BD837D6B334F9BB7","last_results_hash":"6E340B9CFFB37A989CA544E6BB780A2C78901D3FB33738768511A30617AFA01D","evidence_hash":"","proposer_address":"941FB446D1B9B5804435CB05362A25A08189A42A"},"commit":{"height":"1631","round":"0","block_id":{"hash":"5EEA41C5786828802F2D1661BF8EAB63536BBC671DAF2B65ECC98174245B3142","parts":{"total":"1","hash":"D8E4F1825F90835E61204BE75B6B3641DE9FF31E0B31CBAEECF29CD5FEA8735C"}},"signatures":[{"block_id_flag":2,"validator_address":"941FB446D1B9B5804435CB05362A25A08189A42A","timestamp":"2020-05-06T04:18:46.064908145Z","signature":"oNMpNJnMlIDeDtmu7jGpG81q8uCrwFd9fpjR1B/oUKSJbEb4Lfl82yYttBu9DD5Sar0GQ5q93mow8HeDMpVxDQ=="}]}},"validator_set":{"validators":[{"address":"941FB446D1B9B5804435CB05362A25A08189A42A","pub_key":{"type":"tendermint/PubKeyEd25519","value":"JoglzvRzNFPHSbi4XtsHUe6x184S8lI7lf/QPS8S94I="},"voting_power":"90000","proposer_priority":"0"}],"proposer":{"address":"941FB446D1B9B5804435CB05362A25A08189A42A","pub_key":{"type":"tendermint/PubKeyEd25519","value":"JoglzvRzNFPHSbi4XtsHUe6x184S8lI7lf/QPS8S94I="},"voting_power":"90000","proposer_priority":"0"}}},"address":"cosmos13tyaljtac4l9jfen4uz338qdmklaupktgtwsrl"}},{"type":"ibc/channel/MsgPacket","value":{"packet":{"data":"eyJ0eXBlIjoiaWJjL3RyYW5zZmVyL1BhY2tldERhdGFUcmFuc2ZlciIsInZhbHVlIjp7ImFtb3VudCI6W3siYW1vdW50IjoiMSIsImRlbm9tIjoidHJhbnNmZXIvanhiZ3FpZnlxeS91a2V4In1dLCJyZWNlaXZlciI6ImNvc21vczEzdHlhbGp0YWM0bDlqZmVuNHV6MzM4cWRta2xhdXBrdGd0d3NybCIsInNlbmRlciI6ImNvc21vczEzdHlhbGp0YWM0bDlqZmVuNHV6MzM4cWRta2xhdXBrdGd0d3NybCJ9fQ==","sequence":"1","source_port":"transfer","source_channel":"wgnhiazgrj","destination_port":"transfer","destination_channel":"jxbgqifyqy","timeout_height":"2633","timeout_timestamp":"1588781927973649030"},"proof":{"type":"ibc/commitment/MerkleProof","value":{"proof":{"ops":[{"type":"iavl:v","key":"Y29tbWl0bWVudHMvcG9ydHMvdHJhbnNmZXIvY2hhbm5lbHMvd2duaGlhemdyai9wYWNrZXRzLzI=","data":"kAIKjQIKKQgIEAwY3gwiID80dRUWnkEd+NuzUwjJy9cwndNmtvGVnMaI44GgLYCmCikIBhAHGN4MKiCHqOH2o02VJHQ6p/3uufiwJ6JsAj9p3MrOZq0PSV/TmgopCAQQBBjeDCIgwGrvxPAlt29c0HqtkTyI93r2E3LVTZufOIQHG5SRSdEKKQgCEAIY3gwiIKVodWK9hmkgbalm5MXizu1YVUcMj41C+uRDjvtjfdb4Gl8KOGNvbW1pdG1lbnRzL3BvcnRzL3RyYW5zZmVyL2NoYW5uZWxzL3dnbmhpYXpncmovcGFja2V0cy8yEiDCD3UnWR00epwNDNJcT1gBqqKNPeF+xrIyNqqweY2DUxjeDA=="},{"type":"multistore","key":"aWJj","data":"CuoECjcKDGRpc3RyaWJ1dGlvbhInCiUI3gwSIO5iD/X+93AiUHEt9QOnSTpynqhakiug/pmy2rfbgPj1CjEKBnBhcmFtcxInCiUI3gwSIJRaJvXjfYELj8yuWgr14RPLWl9SWw85vOlT+8iFp30tCjMKCHRyYW5zZmVyEicKJQjeDBIgrXcSzKUSdAKJtxSxG5jzxEvR/ybpMTSdLekqEB/ikQcKMgoHc3Rha2luZxInCiUI3gwSIF34y0gj0PUaS6q8MUFuvc4/AHxw0AemY3LEkviQX2mHCjMKCHNsYXNoaW5nEicKJQjeDBIgQumgf6LWDnzbHQHuRL9a8ZxxPtoARAlWbwnFU9vwxkEKLgoDYWNjEicKJQjeDBIgk8q0ndV1l9VG7zbyrXH5SQgBi3MwmdC9+7921HpbV88KLgoDaWJjEicKJQjeDBIgBdauhhHZyvY3c1qtnaV46O4WtjlO/yedHIQ4htFLWYAKLwoEYmFuaxInCiUI3gwSIJrbQ399To/EwY7OXQdE91zQgekkRukcSTOdPKZiVtwsChAKDm1lbV9jYXBhYmlsaXR5Ci4KA2dvdhInCiUI3gwSIGDNE3wZYuysYWOJ1oA0gz8pIVCcLShapfUVOZfOlop0CjUKCmNhcGFiaWxpdHkSJwolCN4MEiDsunAqhqMCHkCFJaQZSnSrVklCj0hrw4G8oLIBIhWHYwoQCgd1cGdyYWRlEgUKAwjeDAoRCghldmlkZW5jZRIFCgMI3gwKLwoEbWludBInCiUI3gwSIEUfZcaLnR4qRvrfOikPDR0K5FqzPHJF55HSrBbQj6yi"}]}}},"proof_height":"1631","signer":"cosmos13tyaljtac4l9jfen4uz338qdmklaupktgtwsrl"}}]
I[2020-05-06|04:18:48.002] codespace for undefined(1) not found in map  
I[2020-05-06|04:18:48.002] ✘ [kira-1]@{0} - msg(0:update_client,1:ics04/opaque) err(undefined: ) 
{"height":"0","txhash":"A52005DEA8528B64FCD539D4BEAC8C5C5E0EEAE67377B1F1C549276979043F9A","codespace":"undefined","code":1,"raw_log":"internal error","gas_wanted":"200000","gas_used":"67508"}

[asmodat]

And for the client update

rly transact raw update-client kira-alpha kira-1 nhzihoslfo --debug
[{"type":"ibc/client/tendermint/MsgUpdateClient","value":{"client_id":"nhzihoslfo","header":{"SignedHeader":{"header":{"version":{"block":"10","app":"0"},"chain_id":"kira-1","height":"1696","time":"2020-05-06T04:23:47.844157277Z","last_block_id":{"hash":"12265D8DA5C791884E671836637BDBC6DD252D861B399EE1203FEB98414DEFAC","parts":{"total":"1","hash":"1359ADCB7A09B82554BCBDBF90849AAF4B11FAAF492B26021021DFD5B48F1838"}},"last_commit_hash":"C307303849232E05016B3961AFD60514E529A05C285EE0766BB69787F3E68438","data_hash":"","validators_hash":"AD3772B2545EB7AD1C6763E0191546245EF43036E2CB4E67479E1B09AA9776C6","next_validators_hash":"AD3772B2545EB7AD1C6763E0191546245EF43036E2CB4E67479E1B09AA9776C6","consensus_hash":"048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F","app_hash":"D7600D693F9307490A2C1828233DA5686A4D194F6D3BBF0F94A7C4B79DE4F77D","last_results_hash":"","evidence_hash":"","proposer_address":"CA88AEEACCA999554ABF03E63A6353632A7CF8EC"},"commit":{"height":"1696","round":"0","block_id":{"hash":"F6A9250AE125A0A63AAD1752458FBEFA518D881E9BC320A243465D220D20C88D","parts":{"total":"1","hash":"D8AE9AC1A13F44E6950A6DCA7C719E5A81DA78B1A01FAF1FD260E80B3AE78DAF"}},"signatures":[{"block_id_flag":2,"validator_address":"CA88AEEACCA999554ABF03E63A6353632A7CF8EC","timestamp":"2020-05-06T04:23:52.866817802Z","signature":"oRXspLNTpyoHtBJaiQgcG4A5TkqsArYfaoctPUxElDvSFzgzKEuYdMng9NupwIzR9keT0nvbZm9Iu0T5W/LhBw=="}]}},"validator_set":{"validators":[{"address":"CA88AEEACCA999554ABF03E63A6353632A7CF8EC","pub_key":{"type":"tendermint/PubKeyEd25519","value":"K5gfnrcRf+GC+LmZiAHaXjbnt3voV06m9jT6EyMMzCo="},"voting_power":"90000","proposer_priority":"0"}],"proposer":{"address":"CA88AEEACCA999554ABF03E63A6353632A7CF8EC","pub_key":{"type":"tendermint/PubKeyEd25519","value":"K5gfnrcRf+GC+LmZiAHaXjbnt3voV06m9jT6EyMMzCo="},"voting_power":"90000","proposer_priority":"0"}}},"address":"cosmos13tyaljtac4l9jfen4uz338qdmklaupktgtwsrl"}}]
{"height":"0","txhash":"8C6753DDADE0A5E6A93F9206A0BA8895D56C0E75476C23FB6CC407331514F07B","codespace":"client","code":6,"raw_log":"cannot update client with ID nhzihoslfo: client type not found","gas_wanted":"200000","gas_used":"50878"}

[asmodat]

Related Issue: https://github.com/iqlusioninc/relayer/issues/228

[asmodat]

I fixed it as follows:

rly l d $s && rly l d $d && rly pth d $p || : && rly pth gen $s transfer $d transfer $p -f && rly l i $s -f && rly l i $d -f && rly l u $s && rly l u $d && rly tst req $s || : && rly q bal $s -j && rly q bal $d -j && rly tx clnts $p -d &> a; wait && rly tx conn $p -d &> b; wait && rly tx chan $p -d &> c; wait && rly pth s $p && dc=$(rly pth s $p -j | jq -r '.chains.dst."client-id"') && sc=$(rly pth s $p -j | jq -r '.chains.src."client-id"') && while :; do rly tx raw uc $d $s $dc && rly tx raw uc $s $d $sc && sleep 60 ; done