search

warden-protocol / wardenprotocol

Monorepo for the Warden Protocol.
https://wardenprotocol.org
Apache License 2.0
1.09k stars 96 forks source link

Integration tests are not working #354

Closed Overseven closed 3 weeks ago

Overseven commented 1 month ago

Currently, integration tests are not working. It falling with warden node never became running error.

Command:

cd tests && just test

Logs:

go test -run 'TestIntegration' -v ./integration_test.go
=== RUN   TestIntegration
=== RUN   TestIntegration/CreateSpace
setup test: CreateSpace
    warden_node.go:83: 
                Error Trace:    /Users/igor/work/cosmos/warden/wardenprotocol/tests/framework/exec/warden_node.go:83
                                                        /usr/local/go/src/runtime/asm_arm64.s:1222
                Error:          Received unexpected error:
                                exec: /var/folders/83/_nhy87k943n6968qf_t87lvw0000gn/T/TestIntegration3218378275/001/wardend --log_no_color start --home /var/folders/83/_nhy87k943n6968qf_t87lvw0000gn/T/TestIntegrationCreateSpace2494717894/001 --x-crisis-skip-assert-invariants
                                err: signal: killed
                                stdout: 
                                stderr: 
                Test:           TestIntegration/CreateSpace
    warden_node.go:91: 
                Error Trace:    /Users/igor/work/cosmos/warden/wardenprotocol/tests/framework/exec/warden_node.go:91
                                                        /Users/igor/work/cosmos/warden/wardenprotocol/tests/cases/create_space.go:26
                                                        /Users/igor/work/cosmos/warden/wardenprotocol/tests/integration_test.go:44
                Error:          Condition never satisfied
                Test:           TestIntegration/CreateSpace
                Messages:       warden node never became running
tests duration: 5.004322542s
--- FAIL: TestIntegration (15.37s)
    --- FAIL: TestIntegration/CreateSpace (5.00s)
FAIL
FAIL    command-line-arguments  15.782s
FAIL
error: Recipe `test` failed on line 3 with exit code 1
Overseven commented 1 month ago

Also, it would be great to write more integration tests that will include all possible user actions.

Pitasi commented 1 month ago

it works for me (on current main branch) so it might be a timing issue. The expected output is this

❯ just test
go test -run 'TestIntegration' -v ./integration_test.go
=== RUN   TestIntegration
=== RUN   TestIntegration/CreateSpace
setup test: CreateSpace
run test: CreateSpace
tests duration: 1.402828709s
--- PASS: TestIntegration (9.93s)
    --- PASS: TestIntegration/CreateSpace (1.40s)
PASS
ok      command-line-arguments  10.242s

In order to understand if it's really just a matter of timing, could you increase the timeout here: https://github.com/warden-protocol/wardenprotocol/blob/dbf46c837138e28e4bda2d3d7078e241d0cba0ef/tests/framework/exec/warden_node.go#L93 from 5 seconds to something like 5 minutes? It should be plenty of time to start a wardend node :)

Overseven commented 1 month ago

Same result:

% just test

go test -run 'TestIntegration' -v ./integration_test.go
=== RUN   TestIntegration
=== RUN   TestIntegration/CreateSpace
setup test: CreateSpace
    warden_node.go:83: 
                Error Trace:    /Users/igor/work/cosmos/warden/wardenprotocol/tests/framework/exec/warden_node.go:83
                                                        /usr/local/go/src/runtime/asm_arm64.s:1222
                Error:          Received unexpected error:
                                exec: /var/folders/83/_nhy87k943n6968qf_t87lvw0000gn/T/TestIntegration1475181834/001/wardend --log_no_color start --home /var/folders/83/_nhy87k943n6968qf_t87lvw0000gn/T/TestIntegrationCreateSpace255166513/001 --x-crisis-skip-assert-invariants
                                err: signal: killed
                                stdout: 
                                stderr: 
                Test:           TestIntegration/CreateSpace
    warden_node.go:91: 
                Error Trace:    /Users/igor/work/cosmos/warden/wardenprotocol/tests/framework/exec/warden_node.go:91
                                                        /Users/igor/work/cosmos/warden/wardenprotocol/tests/cases/create_space.go:26
                                                        /Users/igor/work/cosmos/warden/wardenprotocol/tests/integration_test.go:44
                Error:          Condition never satisfied
                Test:           TestIntegration/CreateSpace
                Messages:       warden node never became running
tests duration: 5m0.012776042s
--- FAIL: TestIntegration (310.41s)
    --- FAIL: TestIntegration/CreateSpace (300.01s)
FAIL
FAIL    command-line-arguments  310.878s
FAIL
error: Recipe `test` failed on line 3 with exit code 1
Pitasi commented 1 month ago

can you change that same line to this:

    }, 5*time.Second, 5*time.Millisecond, "warden node never became running", "stdout", w.Stdout.String(), "stderr", w.Stderr.String())

so we can see the logs from the node

Overseven commented 1 month ago 
Messages: warden node never became running%!(EXTRA string=stdout, string=, string=stderr, string=)
tests duration: 5.00473425s
Pitasi commented 1 month ago

ah gotcha, the right syntax for the log message would be something like this

    }, 5*time.Second, 5*time.Millisecond, "warden node never became running.\nstdout:\n%s\nstderr:\n%s", w.Stdout.String(), w.Stderr.String())
Overseven commented 1 month ago

I also tried this, but didn't get any logs:

require.Eventually(t, func() bool {
  stderr := w.Stderr.String()
  stdout := w.Stderr.String()
  if len(stderr) != 0 {
    println("!!! ERR", stderr)
  }
  if len(stdout) != 0 {
    println("!!! LOG", stdout)
  }
  ...
}
Pitasi commented 1 month ago

I also tried this, but didn't get any logs:

require.Eventually(t, func() bool {
  stderr := w.Stderr.String()
  stdout := w.Stderr.String()
  if len(stderr) != 0 {
    println("!!! ERR", stderr)
  }
  if len(stdout) != 0 {
    println("!!! LOG", stdout)
  }
  ...
}

stdout := w.Stderr.String()

should be

stdout := w.Stdout.String()

Overseven commented 1 month ago

}, 5time.Second, 5time.Millisecond, "warden node never became running.\nstdout:\n%s\nstderr:\n%s", w.Stdout.String(), w.Stderr.String())

Messages:       warden node never became running.
                                stdout:

                                stderr:
tests duration: 5.006430208s
Overseven commented 1 month ago

stdout := w.Stderr.String()

should be

stdout := w.Stdout.String()

Fixed, but got the same result

Pitasi commented 1 month ago

so strange, it works for me

!!! LOG 5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000f2bae0, group}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000d67be0, icacontroller}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x1400052a780, acc}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000eb8eb0, crisis}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000b883d0, slashing}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000e7c780, gov}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000f59bc0, circuit}" version=0
...
Overseven commented 1 month ago

Do I need to make a snapshot or run a node before starting the test?

Pitasi commented 1 month ago

no, i just cloned this repo and only run just test and it worked for me, without doing anything else 🤔

Pitasi commented 1 month ago

I'm a bit out of ideas, maybe you can try running with a debugger attached? It's so weird that the stdout is not being captured

what OS are you using?

@optifat can you try to run these and see if they work for you?

Overseven commented 1 month ago

For more information - I'm running it on Mac M1. I noticed that some commands can't be executed without editing. For example, this line from just localnet falling with error:

sed -i 's/cors_allowed_origins = \[\]/cors_allowed_origins = ["*"]/' ~/.warden/config/config.toml

Error:

sed: 1: "/Users/igor/.warden/con ...": command i expects \ followed by text

And it works only this way:

sed -i'' -e 's/cors_allowed_origins = \[\]/cors_allowed_origins = ["*"]/' ~/.warden/config/config.toml
Pitasi commented 1 month ago

I'm also on Mac and you're right, that sed command only works if you have GNU sed installed (brew install gnu-sed). Still I don't understand why the tests are not working :(

optifat commented 1 month ago

I'm a bit out of ideas, maybe you can try running with a debugger attached? It's so weird that the stdout is not being captured

what OS are you using?

@optifat can you try to run these and see if they work for you?

No, I got

run test: CreateSpace
    warden_cli.go:36: 
            Error Trace:    /Users/pavel/warden/wardenprotocol/tests/framework/exec/warden_cli.go:36
                                        /Users/pavel/warden/wardenprotocol/tests/cases/create_space.go:31
                                        /Users/pavel/warden/wardenprotocol/tests/integration_test.go:46
            Error:          Received unexpected error:
                            exec: sh -c /var/folders/v2/cjl08_v9281cygqjl_vf60lm0000gn/T/TestIntegration1010986435/001/wardend tx warden new-space --from alice --yes --node tcp://127.0.0.1:64518 --home /var/folders/v2/cjl08_v9281cygqjl_vf60lm0000gn/T/TestIntegrationCreateSpace3004294498/001 --chain-id warden --keyring-backend test --keyring-dir /var/folders/v2/cjl08_v9281cygqjl_vf60lm0000gn/T/TestIntegrationCreateSpace3004294498/001
                            err: exit status 1
                            stdout: 
                            stderr: rpc error: code = Unknown desc = warden is not ready; please wait for first block: invalid height
            Test:           TestIntegration/CreateSpace
tests duration: 2.067708833s
Pitasi commented 3 weeks ago

I'm going to close this as we couldn't reproduce on several machines.