search

kaist-cp / cs431

1.42k stars 135 forks source link

[Question] Meaning of 'filtered out' in the test result #904

Closed GWS0428 closed 6 months ago

GWS0428 commented 6 months ago

After submitting my code to gg, I received the 75/100 (score, HW1). However, upon reviewing the log file, it states 'test result : OK' for all tests. The only explanation I can deduce for receiving a partial score is that the log file sometimes shows 'filered out 3' as follows.

What is the meaning of 'filtered out'? 


test cache_no_duplicate_sequential ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

    Finished test profile [unoptimized + debuginfo] target(s) in 0.07s
     Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8) ```
Lee-Janggun commented 6 months ago

In the grade script, we run the tests one by one in separate 'cargo test' invocations to give partial grades. '3 filtered out' means the three tests that were not ran in that specific 'cargo test' invocation.

08kmc09 commented 6 months ago

https://doc.rust-lang.org/book/ch11-02-running-tests.html

You may refer to the Rust book for more information about test.

Oh I didn't fully understand about your question. It's quite misleading answer.

Lee-Janggun commented 6 months ago

Although... the fact that it said ok but you didn't get points is a bit weird. Can you post the full log?

GWS0428 commented 6 months ago

Here is my full log


===== SETUP =====

===== CHECK =====
[2024-03-30 09:43:25,497: WARNING/ForkPoolWorker-32] Demoting to runner...
info: downloading installer
info: profile set to 'default'
info: default host triple is x86_64-unknown-linux-gnu
warning: Updating existing toolchain, profile choice will be ignored
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

  stable-x86_64-unknown-linux-gnu unchanged - rustc 1.77.1 (7cf61ebde 2024-03-27)

Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, you need to source
the corresponding env file under $HOME/.cargo.

This is usually done by running one of the following (note the leading DOT):
. "$HOME/.cargo/env"            # For sh/bash/zsh/ash/dash/pdksh
source "$HOME/.cargo/env.fish"  # For fish
info: component 'rust-src' is up to date
Archive:  submission.zip
  inflating: submission/cache.rs     
  inflating: submission/tcp.rs       
  inflating: submission/thread_pool.rs  
Cloning into 'cs431'...
info: syncing channel updates for 'nightly-2024-03-13-x86_64-unknown-linux-gnu'

  nightly-2024-03-13-x86_64-unknown-linux-gnu unchanged - rustc 1.78.0-nightly (a165f1f65 2024-03-12)

info: checking for self-update
info: component 'rust-src' is up to date
    Updating crates.io index
    Updating git repository `https://github.com/kaist-cp/cs431`
   Compiling crossbeam-utils v0.8.19
   Compiling libc v0.2.153
    Checking cfg-if v1.0.0
   Compiling rayon-core v1.12.1
    Checking memchr v2.7.2
    Checking regex-syntax v0.8.3
    Checking ppv-lite86 v0.2.17
    Checking either v1.10.0
    Checking aho-corasick v1.1.3
    Checking crossbeam-epoch v0.9.18
    Checking crossbeam-channel v0.5.12
    Checking crossbeam-deque v0.8.5
    Checking cs431 v0.1.0 (https://github.com/kaist-cp/cs431#0bb98050)
    Checking getrandom v0.2.12
    Checking rand_core v0.6.4
    Checking rand_chacha v0.3.1
    Checking rayon v1.10.0
    Checking rand v0.8.5
    Checking regex-automata v0.4.6
    Checking regex v1.10.4
    Checking cs431-homework v0.1.0 (/app/files/checks/8c1728a7-85fc-404c-9d70-9034ef6460b6/cs431/homework)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5.33s
Running with cargo...
    Testing basic functionalities of cache.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s
     Running tests/cache.rs (target/debug/deps/cache-d94123a79a74e29b)

running 1 test
test cache_no_duplicate_sequential ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s
     Running tests/cache.rs (target/debug/deps/cache-d94123a79a74e29b)

running 1 test
test cache_no_duplicate_concurrent ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.02s

    Testing nonblockingness of cache.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s
     Running tests/cache.rs (target/debug/deps/cache-d94123a79a74e29b)

running 1 test
test cache_no_block_disjoint ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.03s
     Running tests/cache.rs (target/debug/deps/cache-d94123a79a74e29b)

running 1 test
Test timed out: cargo test  --test cache -- --exact cache_no_reader_block
    Testing tcp.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.04s
     Running tests/tcp.rs (target/debug/deps/tcp-343222ce4ee4b2ec)

running 1 test
test cancellable_listener_cancel ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

    Testing thread_pool.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.04s
     Running tests/thread_pool.rs (target/debug/deps/thread_pool-4f1bb7d517938467)

running 4 tests
test thread_pool_parallel ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_join_block ... ok
test thread_pool_drop_block ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.06s

Running with cargo --release...
    Testing basic functionalities of cache.rs...
    Finished `release` profile [optimized] target(s) in 0.03s
     Running tests/cache.rs (target/release/deps/cache-5610340c0b6d1bed)

running 1 test
test cache_no_duplicate_sequential ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

    Finished `release` profile [optimized] target(s) in 0.03s
     Running tests/cache.rs (target/release/deps/cache-5610340c0b6d1bed)

running 1 test
test cache_no_duplicate_concurrent ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.01s

    Testing tcp.rs...
    Finished `release` profile [optimized] target(s) in 0.03s
     Running tests/tcp.rs (target/release/deps/tcp-3f622f7dd3aec98a)

running 1 test
test cancellable_listener_cancel ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

    Testing thread_pool.rs...
    Finished `release` profile [optimized] target(s) in 0.03s
     Running tests/thread_pool.rs (target/release/deps/thread_pool-add72be625ad535a)

running 4 tests
test thread_pool_parallel ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_join_block ... ok
test thread_pool_drop_block ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.05s

Running with cargo_asan...
    Testing basic functionalities of cache.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
     Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8)

running 1 test
test cache_no_duplicate_sequential ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
     Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8)

running 1 test
test cache_no_duplicate_concurrent ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.11s

    Testing tcp.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
     Running tests/tcp.rs (target/x86_64-unknown-linux-gnu/debug/deps/tcp-3b819169d687e46a)

running 1 test
test cancellable_listener_cancel ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

    Testing thread_pool.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
     Running tests/thread_pool.rs (target/x86_64-unknown-linux-gnu/debug/deps/thread_pool-811f0a12a0be79b6)

running 4 tests
test thread_pool_parallel ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_drop_block ... ok
test thread_pool_join_block ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.07s

Running with cargo_asan --release...
    Testing basic functionalities of cache.rs...
    Finished `release` profile [optimized] target(s) in 0.07s
     Running tests/cache.rs (target/x86_64-unknown-linux-gnu/release/deps/cache-e176c6053387d7e7)

running 1 test
test cache_no_duplicate_sequential ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

    Finished `release` profile [optimized] target(s) in 0.08s
     Running tests/cache.rs (target/x86_64-unknown-linux-gnu/release/deps/cache-e176c6053387d7e7)

running 1 test
test cache_no_duplicate_concurrent ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.03s

    Testing tcp.rs...
    Finished `release` profile [optimized] target(s) in 0.08s
     Running tests/tcp.rs (target/x86_64-unknown-linux-gnu/release/deps/tcp-baa0928a468c4699)

running 1 test
test cancellable_listener_cancel ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

    Testing thread_pool.rs...
    Finished `release` profile [optimized] target(s) in 0.07s
     Running tests/thread_pool.rs (target/x86_64-unknown-linux-gnu/release/deps/thread_pool-21e837687b5387d2)

running 4 tests
test thread_pool_parallel ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_drop_block ... ok
test thread_pool_join_block ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.06s

Running with cargo_tsan...
    Testing basic functionalities of cache.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
     Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8)

running 1 test
test cache_no_duplicate_sequential ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
     Running tests/cache.rs (target/x86_64-unknown-linux-gnu/debug/deps/cache-6939cfa3b319d7f8)

running 1 test
test cache_no_duplicate_concurrent ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.11s

    Testing tcp.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
     Running tests/tcp.rs (target/x86_64-unknown-linux-gnu/debug/deps/tcp-3b819169d687e46a)

running 1 test
test cancellable_listener_cancel ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

    Testing thread_pool.rs...
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
     Running tests/thread_pool.rs (target/x86_64-unknown-linux-gnu/debug/deps/thread_pool-811f0a12a0be79b6)

running 4 tests
test thread_pool_drop_block ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_join_block ... ok
test thread_pool_parallel ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.14s

Running with cargo_tsan --release...
    Testing basic functionalities of cache.rs...
    Finished `release` profile [optimized] target(s) in 0.07s
     Running tests/cache.rs (target/x86_64-unknown-linux-gnu/release/deps/cache-e176c6053387d7e7)

running 1 test
test cache_no_duplicate_sequential ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

    Finished `release` profile [optimized] target(s) in 0.06s
     Running tests/cache.rs (target/x86_64-unknown-linux-gnu/release/deps/cache-e176c6053387d7e7)

running 1 test
test cache_no_duplicate_concurrent ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.04s

    Testing tcp.rs...
    Finished `release` profile [optimized] target(s) in 0.06s
     Running tests/tcp.rs (target/x86_64-unknown-linux-gnu/release/deps/tcp-baa0928a468c4699)

running 1 test
test cancellable_listener_cancel ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

    Testing thread_pool.rs...
    Finished `release` profile [optimized] target(s) in 0.07s
     Running tests/thread_pool.rs (target/x86_64-unknown-linux-gnu/release/deps/thread_pool-21e837687b5387d2)

running 4 tests
test thread_pool_drop_block ... ok
test thread_pool_drop_propagate_panic - should panic ... ok
test thread_pool_join_block ... ok
test thread_pool_parallel ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.12s

Score: 75 / 100```
GWS0428 commented 6 months ago

And... I ran the test again and achieved a full score this time. I modified one line of code, which might have affected the test result. However, I don't understand why I received a partial score before, even though all tests had passed. It's strange... :(

Lee-Janggun commented 6 months ago

If you look at the log, you have a 'test timed out', which is the failed test. If a test fails without sanitizers, we don't run it with sanitizers. This is why you didn't see the tests being run near the end of the log.

GWS0428 commented 6 months ago

Oh I see..! Thank you very much for your help! Fortunately, my modification achieved a full score without receiving a timeout message.

Additionally, I am curious to know: If I achieve a full score on gg, does that mean I got 100/100? Or will there be another evaluation (running test again, or extra test cases) of my code after the due date?

Lee-Janggun commented 6 months ago

If you got a "Yes" on the submission status and the score says 100/100 on the log, you got a 100/100