palfrey / serial_test

Allows for the creation of serialised Rust tests
https://tevps.net/blog/serial_test/
MIT License
223 stars 16 forks source link

Integration tests are not running in "serial" mode #100

Closed mjovanc closed 1 year ago

mjovanc commented 1 year ago

I'm using the attribute #[serial] on my integration tests since I want them to follow a certain sequential order because I can't currently think of any other way to solve my problem.

This is my structure of integration tests:

#[test]
#[serial]
fn open_db() {
    // code
}

#[test]
#[serial]
fn init_tables() {
    // code
}

#[test]
#[serial]
fn insert_row() {
    // code
}

#[test]
#[serial]
fn drop_table() {
    // code
}

When running cargo test the attributes are not doing what I think this crate should do.

This is my current output:

running 4 tests
test insert_row ... FAILED
test init_tables ... ok
test drop_table ... ok
test open_db ... ok

The desired output would be:

running 4 tests
test open_db ... ok
test init_tables ... ok
test insert_row ... ok
test drop_table ... ok

As you can see the ordering is neccessary. I have even tried to add the flag cargo test -- --test-threads=1 but it was the same issue for some reason.

What could be the cause here?

palfrey commented 1 year ago

The problem is integration tests. Rust integration tests each run in separate processes, so the standard serial doesn't help. file_serial OTOH, will work.

mjovanc commented 1 year ago

Thanks @palfrey for quick answer. I tried adding that but for some reason it does not work either as you can see here in my GitHub workflow: https://github.com/mjovanc/njord/actions/runs/6449784695/job/17508477101

Have I done something wrong?

mjovanc commented 1 year ago

Perhaps I should create helper functions to repeat the tasks before "insert_row" and "drop_table" to avoid the error. But that is not related to this crate and I don't expect any support regarding that. But if that is the case I will look further into writing them differently.

palfrey commented 1 year ago

Perhaps I should create helper functions to repeat the tasks before "insert_row" and "drop_table" to avoid the error. But that is not related to this crate and I don't expect any support regarding that. But if that is the case I will look further into writing them differently.

I'll note your insert_row test doesn't make any tables, and the current ordering means that the drop_table before that deletes the table! Also, although that ordering is causing problems right now, you should write the tests so they're able to cope with any ordering as serial_test (and rust testing in general) provides no guarantees about test ordering.