External backends running unit tests of Spectra?

[jorainer]

This comes from an idea/suggestion from @sneumann at the metaRbolomics hackathon 2021: We have MsBackend implementations in external packages. There is currently no way that we can ensure that these work properly or return the data in the expected format. I see there two solutions:

1) we leave this to the developers of the external backend, i.e. it's their responsibility to add corresponding unit tests in their package. 2) we have some unit tests (in Spectra) that can be called from external packages. No idea if and how this is working, but the idea would be that these unit tests allow to provide a MsBackend as input and the test performs the most critical tests, such as:

• check that the returned/provided data types are correct.
• check that subsetting works
• check that it is possible to add spectra variables
• ?

Any thoughts, ideas welcome.

[sgibb]

1. we leave this to the developers of the external backend, i.e. it's their responsibility to add corresponding unit tests in their package.

I guess that's the way we have to go because everything else would be very hard.

Regarding the second solution: we could provide an example mzML and a template testthat file where developers just have to plugin their backend and previously import the example mzML into their backend source. Or we could provide a function e.g.:

testSpectraBackend(Spectra(
  source = "ourExampleConvertedFromMzMlIntoExternalBackend", 
  backend = MsBackend())) {
  # check mz/intensity values
  stopifnot(
    all(mz(spectra) == c(1, 2, 3, 4, 5)),
    all(mz(spectra) == c(1, 2, 3, 4, 5))
  )
  # ...
}

I guess no user ever calls testthat::test_package(). But if we provide such a function and promote it in the vignette, e.g.

always test your backend before

library("Spectra")
library("MsBackendExternal")
data("ExampleSpectrum", package = "MsBackendExternal")
testSpectraBackend(Spectra(ExampleSpectrum, backend = MsBackendExternal))
# TRUE
# or
# Error: mz is not 1:5 or whatever

[sneumann]

I would still be optimistic that there is a way for 2. we'd need testthat::test_dir(Spectra/tests/backendtests) and tests in there need to get the backend and file in that format as a variables or options. Yours, Steffen

[hechth]

So we created some framework and think that defining an abstract set of tests based on a global variable (the backend) and the expected output. The backends can then call the tests and before that define the global variable to be the initialized backend.

[sneumann]

An example for a generic "testsuite" is now in https://github.com/rformassspectrometry/Spectra/blob/eb5f549c68f1a7962aa7fd776f32f62a76bfd155/inst/test_msmslibraries/test_MsBackendMsmslibraries.R

and it is called through https://github.com/rformassspectrometry/MsBackendMsp/blob/master/tests/testthat.R#L10

Yours, Steffen