search

elliohow / fMRI_ROI_Analysis_Tool

An analysis tool that uses per-voxel statistical maps in conjunction with FSL atlases to create per-region statistical maps. Current usage includes the creation of regional maps of temporal signal to noise ratio.
Apache License 2.0
11 stars 4 forks source link

[JOSS Review] Automated tests? #45

Closed billbrod closed 1 year ago

billbrod commented 1 year ago

I mentioned this briefly in #38 , but then we manually disccussed the installation tests, so I wanted to open a separate issue for this.

Have you thought about automated tests? I understand that fRAT is a GUI and so automated testing is more difficult, but some automated tests to make sure it installs and the installation tests run would be nice. What do you think?

https://github.com/openjournals/joss-reviews/issues/5200

elliohow commented 1 year ago

Isn't the test described on the installation page already testing whether installation has been successful? If there have been any issues with the install, the test won't complete and there will be missing files.

billbrod commented 1 year ago

Yes! But it doesn't look to me like this test is automated -- it will require you to manually run it to make sure any changes don't break it. I believe you can create an automated version of that test using Github actions (or another continuous integration service) which will run every time you push or before you merge a PR (which is especially helpful for testing changes from outside contributors).

The test on the installation page does everything you need, but it's the automation I want to highlight here -- does that make sense?

elliohow commented 1 year ago

Ah yes I understand what you mean now. To run automate the running of the tests, the runner would need to be able to setup FSL and be able to install the example data from OSF. I'm not too familiar with the capabilities of Github actions, is this possible? Theres also the storage limits for a runner. I'm not sure if im misunderstanding but I wouldn't be able to set up github actions for my project without getting GitHub Pro since the example dataset is 1.1GB.

billbrod commented 1 year ago

Oh right, I forgot about FSL. I'm not sure there's a way to install FSL on a github runner that follows their license. You could see if / how nipype does it, I guess. And you're right about the storage limit as well: unless you can come up with an even smaller example dataset, you won't be able to fit it on the runner. You could also check other Continuous Integration services to see if any give you enough storage; it looks like Circle CI might give you enough, but I'm not familiar with them.

So all told, this looks like more work than I expected, and so I don't think this is required for JOSS. I'd encourage you to look into it a bit more, especially if you start getting PRs from outside contributors, because (speaking from experience) it does make maintenance much easier. So I'll go ahead and close this.