search

citation-style-language / styles

Official repository for Citation Style Language (CSL) citation styles.
https://citationstyles.org/
3.27k stars 3.76k forks source link

automated testing #6483

Open denismaier opened 1 year ago

denismaier commented 1 year ago

For a style I'm currently working on, I wrote myself a small test script that checks the output of the style against the expected results. Would it be possible to implement something like this for this repo. Say, instead of just showing the sample citations, it could also check whether they are correct. Maybe that would also make reviewing pull requests easier. Thoughts @adam3smith @POBrien333 @bwiernik ?

adam3smith commented 1 year ago

I think some automated testing for complex styles is a good idea -- there's at least two existing approaches on this (by Frank and by Cormac), but I'd also be open for something new. It'd be important to make it very easy to create a test set so that this can be reasonably quickly done and added to. I don't think that's currently true for either existing approach. I do still think this is likely not worthwhile doing this for 90%+ of styles, that mainly cover 4-6item types, but covering at least the big style manuals (Vancouver, APA, Chicago, MLA, IEEE, Cite Them Right, MHRA, SBL) and legal styles (OSCOLA, AGLC, Bluebook) would make issues like this morning's IEEE bug less likely to occur and make it easier to update those.

I'd want to test this outside of CI for a bit before adding it to our toolchain: for one, I don't want to create barriers that make it too hard to add new styles. Also, a finicky CI test will actually add more time to maintenance as users will need hand-holding.

denismaier commented 1 year ago

Yes, I know the solutions by @fbennett and @cormacrelf. Unfortunately, both don't work currently. Anyway, the preview uses citeproc-ruby right, so a ruby based testing solution would maybe be a good choice.

And yes, we should be able to generate tests easily.

bwiernik commented 1 year ago

That would be great. (And great for the upcoming update to APA)

fbennett commented 1 year ago

Yes, I know the solutions by @fbennett and @cormacrelf. Unfortunately, both don't work currently.

Would it help to get mine working again, or would you prefer to build a new tool in ruby?

denismaier commented 1 year ago

I guess whatever comes out of this discussion it would be great if your tool was working again.

Would switching to citeproc-js for the CI preview be an option, given that citeproc-js is more faithful than the ruby processor?