openbases / openbases-python

Open Bases python helpers for https://openbases.github.io
https://openbases.github.io/openbases-python/
Other
1 stars 0 forks source link

Add testing suite for paper markdown #12

Closed vsoch closed 6 years ago

vsoch commented 6 years ago

should be able to define a recipe of criteria, and then run it against a paper.

vsoch commented 6 years ago

paper criteria here --> https://joss.readthedocs.io/en/latest/submitting.html

vsoch commented 6 years ago

Just some brainstorming - I'd like the checking functions that plug in here to be specified in a way that (could if we wanted) be piped into a CircleCI (or similar) yaml config.

# The format of this file is based on levels, messages, and functions.
# For example, to say "issue a warning if the submission length is not between
# 250-1000 words." I would have a function identified with a named module,
# and the function itself would return True or False to indicate passing a test.
#
# - Action: determines the actions / formatting of the line to follow.
# - Levels: correspond with python logging levels
# - Functions: should be complete path
# - Description: describes the test
#  
# environment sections define variables used by functions that must be set
# files define files that must exist for functions
# working_directory defines the working directory

version: 1
checks:
  paper:
    exists
    environment:
        OPENBASES_PAPER
    length:
      - name: Check that the paper is between 250-1000 characters
      - level: warning 
      - function: openbases.main.papers.tests.test_length

Still thinking about how to interact / specify with environment variables, files, etc. Likely this will be run on one file (e.g., paper.md).

vsoch commented 6 years ago

okay finally have a good model for this! The usage will look like this:

validate a paper.md based on the default (provided) criteria (fit for general JOSS submission)

ob-validate paper --infile paper.md

validate based on custom criteria

ob-validate paper --infile paper.md --criteria paper.yml

The criteria is very easy to write and read too - should have a working example soon.