search

redcap-tools / redcap-test-datasets

Example datasets covering a range of scenarios to facilitate API tests
GNU General Public License v2.0
10 stars 4 forks source link

Required components for each test project #3

Open wibeasley opened 8 years ago

wibeasley commented 8 years ago

To keep a test project portable (eg, it's equally useful to REDCapR and PhpCap), I think it should have a set of standardized qualities/files. Here are my suggestions, as exemplified by the existing/only test project called archer.

@aarenson, @sburns, @nutterb, @haozhu233, and anyone else, please critique or add anything you think would help these goals (or feel free to refine the goals). -Will

Required

These files must exist for every project. They should also have the same name across all projects; only their parent directory distinguishes them.

  1. A readme file that fully specifies the characteristics. This is located in the project's root directory.
  2. A "synthetic" project name, that doesn't reflect the project's characteristics. I think if each project name fully described it (eg "Cross-sectional, Single center, Hybrid, Single Arm"), the convention would
    1. promote names that are unfeasibly long
    2. discourage additional test projects that might overlap the existing canonical "Cross-sectional, Single center, Hybrid, Single Arm" project
  3. An entry and short description into the TOC. This readme file is located in the repository's root directory.
  4. The data dictionary
  5. The metadata csv that's returned by the metadata API call.
  6. The record/data wide csv that's returned by the records API call.
  7. The record/data eav csv that's returned by the records API call.

    Optional

  8. Any images/files contained in a REDCap record. The canonical name is "record-x-variable-name.jpg" (eg, "record-3-mugshot.jpg").
  9. A language-specific (or maybe library-specific) representation of the option (eg, example input for R and example output for R). Because these files require each library developer to create multiple files for each test project, they should be optional.

    Future Problems to Anticipate

  10. What happens if the legal output changes across REDCap versions? How are these variations treated robustly?
    • Example 1: REDCap 6.x adds the field_annotation column in the metadata.
    • Example 2: REDCap 13.13 stops enclosing dates in quotes (eg, 1955-04-15 instead of "1955-04-15").
  11. If one of the test projects breaks in a future version, how can we explicitly (and not silently) exclude it from a library's test suite? I'm wondering if there should a yaml file describing these things.
aarenson commented 8 years ago 
I'm afraid it may be a while before I have a chance to look at this

in depth, but I'll get to it eventually.

    I've not actually created any tests for PHPCap yet, though I have

some ideas in my head.

Andy

On Tue, Dec 08, 2015 at 12:01:40PM -0800, Will Beasley wrote:

To keep a test project portable (eg, it's equally useful to REDCapR and PhpCap), I think it should have a set of standardized qualities/files. Here are my suggestions, as exemplified by the existing/only test project called archer.

@aarenson, @sburns, @nutterb, @haozhu233, and anyone else, please critique or add anything you think would help these goals (or feel free to refine the goals). -Will

Required

These files must exist for every project. They should also have the same name across all projects; only their parent directory distinguishes them.

  1. A readme file that fully specifies the characteristics. This is located in the project's root directory.
  2. A "synthetic" project name, that doesn't reflect the project's characteristics. I think if each project name fully described it (eg "Cross-sectional, Single center, Hybrid, Single Arm"), the convention would
    1. promote names that are unfeasibly long
    2. discourage additional test projects that might overlap the existing canonical "Cross-sectional, Single center, Hybrid, Single Arm" project
  3. An entry and short description into the TOC. This readme file is located in the repository's root directory.
  4. The data dictionary
  5. The metadata csv that's returned by the metadata API call.
  6. The record/data wide csv that's returned by the records API call.
  7. The record/data eav csv that's returned by the records API call.

Optional

  1. Any images/files contained in a REDCap record. The canonical name is "record-x-variable-name.jpg" (eg, "record-3-mugshot.jpg").
  2. A language-specific (or maybe library-specific) representation of the option (eg, example input for R and example output for R). Because these files require each library developer to create multiple files for each test project, they should be optional.

Future Problems to Anticipate

  1. What happens if the legal output changes across REDCap versions? How are these variations treated robustly?
    • Example 1: REDCap 6.x adds the field_annotation column in the metadata.
    • Example 2: REDCap 13.13 stops enclosing dates in quotes (eg, 1955-04-15 instead of "1955-04-15").
  2. If one of the test projects breaks in a future version, how can we explicitly (and not silently) exclude it from a library's test suite? I'm wondering if there should a yaml file describing these things.

Reply to this email directly or view it on GitHub: https://github.com/redcap-tools/redcap-test-datasets/issues/3

Andrew D. Arenson | aarenson (@) iu.edu Advanced Biomedical IT Core, Research Technologies, UITS | W (317) 278-1208 RT is a PTI Cyberinfrastructure & Service Center | C (317) 679-4669 Indiana University Purdue University Indianapolis | F (317) 278-1852

wibeasley commented 8 years ago

Based on @nutterb's comments in #2, I'm realizing the preceding approach doesn't account for different users (that (a) belong to different DAGs, and (b) do & don't have access to PHI). The first five items will be the same for all users (ie, the readme through the metadata csv); but the last two won't (ie, the exported wide & eav csv). This needs to be addressed better. I'd like for a single test project to support multiple users

I assume this means there should be a yaml/csv file with credentials, including user tokens (multiple tokens might be good if the project contains DAGs). It makes sense to include the webserver's URL

This is something else I haven't addressed: should specific REDCap instances should be described in this repo? It may make sense for Andy and me to set up each of these projects in the AWS instance that Vandy established for us to test the API. If other people want to make their test servers available to the public, they could recreate the same projects; this could be useful to make sure functions/scenarios work with older version of REDCap.

I don't know if other instances should be simply additional info the credential file, or they should be described in a separate credential file.

If multiple instances are involved, it sounds like there should be a junction table-like structure that describes which instances house which test projects (for which users/tokens).