search

svalinn / r2s-act

Rigorous 2 Step Activation Workflow
4 stars 7 forks source link

Unit tests' comparison file generation #26

Open erelson opened 11 years ago

erelson commented 11 years ago

Some of the tests (e.g. those for write_gammas.py) use input files, generate output, and compare that output with saved expected output. In my case I have a script that regenerates these 'reference files' as needed (in this case, I added the headers to the gammas file, and need to update the test files). The files/scripts for *regenerating the reference files live locally in my cae-cnerg space.

My current working idea is to put these inputs/scripts/etc in a separate branch on Github with suitable readme instructions.

Similar to documentation with gh-pages, a script could copy stuff between branches. However, it might be better not to do this - to make copying the updated reference files a manual process. This would put conscious thought into the process, which I think is particularly important when updating the reference files.

Any objections/other ideas?

*: Updating the reference files is determined to be safe because separate manual testing indicates that modifications to source_gammas.F90 correctly handles the header in gammas.

Edited to clarify confusing statement noted by @elliottbiondo

elliottbiondo commented 11 years ago

Why do your reference files live locally? Why not commit them to /testing?

erelson commented 11 years ago

Because that's where they were created, and they have never been tidied up. Not the wisest set of past decisions, of course.

What I left unsaid in the above is that I'm not sure dumping all of such files into another subfolder of scripts/testing/ vibes with keeping a tidy repository... thus the thought of a separate branch. I suppose this could be abusing the purpose of branches, though.

elliottbiondo commented 11 years ago

I think adding another branch and having readme instructions is probably messier, and also passes burden onto the user. How many files are we talking? Maybe you could make a quick readme explaining some of the files if it is really confusing. I don't think it matters if the /testing folder is a bit messy because all the user needs to be able to do is get there and run nosetests.

erelson commented 11 years ago

For reference, the folder of files in question is /filespace/groups/cnerg/users/relson/test_write_gammas. Since the files originate from a full R2S calculations, there's quite a few files; . The MCNP output files will be ignored thanks to the .gitignore file, at least...

I'm not sure whether the user needs the files, let alone needs to deal with them. By using a separate branch (I agree this does add some messiness) the user still has the files available if truly needed, but it's not a burden for a user, though it is for the developer.

This is all crude hypotheticals as I don't know too well what to expect of the user's needs in this case.

elliottbiondo commented 11 years ago

Oh I see... I just went to relson/test_write_gammas. I didn't realize it was a full R2S calculation.

To be clear:

The files/scripts for the reference files live locally in my cae-cnerg space.

You don't mean that the reference files live locally correct? As long as the reference file live in scripts/testing/ then I don't see why anyone would ever need to see the scripts used to generate the reference files. I think it is acceptable for the users to take it on faith that the developers know what the correct output should be. Or perhaps I am missing something. We can talk tomorrow.

erelson commented 11 years ago

Ahh, I did indeed make that less clear than I meant to. Should be: The files/scripts for generating the reference files live locally in my cae-cnerg space.

gonuke commented 11 years ago

We should probably meet to discuss this so that we are all clear on which files are which. In general, all of the files needed for testing should be in the repository, but I'm no longer clear that's what we are talking about.

elliottbiondo commented 11 years ago

@erelson:

Another thought: We need to make tests for r2s_step1 and r2s_step2. Perhaps the input files that need to be present for this test can double as the generation files for your reference files. That way we only maintain 1 set of R2S inputs in the repo.

erelson commented 11 years ago

That's certainly possible, just as long as the reference files aren't accidentally regenerated when testing other scripts (Aka just make sure we do this tidily).