search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
558 stars 164 forks source link

Fix regtest results upload clobbering for tests with same name #6323

Open stscijgbot-jp opened 3 years ago

stscijgbot-jp commented 3 years ago

Issue JP-2275 was created on JIRA by James Davies [X]:

Currently when one runs some regression tests and a bunch of them fail, the uploads to artifactory from one module's test clobbers the uploads of the other. This results in only the last test to do the clobbering being able to be okified. Subsequent runs of the regression test suite are needed to then okify the others.

There should be no clobbering.

To reproduce this, run some tests that will clobber each other on master against an old snapshot of our regression test suite (on Artifactory) by setting --env=1.3.1:

$ pytest --bigdata --basetemp=/tmp/scratch/ jwst/regtest/ -k test_spec2 --env=1.3.1 -v --slow
...
jwst/regtest/test_nirspec_ifu_spec2.py::test_spec2[assign_wcs] 
jwst/regtest/test_miri_mrs_spec2.py::test_spec2[assign_wcs] 
jwst/regtest/test_miri_mrs_tso.py::test_spec2[assign_wcs] 
...
FAILED jwst/regtest/test_miri_mrs_tso.py::test_spec2[calints] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_spec2.py::test_spec2[cal] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_spec2.py::test_spec2[flat_field] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_tso.py::test_spec2[flat_field] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_spec2.py::test_spec2[fringe] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_spec2.py::test_spec2[photom] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_tso.py::test_spec2[fringe] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_spec2.py::test_spec2[s3d] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_tso.py::test_spec2[photom] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_spec2.py::test_spec2[straylight] - AssertionError: 
FAILED jwst/regtest/test_miri_mrs_spec2.py::test_spec2[x1d] - AssertionError: 
FAILED jwst/regtest/test_nirspec_ifu_spec2.py::test_spec2[s3d] - AssertionError:

Then dig into the output in /tmp/scratch/:

$ cd /tmp/scratch
$ cat */*/*_results.json | grep target
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",
      "target": "jwst-pipeline-results/2021-09-03_NOT_CI_jdavies_0/test_spec2/",

Since the 3 modules all have tests named test_spec2, then the results files will clobber each other when uploaded.

This will need to be fixed in the artifactory_results_path function

https://github.com/spacetelescope/jwst/blob/d6d14f8260e38115e5106ee95e1b5f75d18fabfc/jwst/regtest/conftest.py#L76-L85

so that it not only keys off the test name, but also the module name when constructing the final subdir.

stscijgbot-jp commented 1 year ago

Comment by Nadia Dencheva on JIRA:

Reassigned this to check if the issue still exists.

stscijgbot-jp commented 5 months ago

Comment by David Law on JIRA:

Tyler Pauly Know if this is still an issue?  Have lumped into long-term technical debt for now.

stscijgbot-jp commented 5 months ago

Comment by Tyler Pauly on JIRA:

I don't know - I would guess that it isn't, but I'll leave it to Zach to get around to testing it and verifying that guess.