search

NASA-PDS / pds4-information-model

The software tools and data necessary for generating the Information Model including PDS4 ontology, data, and information model.
https://nasa-pds.github.io/pds4-information-model/
Other
11 stars 7 forks source link

Tests fail when there are multiple tests with multiple schemas/schematrons #822

Closed jordanpadams closed 1 week ago

jordanpadams commented 2 weeks ago

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

When I try to run back to back tests calling validate, it does not pick up later specified schemas.

🕵️ Expected behavior

I expected the tests to run successfully.

📜 To Reproduce

Playing around with the validate.feature file.

Scenario 1

Scenario 2

Scenario 3

🖥 Environment Info

MacOSx

📚 Version of Software Used

Latest snapshot

🩺 Test Data / Additional context

See repo

🦄 Related requirements

No response

⚙️ Engineering Details

No idea what is happening here. It appears validate is not properly reading/loading the GEOM LDD on the second run.

per comments below, blocked by https://github.com/NASA-PDS/validate/issues/1028

🎉 Integration & Test

No response

al-niessner commented 1 week ago

@jordanpadams

Your expectations are wrong. The first test fails because: SEED 2.4' is not known to this version of validate.

While you may be using a modern information model, validate must fix NASA-PDS/validate#1028 before this test can pass. There is a block that requires loading it independent of check contents that is causing an internal error.

The second test always processes the same. However, there are tons of SEED errors there as well indicating that constant behavior is probably luck not validate good behavior.

The third, behaves different depending on 2 because one of the schematron errors becomes a warning. I do not know if this would change with SEED no longer causing the internal errors.

It is not too surprising that an internal error would leave all kinds of dodo in validate. I would prioritize fixing NASA-PDS/validate#1028 then retesting here. It also means that both validate and this StepDefs should be updated to always fail on an internal error.

jordanpadams commented 1 week ago

@al-niessner copy that. let's go with that path forward. Thanks for investigating.

jordanpadams commented 1 week ago

@al-niessner unfortunately, the removing the SEED internal error noted above did not fix failures. Removing that file still causes github797 tests to fail, when all 3 tests execute successfully when run independently.