Test results - Githubissues

climate-resource / copier-bookshelf-dataset

Template repository for generating bookshelf datasets

0 stars 0 forks source link

Test results #6

Open lewisjared opened 1 day ago

lewisjared commented 1 day ago

The problem

Currently we don't compare the results to the published datasets. There is some existing code in bookshelf that performs this operation.

Definition of "done"

[ ] Integration test for existing data (skipped if no data available)
[ ] Results written to MR (Ask Zeb about his example in input4mips)

Additional context

znichollscr commented 1 day ago

@lewisjared is this what you mean? https://github.com/PCMDI/input4MIPs_CVs/blob/main/.github/workflows/pr-comment-database-changes.yaml

Fire away with any questions

lewisjared commented 1 day ago

Yes. Thanks for the link!

On Wed, 16 Oct 2024, 4:45 pm znichollscr, @.***> wrote:

@lewisjared https://github.com/lewisjared is this what you mean? https://github.com/PCMDI/input4MIPs_CVs/blob/main/.github/workflows/pr-comment-database-changes.yaml

Fire away with any questions

— Reply to this email directly, view it on GitHub https://github.com/climate-resource/copier-bookshelf-dataset/issues/6#issuecomment-2415785841, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKFLQJ377QXPYDER43PUI3Z3X4P5AVCNFSM6AAAAABQASSQBCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJVG44DKOBUGE . You are receiving this because you were mentioned.Message ID: @.***>

lewisjared commented 1 day ago

Following the pattern you outlined @znichollscr, a cli target/script will need to be added to generate HTML which the dataset repos upload as part of the CI. I think that also suggests that the comparison scripts actually live in bookshelf (maybe the producer, maybe another package). This repo would then be responsible for using it as part of the CI, and then creating a comment.

The comment should contain the difference between the dataset generated by the MR and the latest edition.

Pass/Fail description
Accordion of the diff of the dataset configuration
Accordion of a table containing the file hashes by dataset version and timeseries collection (this was the default check for the last version of bookshelf)
The data dictionary calculated from the datasets (useful diagnostic and can be added to the configuration)

lewisjared commented 1 day ago

@znichollscr How did you debug this while building without a million commits/CI cycles

znichollscr commented 1 day ago

How did you debug this while building without a million commits/CI cycles

Run the script locally, copy the output into a GitHub comment, look at the preview, update. Then there were a few cycles as I tried to make it work live, but yes most of the iteration happened via running the script locally