search

JGCRI / gcamdata

The GCAM data system
https://jgcri.github.io/gcamdata/
Other
42 stars 26 forks source link

Level2 outputs need more testing #321

Closed bpbond closed 6 years ago

bpbond commented 7 years ago

Level 2 outputs are, or should be, tagged with the relevant IDstring (e.g. "AgResBio") and batch_XML_domain (e.g. AGLU_XML_BATCH). These should be tested against the original data system settings?

See comments by @pkyle in #317

The second part is that the testing currently isn't picking up the meta-info in the level2 tables; while the L200* modeltime files (the only level2 files committed to the master branch) are currently passing the checks, a manual comparison (e.g., tests/testthat/comparison_data/modeltime/L200.hector.csv versus outputs/L200.hector.csv) will show the differences (one has the four rows of meta-info and the other doesn't).

pkyle commented 7 years ago

Not sure if this helps, but in most cases, the csv-to-xml IDstring is equal to the characters between the first "." and the first "_" of the name of the table being written out. The exceptions are the headers for logits, the building sector (energy/L244.*), and the land allocator, where the headers often have their own uses of "_". Also in the emissions module, the IDstring is often not derived from the table name.

rplzzz commented 7 years ago

Isn't the best guide to look for the IDString= and batch_XML_domain= arguments to the write_mi_data call in the original data system? If you just copy those, then you won't go wrong.

However, it's still not clear how these tags are meant to be applied in the new system. Is there a function to add these, along the lines of add_flags, add_comment, etc?

pkyle commented 7 years ago

Yes that would work and is probably what we'll have to do--I just made that suggestion in case we wanted to have a default IDString= set by a function that could be over-ridden (as needed) by a user. That wouldn't help with the batch_XML_domain= argument. Note that the name of the XML file, formerly set in the insert_file_into_batchxml() function, is always just the name of the batch XML file minus the string "batch_"

rplzzz commented 7 years ago

Understood, but most of the people developing these modules aren't going to know the name of the batch XML file, so it's good to be able to just tell them where to look.

@bpbond can you weigh in on how we add these tags to the tables?

bpbond commented 7 years ago

The tags will be automatically generated, pulled directly from the old data system. More info on this coming soon.

bpbond commented 6 years ago

Addressed in #694 .