artemis-analytics / artemis

Apache License 2.0
5 stars 2 forks source link

Python data schema refinement #30

Open DominicParent opened 4 years ago

DominicParent commented 4 years ago

The python data schema for demoday needs to be cleaned up, there's still a lot of code from the unittest base file.

ryanmwhitephd commented 4 years ago

@DominicParent can you reference here the code you refer to? Which unittest? I agree, we want an xls schema and a python coded schema.

Tests Write a test which generates a small dataset from each of those schemas using the same seed reading the protobuf stored table.

Validation Comparison of the data generated (maybe a bit overkill) Generate the table schema protobuf from the python implementation and the xls implementation. Store both (serialize and write to a file) then read back in and loop over the fields and compare. This would be a tool for the table schema class for validation.

DominicParent commented 4 years ago

The code is located in examples/demoday/data/demoday_schema.py. The test used as a basis for the schema is tests/test_simutable.py.

All the demoday example work is currently being done in the demoday folder.