Closed dolsysmith closed 3 years ago
Rebased with master (incorporating changes from t128
.
Updated to include sorting of test data prior to testing.
Looks good! Is there an issue describing the need to update models.py and if not, would you create one?
Unit tests now include a separate suite for the Spark loader. It tests the output of columns for both the Elasticsearch index and the CSV extract.
To run them, build the loader container and run
python -m unittest
(within the container). You will see some Spark warnings (safe to ignore these).The non-Spark tests should still run from outside the loader container as expected. (I did not update the non-Spark tests, on the assumption that we may not be using the non-Spark loader very much, but we probably should make the code in
models.py
consistent with the Spark code in a future release. There is some divergence now due to my having upgraded the latter to comply withjson2csv
v. 1.12.1.)