Issue 41 add automated tests for cleaning file for all years

[alexkcode]

Description

This adds unit tests for Python script that cleans source data that contains building data for all years. The actual transformations tested should be explained by the title of each test.

Fixes # 41

Testing Instructions

Run python3 -m pytest test/data/scripts/unit/test_clean_all_years.py from base project directory. All tests should pass as indicated by console pytest output.

Checklist:

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes

[netlify[bot]]

✅ Deploy Preview for radiant-cucurucho-d09bae ready!

Name	Link
🔨 Latest commit	031e4eaf5c76d087f1fa5492e72b2b32a34e2b5b
🔍 Latest deploy log	https://app.netlify.com/sites/radiant-cucurucho-d09bae/deploys/65f0ff889f257f0008af28eb
😎 Deploy Preview	https://deploy-preview-68--radiant-cucurucho-d09bae.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[alexkcode]

Current test cases only include buildings from 'United Center', 'Crown Hall', 'Art Institute'

[alexkcode]

Current runtime is at around 21 seconds

[vkoves]

Issue: It looks like the actual data process now doesn't run, maybe due to the new imports?

[alexkcode]

@vkoves fixed all the pathing issues!

[vkoves]

QA list for myself:

• [ ] Run tests locally
• [ ] Run data process, confirm output CSV doesn't change
• [ ] Re-generate test data

[alexkcode]

Currently producing a slightly different output csv's than original script. Not sure why since the types and numbers are the same, but I'm still investigating.

[vkoves]

@alexkcode - looks like there's an issue creating the debug JSON, and some numeric columns become repeating numbers, it looks like it's the latitude and longitude column:

It's easiest to debug this by looking at the line by line diff, and then your editor will highlight the specific part of each line that changed.

[alexkcode]

After I changed the string type conversion in the script some of the columns like ChicagoEnergyRating are now empty spaces, instead of Pandas type nan, in the produced .csv files. I think this is actually the correct behavior, but please let me know if it is not.

[alexkcode]

For specific string/column formats, we should check against the original source raw CSV (or as close as possible to it). I will change the Latitude, Longitude column rests and the other tests that test against the original values for the different direct type conversions that take place in the scripts.

[alexkcode]

Rendered websites (production vs. local) appear the same, i.e. the repeating decimals don't show up, unless my eyes missed something.

[alexkcode]

Could potentially be Pandas issue (from an update or something), will look into this.

[alexkcode]

Found decimal issue origin: before the conversion to string type in the Python scripts an interpretation of the data types is already by the get_and_clean_csv function.

[alexkcode]

@vkoves Please re-review!

[vkoves]

@alexkcode - I re-ran the data processing to make sure everything is up to date, but the test now fails with this one error:

I think that is related to the issue you filed, so I'll still merge this, since this isn't running in CI, but I'd love your help to fix that. I wonder if we could fix this just by rounding these floats to a certain relevant precision.