Data extract and prepare refactor

[stevehadd]

Overall aims of this PR

• refactor data extract scripts to be more modular and extensible
• move settings into a JSON file read in by the scripts.
• include the data preprocessing currently in notebooks into these refactored scripts.
• Update scripts to include new variables
• Update preprocessing code to read in settings from the JSON file, including the list of variables.

closes #3 closes #30 closes #31 closes #33 closes #35 closes #17

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[stevehadd]

This is till in progess, so don't merge yet! Things that are still outstanding:

• incorporating preprocessing code into the refactored scripts
• incorporating new variables into the final dataset

[stevehadd]

Things you can look at so far:

• drivers.py The drivers for running extract
• extract_data.py The executable script to run extaction
• event_data_extract.json
• revised notebook for preprocessing. Most of the changes here are for the radar stuff. Hopefully it should have enough description in the notebook to follow.
  • https://github.com/informatics-lab/precip_rediagnosis/blob/data_extract_refactor/data_prep/transform_and_preprocess_spice.ipynb single timestep processing
  • https://github.com/informatics-lab/precip_rediagnosis/blob/data_extract_refactor/data_prep/transform_and_preprocess_spice.ipynb processing all timestepps. Can be harder to follow, hence a separate notebook.

[hannahbrown7]

Line 151 in drivers.py - potentially a typo or end of line missing?

[hannahbrown7]

Tried replacing 4 nested for loops to calculate radar fractions with itertools but unfortunately did not make the function any more efficient. May be some grouping functionality in Xarray that could be helpful to speed this part up.

[hannahbrown7]

The first iteration in the radar fraction calculation appears to have a large radar_cell_in_mg value of 786898, which doesn't seem right, the rest are somewhere in the region of 420, or slightly less (presumable where radar_cell_in_mg is very small or zero, this is where model cells are outside of the radar domain?)

[stevehadd]

Still to do:

• incorporate data prep code (i.e. processing after the extract step) into the scripts created in this pull request
• fix issue listed above
• run processing for 9 and 12 hour lead times for storm dennis
• create separate JSON event description files for each extract, and store them separately
• run processing for the whole of winter 2019/2020 to create a more substantial dataset.
  • improve parallelisation of processing to make this happen in a sensible amount of time. At the very least radar and MOGREPS processing should happen separately.
  • create templating process for event file to process each 5 day period (or some other length )in the winter as an event, to keep data sizes manageable.

[hannahbrown7]

I think there may be different units for model and radar data, MOGREPS-G precip is in m and radar precip is in mm. Could this possibly be added in the preprocessing step? MOGREPS_G precip data just needs to be divided by 0.001

[hannahbrown7]

Not sure if the NaN values are just a problem based on how I'm converting to xarray for plotting the data, but popping this here in case useful. It seems that there are some timestamps each day at 00Z that loose the hour and small number of points that keep it, it would be good to make sure this is consistently including the time as well as date

[stevehadd]

Right, at last I think this is ready for review. (I haven't currently implemented Pete's suggestion above, I'll do that while people are reviewing. Things I'm looking for from this review:

• Does the algorithm look sensible scientifically?
• Is the data sensible (stored in informatics lab project / precip_rediagnosis / ?
• Are there any obvious optimisations I could be making (bearing in mind this is being ported to dagster)
• Data verification - I've created a snmall notebook to check the contents of the dataset, is there anything else I should be checking to confirm its a val;id/correct dataset?

[stevehadd]

Before I find anything else to fix, I'm pulling the trigger and merging. I'll leave any further problems for susequent issues/PRs.