search

keatonwilson / paleotidal_app

Shiny app for paleotidal modeling for Exeter + Bangor
1 stars 0 forks source link

Data Pre-Processing - Phase I #10

Closed keatonwilson closed 1 year ago

keatonwilson commented 1 year ago

This PR puts in place some of the basic pre-processing mechanisms for data using the set of lat/lon-mapped data from Sophie. A few key additions:

  1. A new folder (data_preprocessing/) to house one-off pre-processing scripts (no need to be sourced)
  2. A pre-processing script (data_pre_processing/data_pre_process.R) that contains a short example of rasterization and mapping of one of the time-step-datatype combination files, as well as a pre-processing function to get each folder of data combined into a single data frame in long (tidy) format.
  3. Some code to write these rather large (~11-35 million rows each) as feather files for compression and fast reading into the app later on.

Note: to sync/pull this repository, you'll need to install git ifs before you pull. The feather files exceed the file-size-limit imposed by GitHub.

This closes #8

keatonwilson commented 1 year ago

@jessicaguo Can we get this one through soon if you get a sec? It will help on my end for work later this week and into the weekend if we can get main close to updated! Thank you!!!

keatonwilson commented 1 year ago

@jessicaguo I'm going to merge this - I need the pre-processing changes in addition to your first set of UI modifications that are in main right now. Feel free to review post-merge!