ams-goonies / usda-hackathon-2024

R Shiny app for visualizing 2022 Ag Census data
GNU General Public License v3.0
0 stars 0 forks source link

Create sharepoint folder where datasets can be stored #12

Closed elhohn closed 3 months ago

elhohn commented 3 months ago

@aishwarya-sreenivasan @shauncarl @stel-ls @selazan We can discuss this in our next check-in...

We're going to have a fairly large amount of data from these tasks, and we don't know exactly how we are going to manage it for the app yet, so I think we should put these into a shared folder that we can all access rather than pushing them to GitHub because we'll break the repo.

Workflow should be something like:

  1. You create a new dataset through one of your tasks
  2. You export that dataset from R to your computer
  3. You drag that dataset into the new shared folder
  4. You notify the rest of the team
  5. The rest of the team goes to the new shared folder, finds the new dataset, and drags it into their local repo
  6. None of us commit datasets to GitHub
elhohn commented 3 months ago

Also, my current thinking is that once the data is finalized we put it all in a SQLite database that gets deployed with the app, which would be fast and light without requiring a database hosted on a separate remote server. Depending on complexity, an alternative could be to standardize everything, save it all together as an .RData file, and have the app load that at startup.

elhohn commented 3 months ago

Folder is located here: https://usdagcc.sharepoint.com/:f:/r/sites/ams/AMS-TMIntranet/AMS-GD/Data%20Room/Division%20Wide/Hackathon%202024?csf=1&web=1&e=XNpzfe

There are two folders, "raw" and "ready_for_app", with README files in each explaining their use.