search

cct-datascience / setaria-predict

Using ED2 ensembles to train a spatially explicit model
0 stars 0 forks source link

Extract data from Welsch and run locally #13

Closed Aariq closed 1 year ago

Aariq commented 1 year ago

I've re-written the pipeline to start from a single file. I haven't committed that single file yet, because it'll be large and I'm not sure what to do about that. Options

  1. Commit a (diffable) 62.9 MB .csv file
  2. Commit a (non-diffable) 14.6 MB .arrow file
  3. Don't commit the data, but publish it as a pin on Posit Connect
  4. None of the above (e.g. just keep the data locally)

Suggestions?

Aariq commented 1 year ago

BTW, I have no idea why the maps look different when run locally. I'll look into that a bit

Aariq commented 1 year ago

Decided to pin the dataset for now. Can always commit a .csv or other file later.

KristinaRiemer commented 1 year ago

I would suggest committing the smaller file, so that data file is somewhere besides your local machine and Posit Connect?

Aariq commented 1 year ago

I'll try committing a parquet or arrow file in a new PR