search

ai2cm / fv3net

explore the FV3 data for parameterization
MIT License
16 stars 3 forks source link

Switch all cloud IO operations to fsspec #128

Closed frodre closed 4 years ago

frodre commented 4 years ago

We've been considering cleaning up the IO operations for a while. (As referenced in #88) Since we are consolidating the overall workflows, it's probably time to address this. I believe we've centered around using fsspec to accomplish most of cloud file interaction needs. Let's discuss the steps necessary to accomplish this!

Proposal for switching to fsspec.

  1. Finish the coarsening workflow #120
  2. Finish PR for dataflow regression tests #126 (basic simple dataflow and coarsening tests)
  3. Prepare a new PR that deprecates our old GCS interaction methods and introduces the common fsspec operationsinto vcm with documentation & tests
  4. Switch over individual workflows with regression tests to use the new fsspec operations
nbren12 commented 4 years ago

I would agree with this and go one further. We should use fsspec for all local file interactions as well.

oliverwm1 commented 4 years ago

I think this is covered by #88. Also, our stance has changed somewhat on this. We are okay with using gsutil in places since it is robust and multi-threads well.