Default disk space is insufficient for longer forecasts

[mkavulich]

The default AWS instance created by wrfcloud-connect has 50 GB of disk space available. While this is sufficient for testing, this will fairly quickly become inadequate once put into real-world use. Based on some test cases I ran with the default 6km domain:

• For the default settings (3hr input, 1hr output), a 48-hour forecast fails at the ungrib step. A 36-hour forecast failed during the WRF forecast step.
• For the default 6km domain, 6hr input, 3hr output, a 72-hour forecast fails during ungrib, and a 60-hour forecast fails during WRF. A 48-hour forecast has only 4 GB remaining on disk after WRF, which is cutting it close, especially once post-processing is added.

On a per-file basis, here are the figures based on the default 6km domain:

• GFS input files: ~520 MB (506-526 from the sample I looked at)
• PFILE (temporary files created by ungrib, removed after completion): 769 MB
• FILE (un-gribbed files used by metgrid): 781 MB
• wrfbdy and wrfinput: 445 MB (total)
• wrfout: ~410 MB (400 - 414 in the sample I looked at; initial time is smaller ~300 MB)

Describe the Enhancement

In order to allow longer forecasts and/or higher-resolution, and/or nested domains, we will need more memory. The crude way to do this would be to make the default disk space much larger, but this seems wasteful. A more elegant solution would be to make a function that uses the above figures prior to spinning up an instance to decide what disk size to allocate.

Another way to achieve some space-savings, though ultimately probably less impactful than the above suggested changes, is to delete temporary files once they are no longer needed. This includes GFS input files and ungrib output (FILE:*) files.

Time Estimate

2 days

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• [ ] Modify workflow to delete files once no longer necessary (GFS input, ungrib output)
• [ ] When instances are spun up via web interface, modify disk space needed based on forecast parameters

Relevant Deadlines

NONE

Define the Metadata

Assignee

• [ ] Select engineer(s) required
• [ ] Select scientist(s) required

Labels

• [ ] Select component(s)
• [ ] Select priority

Projects and Milestone

• [ ] Select Project
• [ ] Select Milestone as the next official version or Backlog of Development Ideas

Enhancement Checklist

• [ ] Complete the issue definition above, including the Time Estimate and Funding Source.
• [ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>/<Description>
• [ ] Complete the development and test your changes.
• [ ] Add/update log messages for easier debugging.
• [ ] Add/update tests.
• [ ] Add/update documentation.
• [ ] Push local changes to GitHub.
• [ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
• [ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s), Project, and Development issue Select: Milestone as the next official version
• [ ] Iterate until the reviewer(s) accept and merge your changes.
• [ ] Delete your fork or branch.
• [ ] Close this issue.

[hahnd]

Disk size can be changed here.

[fossell]

Disk space was increased in recent commits.