search

modflowpy / flopy

A Python package to create, run, and post-process MODFLOW-based models.
https://flopy.readthedocs.io
Other
506 stars 306 forks source link

feat(datafile): add .headers property with data frame #2221

Closed mwtoews closed 2 months ago

mwtoews commented 2 months ago

This feature adds a .headers property for a pandas data frame of the headers of various data files, including HeadFile, FormattedHeadFile, UcnFile and CellBudgetFile. This is a modern accessor to the headers, which is created internally using:

pd.DataFrame(self.recordarray, index=self.iposarray)

where the index is the file position to the start of each array. Text fields are decoded to str types (rather than dealing with bytes types, which is probably a hang-over from Python2). Int32 types are left as-is, except for iposarray which should be int64 to read large files >2GB. Float32 types are also left as-is, since these would otherwise have a lossy conversion to float64.

With the CellBudgetFile, the headers vary depending on the type of budget file. For instance, "classic" files (created without "COMPACT BUDGET" option) don't have imeth, delt, pertim, or totim columns. Furthermore, only files with imeth=6 have the extra text columns modelnam, paknam, modelnam2, and paknam2, since these fields are always empty.

This PR also adds more checks to the outputs created with ._build_index() methods. Eventually, I'm planning to deprecate other properties and functions that are using the .recordarray structured array, so it is important to have a detailed trace of these outputs before replacing them.

This PR also moves the __enter__ / __exit__ methods from BinaryLayerFile to LayerFile (this feature was from #669). This is so FormattedHeadFile can also use the "with" context statement to also auto-close the file.

Examples

E.g. show data frames to markdown with:

python -c "from flopy.utils import HeadFile; print(HeadFile('examples/data/mf6/create_tests/test004_bcfss/expected_output/bcf2ss.hds').headers.to_markdown())
examples/data/mf6/create_tests/test004_bcfss/expected_output/bcf2ss.hds kstp kper pertim totim text ncol nrow ilay
52 1 1 1 1 HEAD 15 10 1
1304 1 1 1 1 HEAD 15 10 2
2556 1 2 1 2 HEAD 15 10 1
3808 1 2 1 2 HEAD 15 10 2
examples/data/mt3d_test/mf2kmt3d/reinject/MT3D001.UCN ntrans kstp kper totim text ncol nrow ilay
44 2 1 1 25 CONCENTRATION 46 31 1
5792 4 1 1 50 CONCENTRATION 46 31 1
11540 6 1 1 75 CONCENTRATION 46 31 1
17288 8 1 1 100 CONCENTRATION 46 31 1
23036 10 1 1 125 CONCENTRATION 46 31 1
28784 12 1 1 150 CONCENTRATION 46 31 1
34532 14 1 1 175 CONCENTRATION 46 31 1
40280 16 1 1 200 CONCENTRATION 46 31 1
46028 18 1 1 225 CONCENTRATION 46 31 1
51776 20 1 1 250 CONCENTRATION 46 31 1
57524 22 1 1 275 CONCENTRATION 46 31 1
63272 24 1 1 300 CONCENTRATION 46 31 1
69020 26 1 1 325 CONCENTRATION 46 31 1
74768 28 1 1 350 CONCENTRATION 46 31 1
80516 30 1 1 365 CONCENTRATION 46 31 1
examples/data/mfusg_test/01A_nestedgrid_nognc/output/flow.cbc kstp kper text ncol nrow nlay
36 1 1 CONSTANT HEAD 121 1 1
556 1 1 FLOW JA FACE 601 1 1
examples/data/mfusg_test/03A_conduit_unconfined/output/ex3A.cbb kstp kper text ncol nrow nlay imeth delt pertim totim
52 160 1 STORAGE 100 100 -2 1 1 160 160
80104 160 1 CONSTANT HEAD 100 100 -2 2 1 160 160
80160 160 1 FLOW RIGHT FACE 100 100 -2 1 1 160 160
160212 160 1 FLOW FRONT FACE 100 100 -2 1 1 160 160
240264 160 1 FLOW LOWER FACE 100 100 -2 1 1 160 160
examples/data/mf6/create_tests/test004_bcfss/expected_output/bcf2ss.cbb kstp kper text ncol nrow nlay imeth delt pertim totim modelnam paknam modelnam2 paknam2
64 1 1 FLOW-JA-FACE 1700 1 -1 1 1 1 1
13792 1 1 WEL 15 10 -2 6 1 1 1 GWF_1 GWF_1 GWF_1 WEL-1
13976 1 1 RIV 15 10 -2 6 1 1 1 GWF_1 GWF_1 GWF_1 RIV-1
14272 1 1 RCH 15 10 -2 6 1 1 1 GWF_1 GWF_1 GWF_1 RCH-1
19176 1 2 FLOW-JA-FACE 1700 1 -1 1 1 1 2
32904 1 2 WEL 15 10 -2 6 1 1 2 GWF_1 GWF_1 GWF_1 WEL-1
33168 1 2 RIV 15 10 -2 6 1 1 2 GWF_1 GWF_1 GWF_1 RIV-1
33464 1 2 RCH 15 10 -2 6 1 1 2 GWF_1 GWF_1 GWF_1 RCH-1
codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 70.4%. Comparing base (d81d7c0) to head (8dc1bc4). Report is 6 commits behind head on develop.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop #2221 +/- ## ========================================= + Coverage 69.8% 70.4% +0.5% ========================================= Files 294 294 Lines 59008 59025 +17 ========================================= + Hits 41228 41578 +350 + Misses 17780 17447 -333 ``` | [Files](https://app.codecov.io/gh/modflowpy/flopy/pull/2221?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=modflowpy) | Coverage Δ | | |---|---|---| | [flopy/utils/binaryfile.py](https://app.codecov.io/gh/modflowpy/flopy/pull/2221?src=pr&el=tree&filepath=flopy%2Futils%2Fbinaryfile.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=modflowpy#diff-ZmxvcHkvdXRpbHMvYmluYXJ5ZmlsZS5weQ==) | `81.8% <100.0%> (+0.3%)` | :arrow_up: | | [flopy/utils/datafile.py](https://app.codecov.io/gh/modflowpy/flopy/pull/2221?src=pr&el=tree&filepath=flopy%2Futils%2Fdatafile.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=modflowpy#diff-ZmxvcHkvdXRpbHMvZGF0YWZpbGUucHk=) | `72.6% <100.0%> (+0.5%)` | :arrow_up: | | [flopy/utils/formattedfile.py](https://app.codecov.io/gh/modflowpy/flopy/pull/2221?src=pr&el=tree&filepath=flopy%2Futils%2Fformattedfile.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=modflowpy#diff-ZmxvcHkvdXRpbHMvZm9ybWF0dGVkZmlsZS5weQ==) | `88.9% <100.0%> (+0.2%)` | :arrow_up: | ... and [36 files with indirect coverage changes](https://app.codecov.io/gh/modflowpy/flopy/pull/2221/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=modflowpy)