update results example to return dataframes from functions, optionally write to file

[ghost]

@WPringle perhaps we can move these functions into the library, and also make a dedicated CLI entry point, maybe combine_results --output_directory ./runs or something

[ghost]

console output of script:

(enembleperturbation)  Zachary.Burnett@hfe07  /scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble
 > python ensemble_results_parser.py

                                   cross_track  along_track  radius_of_maximum_winds  max_sustained_wind_speed
vortex_4_variable_perturbation_1     -0.220716     1.059494                 0.168378                 -1.140257
vortex_4_variable_perturbation_2     -0.447978     0.274827                 0.651191                 -1.614475
vortex_4_variable_perturbation_3     -0.934847     0.191482                 0.030533                 -0.914144
vortex_4_variable_perturbation_4      0.007890     0.059201                 0.909304                 -0.159048
vortex_4_variable_perturbation_5      1.431040    -1.431809                 0.059425                  1.766594
vortex_4_variable_perturbation_6      1.679410    -0.357124                 0.563604                  2.014533
vortex_4_variable_perturbation_7      0.461829     0.620333                 0.682850                 -2.341930
vortex_4_variable_perturbation_8     -0.682090    -2.454760                 0.971933                  0.375695
vortex_4_variable_perturbation_9      0.827994     1.164105                 0.062004                  1.280453
vortex_4_variable_perturbation_10    -2.196805     0.800147                 0.783648                  2.728787
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_9.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/original.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_10.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_7.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_4.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_2.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_5.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_6.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_3.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_1.22/maxele.63.nc
/scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210720_large_ensemble/runs/vortex_4_variable_perturbation_8.22/maxele.63.nc
                x          y  vortex_4_variable_perturbation_8.22  ...  original.22  vortex_4_variable_perturbation_9.22      depth
0      -77.547573  35.839343                                  NaN  ...          NaN                                  NaN  -9.261350
1      -77.551062  35.829987                                  NaN  ...          NaN                                  NaN  -7.806330
2      -77.549422  35.834698                                  NaN  ...          NaN                                  NaN  -9.252210
3      -77.541917  35.835456                                  NaN  ...          NaN                                  NaN -10.394100
4      -77.552008  35.825088                                  NaN  ...          NaN                                  NaN -10.484900
...           ...        ...                                  ...  ...          ...                                  ...        ...
628912 -80.832467  32.044860                             1.256547  ...     1.256547                             1.256547   7.250000
632915 -80.860455  32.052438                             1.266166  ...     1.266166                             1.266166   1.714286
633209 -80.755243  32.025054                             1.232645  ...     1.232645                             1.232645   8.125000
633864 -80.845800  32.080978                             1.262864  ...     1.262864                             1.262864   2.000000
633865 -80.844697  32.086670                             1.262682  ...     1.262682                             1.262682   1.500000

[280928 rows x 14 columns]

[WPringle]

@zacharyburnettNOAA Your suggestion about creating a CLI sounds great to me.

[WPringle]

@zacharyburnettNOAA My other request would be if we can allow for subsetting by a polygon instead/in addition to the bounding box. Can keep min_depth I suppose and make that infinity by default. Then for passing with CLI could use a .txt file or other to define the polygon.

[codecov-commenter]

Codecov Report

Merging #56 (0e9dc1e) into main (ab3575a) will increase coverage by 0.33%. The diff coverage is 59.09%.

@@            Coverage Diff             @@
##             main      #56      +/-   ##
==========================================
+ Coverage   47.72%   48.05%   +0.33%     
==========================================
  Files           8        9       +1     
  Lines        1075     1207     +132     
==========================================
+ Hits          513      580      +67     
- Misses        562      627      +65     

Flag	Coverage Δ
python3.9-Linux	48.05% <59.09%> (+0.33%)	:arrow_up:
python3.9-Windows	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
ensembleperturbation/client/combine_results.py	0.00% <0.00%> (ø)
ensembleperturbation/perturbation/atcf.py	83.47% <50.00%> (-2.91%)	:arrow_down:
ensembleperturbation/parsing/adcirc.py	68.29% <71.66%> (+0.72%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0e4e193...0e9dc1e. Read the comment docs.

[ghost]

@WPringle I've updated the combine_results command to fix a bunch of problems, and made the following output:

(ensembleperturbation)  Zachary.Burnett@hfe07  /scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210816_florence_multivariate_besttrack_250msubset_5members
 > du output.h5 -csh
651M    output.h5

I will also copy this run to Orion in the working/hera folder.

things left to do:

• [ ] handle the variable fields that are not data frames (zeta, u-vel, v-vel, etc.) that exist on a 2D field
• [x] find and handle duplicate records using the same mesh node (I think there might be some because of how pandas.merge() operates)

[WPringle]

@WPringle I've updated the combine_results command to fix a bunch of problems, and made the following output:

(ensembleperturbation)  Zachary.Burnett@hfe07  /scratch2/COASTAL/coastal/save/shared/working/zach/nems_adcirc/run_20210816_florence_multivariate_besttrack_250msubset_5members
 > du output.h5 -csh
651M    output.h5

I will also copy this run to Orion in the working/hera folder.

This is great thank you I will check out that .h5 you produced once on Orion

[ghost]

@WPringle here:

/work/noaa/nosofs/share/working/hera/nems_adcirc/run_20210816_florence_multivariate_besttrack_250msubset_5members

[ghost]

moved TODO to #58