search

NOAA-GFDL / fre-workflows

Code to generate, describe, validate, and configure scientific workflows within the FRE software framework
2 stars 4 forks source link

regrid-xy scripting to use pathlib for filename parsing #32

Open ceblanton opened 2 months ago

ceblanton commented 2 months ago

resolves #31

ceblanton commented 2 months ago

The original problem was Tim noticing these files in my /xtmp directory:

an200:~%>ls /xtmp/$USER  

job37278991work  job37467327work  job37656721work  job37657433work  job37657673work  job37803646work  job37804186work  job37819491work  job38426463work  job38946649work
job37278992work  job37467331work  job37656722work  job37657437work  job37657674work  job37803657work  job37804187work  job37819492work  job38426464work  job38946697work
job37278993work  job37467333work  job37656724work  job37657439work  job37657679work  job37803658work  job37804188work  job37819493work  job38426465work  job38946713work
job37278994work  job37467335work  job37656725work  job37657444work  job37657680work  job37803661work  job37804192work  job37819494work  job38426466work  job38946716work

Instead of just fixing the problem, I thought I'd "fix" other string filename parsing. But it took me a lot longer than it should have and I'm not sure it's greatly improved.

ilaflott commented 2 months ago

did you check the tests in app/regrid-xy/t ?

ilaflott commented 2 months ago

Let me applaud the turn-towards pathlib though

ceblanton commented 2 months ago

You got me- I must have missed something.

With the main branch, 10 out of 12 pass, and with these changes, only 6 out of 12 pass.

ceblanton commented 2 months ago

I take it back- I somehow was using an old branch. 10 / 12 still pass, with this update.

ilaflott commented 2 months ago

OK so in light of fre-cli's add_regrid_xy PR do we want to merge this here now?

we should start a new issue/branch of dev using the new fre-cli functionality

ilaflott commented 2 months ago

I was hoping to be lucky enough to get the pipeline working for reviewing this. but regrid-xy in the state its in here is not pipeline-ready to be assessed this way.

Running the tests within the main branch on PPAN by hand, using a personally-built fre-cli conda environment, I do,

cd app/regrid-xy
pytest -v -v -rx tests/test_regrid_xy.py

As output i see:

platform linux -- Python 3.9.19, pytest-8.3.3, pluggy-1.5.0 -- /home/Ian.Laflotte/conda/envs/fre_cli/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/Ian.Laflotte/Working/fre-workflows
configfile: pytest.ini
plugins: anyio-4.4.0, cov-5.0.0
collected 11 items

t/test_regrid_xy.py::test_make_ncgen3_nc_inputs PASSED                                                                                                                                                                                                                                                                 [  9%]
t/test_regrid_xy.py::test_make_ncgen_tile_nc_inputs PASSED                                                                                                                                                                                                                                                             [ 18%]
t/test_regrid_xy.py::test_make_ncgen_grid_spec_nc_inputs PASSED                                                                                                                                                                                                                                                        [ 27%]
t/test_regrid_xy.py::test_make_hgrid_gold_input PASSED                                                                                                                                                                                                                                                                 [ 36%]
t/test_regrid_xy.py::test_make_fregrid_comparison_input PASSED                                                                                                                                                                                                                                                         [ 45%]
t/test_regrid_xy.py::test_import_regrid_xy PASSED                                                                                                                                                                                                                                                                      [ 54%]
t/test_regrid_xy.py::test_failure_wrong_DT_regrid_xy PASSED                                                                                                                                                                                                                                                            [ 63%]
t/test_regrid_xy.py::test_success_regrid_xy PASSED                                                                                                                                                                                                                                                                     [ 72%]
t/test_regrid_xy.py::test_nccmp1_regrid_xy PASSED                                                                                                                                                                                                                                                                      [ 81%]
t/test_regrid_xy.py::test_nccmp2_regrid_xy PASSED                                                                                                                                                                                                                                                                      [ 90%]
t/test_regrid_xy.py::test_regrid_one_for_two_comps SKIPPED (TODO)                                                                                                                                                                                                                                                      [100%]

Now, I checkout this branch and do the same thing!

cd ../..
git checkout 31.use-pathlib
cd -
pytest -v -v -rx tests/test_regrid_xy.py

and i see:

platform linux -- Python 3.9.19, pytest-8.3.3, pluggy-1.5.0 -- /home/Ian.Laflotte/conda/envs/fre_cli/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/Ian.Laflotte/Working/fre-workflows
configfile: pytest.ini
plugins: anyio-4.4.0, cov-5.0.0
collected 11 items

t/test_regrid_xy.py::test_make_ncgen3_nc_inputs PASSED                                                                                                                                                                                                                                                                 [  9%]
t/test_regrid_xy.py::test_make_ncgen_tile_nc_inputs PASSED                                                                                                                                                                                                                                                             [ 18%]
t/test_regrid_xy.py::test_make_ncgen_grid_spec_nc_inputs PASSED                                                                                                                                                                                                                                                        [ 27%]
t/test_regrid_xy.py::test_make_hgrid_gold_input PASSED                                                                                                                                                                                                                                                                 [ 36%]
t/test_regrid_xy.py::test_make_fregrid_comparison_input PASSED                                                                                                                                                                                                                                                         [ 45%]
t/test_regrid_xy.py::test_import_regrid_xy FAILED                                                                                                                                                                                                                                                                      [ 54%]
t/test_regrid_xy.py::test_failure_wrong_DT_regrid_xy PASSED                                                                                                                                                                                                                                                            [ 63%]
t/test_regrid_xy.py::test_success_regrid_xy FAILED                                                                                                                                                                                                                                                                     [ 72%]
t/test_regrid_xy.py::test_nccmp1_regrid_xy FAILED                                                                                                                                                                                                                                                                      [ 81%]
t/test_regrid_xy.py::test_nccmp2_regrid_xy FAILED                                                                                                                                                                                                                                                                      [ 90%]
t/test_regrid_xy.py::test_regrid_one_for_two_comps SKIPPED (TODO)                                                                                                                                                                                                                                                      [100%]

looking at the output, i see syntax errors. pylint regrid_xy.py will guide you to quick fixes i think!