search

NOAA-EMC / NCEPLIBS

Top level repo containing submodules for NCEPLIBS and associated dependencies for superproject builds
Other
42 stars 18 forks source link

Check for usage of these subroutines #213

Open edwardhartnett opened 3 years ago

edwardhartnett commented 3 years ago

What would be super useful would be a python script that takes two directory names as arguments, a source and a bin directory.

Then the script should parse all the code files in the source directory and come up with a list of subroutine/function names. Then it should use nm -a on all the binaries in the other directory and give a list of which binaries use which subroutines.

It should also then provide a list of all the unused subroutines.

This script would be very helpful in dealing with NCEPLIBS issues.

kgerheiser commented 3 years ago

I was working on implementation related things earlier this week, so I'll take a stab at some of the things on our list now that I'm done with that.

edwardhartnett commented 3 years ago

Meanwhile I have been looking at some of the functions here for testing. Holy. Freaking. Crap.

There is some hairy math here!

kgerheiser commented 3 years ago

I found some tests that used to be in the repository by going back in the commit history that might be helpful.

https://github.com/NOAA-EMC/NCEPLIBS-sp/tree/b2b21364ab4a289a1b352b055d2eae88fb795e47/tags/WCOSS_sp_generic_1.0.1/utest

https://github.com/NOAA-EMC/NCEPLIBS-sp/tree/77a896f73bfbd954e034ffdfec444b556f22c4ec/branches/utest_v2

kgerheiser commented 3 years ago

See https://github.com/kgerheiser/symbol_checker

I ran it on sp and NEMS.exe (weather model) and the results were:

Symbols provided by './libsp_4.a' that are used by 'NEMS.exe':
    _radb5_  from  fftpack.F.o
    _radf3_  from  fftpack.F.o
    _rfftf1_  from  fftpack.F.o
    _rfftb1_  from  fftpack.F.o
    _rfftb_  from  fftpack.F.o
    _splat_  from  splat.F.o
    _rffti1_  from  fftpack.F.o
    _ludcmp_  from  lapack_gen.F.o
    _lubksb_  from  lapack_gen.F.o
    _srcft_  from  fftpack.F.o
    _scrft_  from  fftpack.F.o
    _radb2_  from  fftpack.F.o
    _radf2_  from  fftpack.F.o
    _drcft_  from  fftpack.F.o
    _scfft_  from  fftpack.F.o
    _radfg_  from  fftpack.F.o
    _radf4_  from  fftpack.F.o
    _radb3_  from  fftpack.F.o
    _dcrft_  from  fftpack.F.o
    _radf5_  from  fftpack.F.o
    _radbg_  from  fftpack.F.o
    _radb4_  from  fftpack.F.o
    _csfft_  from  fftpack.F.o
    _rfftf_  from  fftpack.F.o
    _rffti_  from  fftpack.F.o
edwardhartnett commented 3 years ago

Well that is not a lot! How about the _8 and _d versions?

kgerheiser commented 3 years ago

The library kind does not matter. They all provide the same symbols and rely on the user to link the correct version.

edwardhartnett commented 3 years ago

So does that mean this particular NEMS.exe was built for the _4 libraries?

edwardhartnett commented 3 years ago

BTW in this case there is no need to print out the unused functions/subroutines. We can see at a glance that almost all of them are unused!

kgerheiser commented 3 years ago

No, it was built the default way, which is _d, I think.

I just used _4 because that's what autocompleted first, or something like that. Running it with _d, or _8 wouldn't change the output since the symbol names are the same (in three different libraries).

The executable would link fine with any of the other versions and it wouldn't break until you run it.

edwardhartnett commented 3 years ago

OK, because the functions have the same name, and Fortran does not check the type of arguments when matching a function.

kgerheiser commented 3 years ago

Yes, that would be the compiler's job with module files (and name mangling), but the object files contain no type information.

edwardhartnett commented 3 years ago

OK, also, almost none of the subroutines in NCEPLIBS-sp are actually used. That's kind of a big deal! ;-)

kgerheiser commented 3 years ago

Here is UFS_UTILS:

Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/inland':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/fregrid':
    None
Symbols provided by 'libsp_4.a' that are used by '/Users/KIG/Desktop/xnceplibs-dev/UFS_UTILS/build/install/bin/nemsio_get':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/nst_tf_chg':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/emcsfc_snow2mdl':
    _csfft_  from  fftpack.F.o
    _radb5_  from  fftpack.F.o
    _sptranf_  from  sptranf.f.o
    _sptrung_  from  sptrung.f.o
    _spffte_  from  spffte.f.o
    _sptruns_  from  sptruns.f.o
    _radf2_  from  fftpack.F.o
    _splegend_  from  splegend.f.o
    _sptgpm_  from  sptgpm.f.o
    _radf3_  from  fftpack.F.o
    _sptrunm_  from  sptrunm.f.o
    _ncpus_  from  ncpus.F.o
    _lubksb_  from  lapack_gen.F.o
    _ludcmp_  from  lapack_gen.F.o
    _spwget_  from  spwget.f.o
    _rfftf1_  from  fftpack.F.o
    _radfg_  from  fftpack.F.o
    _sptgpt_  from  sptgpt.f.o
    _sptranf1_  from  sptranf1.f.o
    _radb4_  from  fftpack.F.o
    _radf4_  from  fftpack.F.o
    _sptrun_  from  sptrun.f.o
    _radb2_  from  fftpack.F.o
    _srcft_  from  fftpack.F.o
    _sptranf0_  from  sptranf0.f.o
    _radb3_  from  fftpack.F.o
    _rffti_  from  fftpack.F.o
    _scrft_  from  fftpack.F.o
    _splat_  from  splat.F.o
    _rfftb_  from  fftpack.F.o
    _drcft_  from  fftpack.F.o
    _sptran_  from  sptran.f.o
    _spfftpt_  from  spfftpt.f.o
    _rfftb1_  from  fftpack.F.o
    _dcrft_  from  fftpack.F.o
    _speps_  from  speps.f.o
    _radf5_  from  fftpack.F.o
    _sptgps_  from  sptgps.f.o
    _spanaly_  from  spanaly.f.o
    _scfft_  from  fftpack.F.o
    _radbg_  from  fftpack.F.o
    _rffti1_  from  fftpack.F.o
    _spsynth_  from  spsynth.f.o
    _rfftf_  from  fftpack.F.o
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/global_cycle':
    _lubksb_  from  lapack_gen.F.o
    _ludcmp_  from  lapack_gen.F.o
    _splat_  from  splat.F.o
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/chgres_cube':
    _csfft_  from  fftpack.F.o
    _radb5_  from  fftpack.F.o
    _sptranf_  from  sptranf.f.o
    _spffte_  from  spffte.f.o
    _radf2_  from  fftpack.F.o
    _splegend_  from  splegend.f.o
    _radf3_  from  fftpack.F.o
    _ncpus_  from  ncpus.F.o
    _spdz2uv_  from  spdz2uv.f.o
    _sptezm_  from  sptezm.f.o
    _lubksb_  from  lapack_gen.F.o
    _ludcmp_  from  lapack_gen.F.o
    _spwget_  from  spwget.f.o
    _rfftf1_  from  fftpack.F.o
    _radfg_  from  fftpack.F.o
    _sptranf1_  from  sptranf1.f.o
    _radb4_  from  fftpack.F.o
    _radf4_  from  fftpack.F.o
    _sptezmv_  from  sptezmv.f.o
    _sptez_  from  sptez.f.o
    _radb2_  from  fftpack.F.o
    _srcft_  from  fftpack.F.o
    _sptranf0_  from  sptranf0.f.o
    _radb3_  from  fftpack.F.o
    _rffti_  from  fftpack.F.o
    _scrft_  from  fftpack.F.o
    _splat_  from  splat.F.o
    _rfftb_  from  fftpack.F.o
    _drcft_  from  fftpack.F.o
    _rfftb1_  from  fftpack.F.o
    _dcrft_  from  fftpack.F.o
    _spuv2dz_  from  spuv2dz.f.o
    _speps_  from  speps.f.o
    _radf5_  from  fftpack.F.o
    _spanaly_  from  spanaly.f.o
    _scfft_  from  fftpack.F.o
    _radbg_  from  fftpack.F.o
    _sptranfv_  from  sptranfv.f.o
    _rffti1_  from  fftpack.F.o
    _spsynth_  from  spsynth.f.o
    _rfftf_  from  fftpack.F.o
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/nemsio_chgdate':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/orog':
    _csfft_  from  fftpack.F.o
    _radb5_  from  fftpack.F.o
    _sptranf_  from  sptranf.f.o
    _sptrung_  from  sptrung.f.o
    _spffte_  from  spffte.f.o
    _sptruns_  from  sptruns.f.o
    _radf2_  from  fftpack.F.o
    _splegend_  from  splegend.f.o
    _sptgpm_  from  sptgpm.f.o
    _radf3_  from  fftpack.F.o
    _sptrunm_  from  sptrunm.f.o
    _ncpus_  from  ncpus.F.o
    _lubksb_  from  lapack_gen.F.o
    _ludcmp_  from  lapack_gen.F.o
    _spwget_  from  spwget.f.o
    _rfftf1_  from  fftpack.F.o
    _radfg_  from  fftpack.F.o
    _sptgpt_  from  sptgpt.f.o
    _sptranf1_  from  sptranf1.f.o
    _radb4_  from  fftpack.F.o
    _radf4_  from  fftpack.F.o
    _sptrun_  from  sptrun.f.o
    _sptez_  from  sptez.f.o
    _radb2_  from  fftpack.F.o
    _srcft_  from  fftpack.F.o
    _sptranf0_  from  sptranf0.f.o
    _radb3_  from  fftpack.F.o
    _rffti_  from  fftpack.F.o
    _scrft_  from  fftpack.F.o
    _splat_  from  splat.F.o
    _rfftb_  from  fftpack.F.o
    _drcft_  from  fftpack.F.o
    _sptran_  from  sptran.f.o
    _spfftpt_  from  spfftpt.f.o
    _rfftb1_  from  fftpack.F.o
    _dcrft_  from  fftpack.F.o
    _speps_  from  speps.f.o
    _radf5_  from  fftpack.F.o
    _sptgps_  from  sptgps.f.o
    _spanaly_  from  spanaly.f.o
    _scfft_  from  fftpack.F.o
    _radbg_  from  fftpack.F.o
    _spfft1_  from  spfft1.f.o
    _rffti1_  from  fftpack.F.o
    _spsynth_  from  spsynth.f.o
    _rfftf_  from  fftpack.F.o
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/global_equiv_resol':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/vcoord_gen':
    _lubksb_  from
    _ludcmp_  from
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/shave':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/fvcom_to_FV3':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/regional_esg_grid':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/global_chgres':
    _csfft_  from  fftpack.F.o
    _radb5_  from  fftpack.F.o
    _sptranf_  from  sptranf.f.o
    _sptrung_  from  sptrung.f.o
    _spffte_  from  spffte.f.o
    _sptruns_  from  sptruns.f.o
    _radf2_  from  fftpack.F.o
    _splegend_  from  splegend.f.o
    _sptgpm_  from  sptgpm.f.o
    _radf3_  from  fftpack.F.o
    _sptrunm_  from  sptrunm.f.o
    _sptrunsv_  from  sptrunsv.f.o
    _ncpus_  from  ncpus.F.o
    _sptrunmv_  from  sptrunmv.f.o
    _spdz2uv_  from  spdz2uv.f.o
    _sptezm_  from  sptezm.f.o
    _sptgpmv_  from  sptgpmv.f.o
    _lubksb_  from  lapack_gen.F.o
    _ludcmp_  from  lapack_gen.F.o
    _spwget_  from  spwget.f.o
    _rfftf1_  from  fftpack.F.o
    _radfg_  from  fftpack.F.o
    _sptrunv_  from  sptrunv.f.o
    _sptgpsv_  from  sptgpsv.f.o
    _sptgpt_  from  sptgpt.f.o
    _sptranf1_  from  sptranf1.f.o
    _sptrand_  from  sptrand.f.o
    _radb4_  from  fftpack.F.o
    _radf4_  from  fftpack.F.o
    _sptrun_  from  sptrun.f.o
    _sptezmv_  from  sptezmv.f.o
    _sptez_  from  sptez.f.o
    _radb2_  from  fftpack.F.o
    _sptrungv_  from  sptrungv.f.o
    _sptranfv_  from  sptranfv.f.o
    _srcft_  from  fftpack.F.o
    _sppad_  from  sppad.f.o
    _sptranf0_  from  sptranf0.f.o
    _radb3_  from  fftpack.F.o
    _splaplac_  from  splaplac.f.o
    _rffti_  from  fftpack.F.o
    _scrft_  from  fftpack.F.o
    _splat_  from  splat.F.o
    _rfftb_  from  fftpack.F.o
    _drcft_  from  fftpack.F.o
    _sptran_  from  sptran.f.o
    _spfftpt_  from  spfftpt.f.o
    _rfftb1_  from  fftpack.F.o
    _dcrft_  from  fftpack.F.o
    _spuv2dz_  from  spuv2dz.f.o
    _sptranv_  from  sptranv.f.o
    _speps_  from  speps.f.o
    _radf5_  from  fftpack.F.o
    _sptgps_  from  sptgps.f.o
    _spanaly_  from  spanaly.f.o
    _scfft_  from  fftpack.F.o
    _sptgptv_  from  sptgptv.f.o
    _radbg_  from  fftpack.F.o
    _spfft1_  from  spfft1.f.o
    _rffti1_  from  fftpack.F.o
    _spsynth_  from  spsynth.f.o
    _rfftf_  from  fftpack.F.o
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/mkgfsnemsioctl':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/nemsio_read':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/filter_topo':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/sfc_climo_gen':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/lakefrac':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/make_hgrid':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/make_solo_mosaic':
    None
Symbols provided by 'libsp_4.a' that are used by 'UFS_UTILS/build/install/bin/emcsfc_ice_blend':
    None

And from ip (when comparing directly to another library instead of executable it doesn't include sp's internal calls, to fft for example):

Symbols provided by './libsp_4.a' that are used by './libip_4.a':
    _sptrungv_  from  sptrungv.f.o
    _sptrun_  from  sptrun.f.o
    _sptrunsv_  from  sptrunsv.f.o
    _sptrunmv_  from  sptrunmv.f.o
    _sptrung_  from  sptrung.f.o
    _sptruns_  from  sptruns.f.o
    _splat_  from  splat.F.o
    _sptrunm_  from  sptrunm.f.o
    _sptrunv_  from  sptrunv.f.o
edwardhartnett commented 3 years ago

I guess the next question is: what is a comprehensive set of binaries that we must check?

That is, what is the list of binraries that, once we show a program is not used anywhere, we can remove it?

edwardhartnett commented 3 years ago

directory /nwprod on WCOSS.

AlexanderRichert-NOAA commented 6 months ago

Putting this under NCEPLIBS because I think something like this could be a useful utility for generally tracking usage of NCEPLIBS routines throughout the EMC ecosystem.