search

JCSDA / CRTMv3

CRTMv3 repository for coordinated development and releases. Code history is not carried in this repository prior to v3, to reduce the cloning overhead. For v2.x history leading up to v3, see JCSDA/crtm repository.
Other
6 stars 6 forks source link

Code updates tracking for v3 merge #21

Closed chengdang closed 1 year ago

chengdang commented 1 year ago

To merge the v2.4.1 updates in to this fresh v3 repository, we cannot use git to track code differences since the commits history are new in this repo.

Assuming the CRTM V3 develop branch was generated mostly based on Mark Liu's initial v3 branch, I started a draft PR (https://github.com/JCSDA-internal/crtm/pull/412) to show the code differences between Mark Liu's branch (with fix folder removed, feature/cd_qliuv3_merge_withoutfix) and the latest v2.4.1 updates before Issac's updates (feature/cd_snow_EM). The code differences are showing in roughly ~170 commits before/after after Mark Liu's branch was created: https://github.com/JCSDA-internal/crtm/pull/412/commits

Besides the aerosol and surface emissivity updates, some updates in these commits have already been included in V3 branch, but some have not.

For example: v2.4.1: https://github.com/JCSDA-internal/crtm/blob/develop/src/CRTM_Tangent_Linear_Module.f90 v3: https://github.com/JCSDA/CRTMv3/blob/develop/src/CRTM_Tangent_Linear_Module.f90 If you search function CRTM_RTSolution_Zero, there are some differences, and this seems to be part of this commit: https://github.com/JCSDA-internal/crtm/pull/412/commits/cfc9f7c89fe56895cf40bb5048c17374d6816b5d

I'm not sure if such code differences are causing the failures in ctests, and wonder if there is a better way to track the code updates with this fresh repo.

StegmannJCSDA commented 1 year ago

The draft PR was a good idea. I have compiled a current list of CRTM Fortran source files that differ between JCSDA-internal/crtm and JCSDA/CRTMv3 in issue #22 .

chengdang commented 1 year ago

Build upon the list Patrick generated in #22, here is an excel file tracking the code differences: https://docs.google.com/spreadsheets/d/1weMoPXehyEhSfj7UyZlIOeZ01TQHG5OK9FqTOZFzkF8/edit#gid=0

chengdang commented 1 year ago

@BenjaminTJohnson @StegmannJCSDA I went through the differences in listed programs.

I grouped most of the codes into four incoming PRs, which are unlikely the causes of the current ctest failures (to my opinion).

I did not check the code differences under TauProd/TauRegress/TauCoeff folders (assuming they are unlikely the causes?)

I marked some major code differences: Forward, TL, adjoint modules, as well as the change of RT solvers with polarization. It is unclear to me if they are causing the ctest failures.

StegmannJCSDA commented 1 year ago

@chengdang thank you for compiling the sheet. I will check the TauProd/TauCoeff related source. I don't expect these to affect the runtime behaviour, but it's good to check for inconsistencies.

chengdang commented 1 year ago

Besides the src code, the ctest codes also need updates perhaps.

There are noticeable differences between the test source code, for example: https://github.com/JCSDA-internal/crtm/blob/develop/test/CMakeLists.txt https://github.com/JCSDA/CRTMv3/blob/develop/test/CMakeLists.txt

chengdang commented 1 year ago

List of modified test files from v2.4.1 to v3:

Modified or deleted:
    modified:   test/CMakeLists.txt
    modified:   test/crtm_data_downloader.py
    modified:   test/mains/application/check_crtm.F90
    modified:   test/mains/application/check_crtm.fpp
    modified:   test/mains/regression/k_matrix/test_Simple/test_Simple.f90
    modified:   test/mains/unit/Unit_Test/Load_Atm_Data.inc
    modified:   test/mains/unit/Unit_Test/test_AD.f90
    deleted:    test/mains/unit/input_output/test_AerosolCoeff/data/AerosolCoeff.bin
    modified:   test/mains/unit/input_output/test_AerosolCoeff/test_aerosol_coeff_io.f90
    deleted:    test/mains/unit/input_output/test_AerosolCoeff_NC/data/AerosolCoeff.nc4
    modified:   test/mains/unit/input_output/test_AerosolCoeff_NC/test_aerosol_coeff_io_nc.f90
    deleted:    test/mains/unit/input_output/test_SpcCoeff_NC/data/amsua_aqua.SpcCoeff.nc
    deleted:    test/mains/unit/input_output/test_TauCoeff_NC/data/amsua_aqua.TauCoeff.nc
    modified:   test/readme_crtm_tests.txt

New files:
    test/mains/regression/forward/test_OMPoverChannels/
    test/mains/regression/k_matrix/test_AOD/tx.sh
    test/mains/unit/Unit_Test/test_AD_Active_Sensor.f90
    test/mains/unit/Unit_Test/test_AD_Active_Sensor_test.f90
    test/mains/unit/Unit_Test/test_TL_convergence_active_sensor.f90
    test/mains/unit/input_output/test_EmisCoeff/
    test/mains/unit/input_output/test_EmisCoeff_NC/
StegmannJCSDA commented 1 year ago

My first impulse would be to update all the tests to the 2.4.1 versions.

chengdang commented 1 year ago

My first impulse would be to update all the tests to the 2.4.1 versions.

Yes me too! I tried that and there are some changes need to be done. I added the summary in the same excel file shared earlier.

The CMakeLists.txt also contains some changes of sensor id, should we just go ahead and use the v2.4.1 version?

StegmannJCSDA commented 1 year ago

Alright, in preparation for the meeting later, this is a list of currently failing tests (actually almost all of them) for the standalone build:

The following tests FAILED:
      1 - get_crtm_coeffs (Failed)
      4 - Unit_TL_TEST (Failed)
      5 - test_forward_Simple_atms_n21 (Failed)
      6 - test_forward_Simple_cris-fsr_n21 (Failed)
      7 - test_forward_Simple_v.abi_g18 (Failed)
      8 - test_forward_Simple_modis_aqua (Failed)
      9 - test_forward_AOD_cris-fsr_n21 (Failed)
     10 - test_forward_AOD_v.abi_g18 (Failed)
     11 - test_forward_AOD_airs_aqua (Failed)
     12 - test_forward_Zeeman_ssmis_f20 (Failed)
     13 - test_forward_Zeeman_ssmis_f19 (Failed)
     14 - test_forward_Zeeman_ssmis_f18 (Failed)
     15 - test_forward_Zeeman_ssmis_f17 (Failed)
     16 - test_forward_Zeeman_ssmis_f16 (Failed)
     17 - test_forward_ChannelSubset_iasi_metop-b (Failed)
     18 - test_forward_ClearSky_atms_n21 (Failed)
     19 - test_forward_ClearSky_cris-fsr_n21 (Failed)
     20 - test_forward_ClearSky_v.abi_g18 (Failed)
     21 - test_forward_ClearSky_modis_aqua (Failed)
     22 - test_forward_Aircraft_cris-fsr_n21 (Failed)
     23 - test_forward_ScatteringSwitch_atms_n21 (Failed)
     24 - test_forward_ScatteringSwitch_cris-fsr_n21 (Failed)
     25 - test_forward_ScatteringSwitch_v.abi_g18 (Failed)
     26 - test_forward_ScatteringSwitch_modis_aqua (Failed)
     27 - test_forward_SOI_atms_n21 (Failed)
     28 - test_forward_SOI_cris-fsr_n21 (Failed)
     29 - test_forward_SOI_v.abi_g18 (Failed)
     30 - test_forward_SOI_modis_aqua (Failed)
     31 - test_forward_SSU_ssu_n06 (Failed)
     32 - test_forward_SSU_ssu_n14 (Failed)
     33 - test_forward_VerticalCoordinates_atms_n21 (Failed)
     34 - test_forward_VerticalCoordinates_cris-fsr_n21 (Failed)
     35 - test_forward_VerticalCoordinates_v.abi_g18 (Failed)
     36 - test_forward_VerticalCoordinates_modis_aqua (Failed)
     37 - test_k_matrix_Simple_atms_n21 (Failed)
     38 - test_k_matrix_Simple_cris-fsr_n21 (Failed)
     39 - test_k_matrix_Simple_v.abi_g18 (Failed)
     40 - test_k_matrix_Simple_modis_aqua (Failed)
     41 - test_k_matrix_AOD_cris-fsr_n21 (Failed)
     42 - test_k_matrix_AOD_v.abi_g18 (Failed)
     43 - test_k_matrix_AOD_airs_aqua (Failed)
     44 - test_k_matrix_Zeeman_ssmis_f20 (Failed)
     45 - test_k_matrix_Zeeman_ssmis_f19 (Failed)
     46 - test_k_matrix_Zeeman_ssmis_f18 (Failed)
     47 - test_k_matrix_Zeeman_ssmis_f17 (Failed)
     48 - test_k_matrix_Zeeman_ssmis_f16 (Failed)
     49 - test_k_matrix_ChannelSubset_iasi_metop-b (Failed)
     50 - test_k_matrix_ClearSky_atms_n21 (Failed)
     51 - test_k_matrix_ClearSky_cris-fsr_n21 (Failed)
     52 - test_k_matrix_ClearSky_v.abi_g18 (Failed)
     53 - test_k_matrix_ClearSky_modis_aqua (Failed)
     54 - test_k_matrix_ScatteringSwitch_atms_n21 (Failed)
     55 - test_k_matrix_ScatteringSwitch_cris-fsr_n21 (Failed)
     56 - test_k_matrix_ScatteringSwitch_v.abi_g18 (Failed)
     57 - test_k_matrix_ScatteringSwitch_modis_aqua (Failed)
     58 - test_k_matrix_SOI_atms_n21 (Failed)
     59 - test_k_matrix_SOI_cris-fsr_n21 (Failed)
     60 - test_k_matrix_SOI_v.abi_g18 (Failed)
     61 - test_k_matrix_SOI_modis_aqua (Failed)
     62 - test_k_matrix_SSU_ssu_n06 (Failed)
     63 - test_k_matrix_SSU_ssu_n14 (Failed)
     64 - test_k_matrix_VerticalCoordinates_atms_n21 (Failed)
     65 - test_k_matrix_VerticalCoordinates_cris-fsr_n21 (Failed)
     66 - test_k_matrix_VerticalCoordinates_v.abi_g18 (Failed)
     67 - test_k_matrix_VerticalCoordinates_modis_aqua (Failed)
     68 - test_adjoint_Simple_atms_n21 (Failed)
     69 - test_adjoint_Simple_cris-fsr_n21 (Failed)
     70 - test_adjoint_Simple_v.abi_g18 (Failed)
     71 - test_adjoint_Simple_modis_aqua (Failed)
     72 - test_adjoint_ClearSky_atms_n21 (Failed)
     73 - test_adjoint_ClearSky_cris-fsr_n21 (Failed)
     74 - test_adjoint_ClearSky_v.abi_g18 (Failed)
     75 - test_adjoint_ClearSky_modis_aqua (Failed)
     76 - test_tangent_linear_Simple_atms_n21 (Failed)
     77 - test_tangent_linear_Simple_cris-fsr_n21 (Failed)
     78 - test_tangent_linear_Simple_v.abi_g18 (Failed)
     79 - test_tangent_linear_Simple_modis_aqua (Failed)
     80 - test_tangent_linear_ClearSky_atms_n21 (Failed)
     81 - test_tangent_linear_ClearSky_cris-fsr_n21 (Failed)
     82 - test_tangent_linear_ClearSky_v.abi_g18 (Failed)
     83 - test_tangent_linear_ClearSky_modis_aqua (Failed)