MFC seems to rebuild all targets on `run` even if they're already been built via `build`

[sbryngelson]

MFC rebuilds all targets on ./mfc.sh run even if they've already been built via ./mfc.sh build. Seems to be triggered via a GLOB MISMATCH though I haven't investigated why this occurs. Even occurs if one immediately runs after a build. One option is to avoid build altogether though this is obviously not a very attractive idea.

It even partially rebuilds some dependencies like FFTW/HDF5.

Example snippet:

[I]br011: bryngel-startup-proj/MFC $ ./mfc.sh run examples/1D_sodshocktube/case.py -t pre_process simulation -e batch -N 1 -n 8 -w "00:10:00" -p GPU -# "shb-test" -b mpirun -a phy210041p
mfc: OK > (venv) Entered the Python virtual environment.

      .=++*:          -+*+=.          bryngel@br011.ib.bridges2.psc.edu [Linux]
     :+   -*-        ==   =* .        -----------------------------------------
   :*+      ==      ++    .+-         --jobs 1
  :*##-.....:*+   .#%+++=--+=:::.     --mpi
  -=-++-======#=--**+++==+*++=::-:.   --gpu
 .:++=----------====+*= ==..:%.....   --no-debug
  .:-=++++===--==+=-+=   +.  :=       --targets pre_process and simulation
  +#=::::::::=%=. -+:    =+   *:
-----------------------------------------------------------
 .*=-=*=..    :=+*+:      -...--      $ ./mfc.sh [build, run, test, clean, count, packer]
--help

Run
  Acquiring /ocean/projects/phy210041p/bryngel/MFC/examples/1D_sodshocktube/case.py...
  Configuration:
    Input
/ocean/projects/phy210041p/bryngel/MFC/examples/1D_sodshocktube/case.py
    Job Name      (-#)  shb-test
    Engine        (-e)  batch
    Nodes         (-N)  1
    Tasks (/node) (-n)  8
    Walltime      (-w)  00:10:00
    Partition     (-p)  GPU
    Account       (-a)  phy210041p
    Email         (-@)

  Generating input files for pre_process...

    Generating pre_process.inp.
      INFO: Forwarded 32/50 parameters.

  Generating input files for simulation...

    Generating simulation.inp.
      INFO: Forwarded 32/50 parameters.

  Generating syscheck/include/case.fpp.
    INFO: Custom case.fpp file is up to date.

  $ cmake --build /ocean/projects/phy210041p/bryngel/MFC/build/no-debug_gpu_mpi/syscheck
--target syscheck -j 1 --config Release

[100%] Built target syscheck

  $ cmake --install
/ocean/projects/phy210041p/bryngel/MFC/build/no-debug_gpu_mpi/syscheck

-- Install configuration: "Release"
-- Installing: /jet/home/bryngel/bryngel-startup-proj/MFC/build/install/no-debug_gpu_mpi/bin/syscheck
-- Set runtime path of "/jet/home/bryngel/bryngel-startup-proj/MFC/build/install/no-debug_gpu_mpi/bin/syscheck" to ""
  Generating pre_process/include/case.fpp.
    INFO: Custom case.fpp file is up to date.

  $ cmake --build
/ocean/projects/phy210041p/bryngel/MFC/build/no-debug_gpu_mpi/pre_process --target
pre_process -j 1 --config Release

-- GLOB mismatch!
-- Enabled IPO / LTO
-- Configuring done
-- Generating done
-- Build files have been written to: /jet/home/bryngel/bryngel-startup-proj/MFC/build/no-debug_gpu_mpi/pre_process
[  2%] Preprocessing (Fypp) m_variables_conversion.fpp
[  5%] Preprocessing (Fypp) m_check_patches.fpp
[  8%] Preprocessing (Fypp) m_constants.fpp
[ 11%] Preprocessing (Fypp) m_data_output.fpp
[ 14%] Preprocessing (Fypp) m_derived_types.fpp
[ 17%] Preprocessing (Fypp) m_global_parameters.fpp
[ 20%] Preprocessing (Fypp) m_helper.fpp
[ 23%] Preprocessing (Fypp) m_initial_condition.fpp
[ 26%] Preprocessing (Fypp) m_model.fpp
[ 29%] Preprocessing (Fypp) m_mpi_common.fpp
[ 32%] Preprocessing (Fypp) m_mpi_proxy.fpp
[ 35%] Preprocessing (Fypp) m_patches.fpp
[ 38%] Preprocessing (Fypp) m_start_up.fpp
Scanning dependencies of target pre_process
[ 41%] Building Fortran object CMakeFiles/pre_process.dir/src/pre_process/autogen/m_constants.fpp.f90.o
nvfortran-Warning-CUDA_HOME has been deprecated. Please, use NVHPC_CUDA_HOME instead.
/jet/home/bryngel/bryngel-startup-proj/MFC/src/pre_process/autogen/m_constants.fpp.f90:
[ 44%] Building Fortran object CMakeFiles/pre_process.dir/src/pre_process/autogen/m_derived_types.fpp.f90.o
nvfortran-Warning-CUDA_HOME has been deprecated. Please, use NVHPC_CUDA_HOME instead.
/jet/home/bryngel/bryngel-startup-proj/MFC/src/pre_process/autogen/m_derived_types.fpp.f90:

[...]

[henryleberre]

It would be interesting if you could find a way to reliably reproduce this issue. Here are a few ideas for what could be going wrong:

• (improbable) A source file got written to.
• The system's date/time is imprecise/unreliable.
• Some process changed the last modification date of one or more files.
• Maybe an issue with symbolic links? (multiple paths can point to the same file)

[sbryngelson]

@henryleberre logging into Bridges2 then issuing:

[C]br013: bryngel-startup-proj/MFC $ . ./mfc.sh load -c b -m c

[I]br013: bryngel-startup-proj/MFC $ ./mfc.sh build -t pre_process simulation -j 8
[...builds...]

[I]br013: bryngel-startup-proj/MFC $ ./mfc.sh run ./examples/1D_sodshocktube/case.py -t pre_process -n 1
mfc: OK > (venv) Entered the Python virtual environment.

      .=++*:          -+*+=.          bryngel@br013.ib.bridges2.psc.edu [Linux]
     :+   -*-        ==   =* .        -----------------------------------------
   :*+      ==      ++    .+-         --jobs 1
  :*##-.....:*+   .#%+++=--+=:::.     --mpi
  -=-++-======#=--**+++==+*++=::-:.   --no-gpu
 .:++=----------====+*= ==..:%.....   --no-debug
  .:-=++++===--==+=-+=   +.  :=       --targets pre_process
  +#=::::::::=%=. -+:    =+   *:
-----------------------------------------------------------
 .*=-=*=..    :=+*+:      -...--      $ ./mfc.sh [build, run, test, clean, count, packer]
--help

Run
  Acquiring /ocean/projects/phy210041p/bryngel/MFC/examples/1D_sodshocktube/case.py...
  Configuration:
    Input
/ocean/projects/phy210041p/bryngel/MFC/examples/1D_sodshocktube/case.py
    Job Name      (-#)  MFC
    Engine        (-e)  interactive
    Nodes         (-N)  1
    Tasks (/node) (-n)  1
    MPI Binary    (-b)  srun
  Generating input files for pre_process...

    Generating pre_process.inp.
      INFO: Forwarded 32/50 parameters.

  Generating syscheck/include/case.fpp.
    INFO: Custom case.fpp file is up to date.

  $ cmake --build
/ocean/projects/phy210041p/bryngel/MFC/build/no-debug_no-gpu_mpi/syscheck --target
syscheck -j 1 --config Release

[100%] Built target syscheck

  $ cmake --install
/ocean/projects/phy210041p/bryngel/MFC/build/no-debug_no-gpu_mpi/syscheck

-- Install configuration: "Release"
-- Installing: /jet/home/bryngel/bryngel-startup-proj/MFC/build/install/no-debug_no-gpu_mpi/bin/syscheck
  Generating pre_process/include/case.fpp.
    INFO: Custom case.fpp file is up to date.

  $ cmake --build
/ocean/projects/phy210041p/bryngel/MFC/build/no-debug_no-gpu_mpi/pre_process --target
pre_process -j 1 --config Release

-- GLOB mismatch!
-- Enabled IPO / LTO
-- Configuring done
-- Generating done
-- Build files have been written to: /jet/home/bryngel/bryngel-startup-proj/MFC/build/no-debug_no-gpu_mpi/pre_process
[  2%] Preprocessing (Fypp) m_variables_conversion.fpp
[  5%] Preprocessing (Fypp) m_check_patches.fpp
[  8%] Preprocessing (Fypp) m_constants.fpp
[ 11%] Preprocessing (Fypp) m_data_output.fpp
[...]

[henryleberre]

Thanks, I'll take a look!

[wilfonba]

@henryleberre --case-optimization seems to be hit or miss lately too... My fork is up to date with master, and I keep getting an unsupported value of weno_order error when using case optimization. When I look at the case.fpp file, it's empty. I'm also seeing the rebuild on every run behavior that @sbryngelson is seeing.

[henryleberre]

@wilfonba, I can look into it. Are you looking at the case.fpp in the build folder or src/? It transitioned to the former recently. Since we don’t test for case-optimization right now, it’s possible that someone (including myself) might have broken it.

[wilfonba]

Hmm, it appears to have been updated in the build directory. Maybe I'm missing something else then.