Before or while the unpacked files are distributed by download_distribute(), the run was preempted/requeued and so that was not finalized
when the run was restarted, the existence of source_files.log with the correct information made the run think everything is fine
it fails with something like cp: cannot stat 'config/gdx-files/indu_subsectors-buil_simple-tran_edge_esm-POP_pop_SSP5-GDP_gdp_SSP5-En_gdp_SSP5-Kap_debt_limit-Reg_62eff8f7.gdx': No such file or directory
It might be sufficient to write the source_files.log only if the file distribution was successful, not the unpacking alone.
in REMIND, we will fix that with an additional check that if not all input data files are present, input data is downloaded and distributed again
I would leave it up to you, @tscheypidi, whether you still want to fix that in gms, whether you want to add a similar check to MAgPIE (assuming this problem exists there as well) or none of the above and simply close here.
standby
nodesdownload_unpack()
is executed which writessource_files.log
download_distribute()
, the run was preempted/requeued and so that was not finalizedsource_files.log
with the correct information made the run think everything is finecp: cannot stat 'config/gdx-files/indu_subsectors-buil_simple-tran_edge_esm-POP_pop_SSP5-GDP_gdp_SSP5-En_gdp_SSP5-Kap_debt_limit-Reg_62eff8f7.gdx': No such file or directory
source_files.log
only if the file distribution was successful, not the unpacking alone.