Closed falkamelung closed 2 years ago
- Simplification of
job_submission.py
There will be two bash script calls in eachrun_0*.job
fileinstall_code_on_tmp.bash copy_data_to_tmp.bash
This part is complete. I tested it against both unittestGalapagos and KokoxiliChunk30 successfully. Please test to verify it works as expected, and we can remove the former, commented, code from job_submission.py
@falkamelung Should runfiles generated by create_runfiles.py
without the --tmp
flag include calls to copy_to_tmp.bash
and install_to_tmp.bash
?
I think we need 2 options: —copy_data_to_temp and —install_code_on_tmp. Then it is clear, right?
Sent from my IPhone.
On Aug 13, 2021, at 3:17 PM, Joshua Zahner @.***> wrote:
CAUTION: This email originated from outside the organization. DO NOT CLICK ON LINKS or OPEN ATTACHMENTS unless you know and trust the sender.
@falkamelunghttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffalkamelung&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654747815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lh9koRdfg%2Fdns5IvgiuMbmZyY%2B55SoEFKM8M8q%2Byr1g%3D&reserved=0 Should runfiles generated by create_runfiles.py without the --tmp flag include calls to copy_to_tmp.bash and install_to_tmp.bash?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeodesymiami%2Frsmas_insar%2Fissues%2F496%23issuecomment-898670202&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654747815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QJuN51PGc4T%2FoI6hw2J%2BWgQSDaSg2hxGDuGJxNxZpeY%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACVFHXHD2OIMVGQGVPERL3TT4VVVLANCNFSM5BPSWEEA&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654757773%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lwhKhx83yqS0mjo%2BNYukQDzEeXUQYDps7qRv43DTa5s%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654757773%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jmCM%2BXeqHg6D4cS6ROOtVFtspinL2UCo5TygAQXC4ec%3D&reserved=0 or Androidhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26utm_campaign%3Dnotification-email&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654757773%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FIq7nB%2FyA0i4FVuZp%2F9XR9XUNN2u3tuKSwj5RfSYxYc%3D&reserved=0.
@falkamelung What scripts are generating the configs
and run_files
directories? Is it something we wrote, or is it someone else's code (ISCE)? Is there a way we can redirect those scripts to write to a different directory?
I think we need 2 options: —copy_data_to_temp and —install_code_on_tmp. Then it is clear, right?
Why would we need both? When would you ever want to copy data to tmp but not install the code there or vice-a-versa?
Sorry. I was on travel and will be on jury duty starting tomorrow. Not sure how much time I wil have.
I lost a bit track of where we are. I can imagine that this happen but it is not a big deal.
create_runfiles
calls the ISCE module stackSentinel which creates one runfile for each step and the config files.
create_runfiles
then calls job_submission
which splits the runfile for each step into multiple runfiles and creates a jobfile for each mini-runfile. At the same time it needs to modify the config_tmp files for a step (as currently done in minsarApp.bash)
We want an option in minsarApp.bash to optionally copy data to /tmp:
minsarApp.bash --copy_data_to_tmp
. This requires:
- Simplification of
job_submission.py
There will be two bash script calls in eachrun_0*.job
fileinstall_code_on_tmp.bash copy_data_to_tmp.bash
- Removal of copy_to_tmp section from
minsarApp.bash
- There will be no
sed
commands inminsarApp.bash
create_runfiles.py --copy_data_to_tmp
: This would a) copyrun_files
torun_files_tmp
and replaceconfig
byconfig_tmp
inrun_files_tmp/run_*
b) copyconfigs
toconfigs_tmp
and replace$SCRATCHDIR
bytmp
as in minsarApp.bash c) modify job_submission.py calls to create jobfiles in run_files_tmpjob_submission.py --template /home1/05861/tg851601/code/rsmas_insar/samples/unittestGalapagosSenDT128.template /scratch1/05861/tg851601/unittestGalapagosSenDT128/run_files_TMP/run_08_generate_burst_igram --outdir /scratch1/05861/tg851601/unittestGalapagosSenDT128 --numBursts 2 --writeonly
(job_submission.py assumes for
--outdir
run_files
. This needs to be modified so that it accepts the full path and them userun_files_tmp
).Without
--copy_data_to_tmp
it would userun_files
andconfigs
and nothing would be modified.Note: Actually the default is
copy_data_to_tmp
. So we would switch it off withminsarApp.bash --no_copy_data_to_tmp
We want an option in minsarApp.bash to optionally copy data to /tmp:
minsarApp.bash --copy_data_to_tmp
. This requires:job_submission.py
There will be two bash script calls in eachrun_0*.job
fileminsarApp.bash
sed
commands inminsarApp.bash
create_runfiles.py --copy_data_to_tmp
: This would a) copyrun_files
torun_files_tmp
and replaceconfig
byconfig_tmp
inrun_files_tmp/run_*
b) copyconfigs
toconfigs_tmp
and replace$SCRATCHDIR
bytmp
as in minsarApp.bash c) modify job_submission.py calls to create jobfiles in run_files_tmp(job_submission.py assumes for
--outdir
run_files
. This needs to be modified so that it accepts the full path and them userun_files_tmp
).Without
--copy_data_to_tmp
it would userun_files
andconfigs
and nothing would be modified.Note: Actually the default is
copy_data_to_tmp
. So we would switch it off withminsarApp.bash --no_copy_data_to_tmp