minsarApp.bash --copy_data_to_tmp option

[falkamelung]

We want an option in minsarApp.bash to optionally copy data to /tmp: minsarApp.bash --copy_data_to_tmp. This requires:

1. Simplification of job_submission.py There will be two bash script calls in each run_0*.job file

   install_code_on_tmp.bash
   copy_data_to_tmp.bash 

2. Removal of copy_to_tmp section from minsarApp.bash
   • There will be no sed commands in minsarApp.bash
   • create_runfiles.py --copy_data_to_tmp: This would a) copy run_files to run_files_tmp and replace config by config_tmp in run_files_tmp/run_* b) copy configs to configs_tmp and replace $SCRATCHDIR by tmp as in minsarApp.bash c) modify job_submission.py calls to create jobfiles in run_files_tmp

     job_submission.py --template /home1/05861/tg851601/code/rsmas_insar/samples/unittestGalapagosSenDT128.template /scratch1/05861/tg851601/unittestGalapagosSenDT128/run_files_TMP/run_08_generate_burst_igram --outdir /scratch1/05861/tg851601/unittestGalapagosSenDT128 --numBursts 2 --writeonly

(job_submission.py assumes for --outdir run_files. This needs to be modified so that it accepts the full path and them use run_files_tmp).

Without --copy_data_to_tmp it would use run_files and configs and nothing would be modified.

Note: Actually the default is copy_data_to_tmp. So we would switch it off with minsarApp.bash --no_copy_data_to_tmp

[Ovec8hkin]

1. Simplification of job_submission.py There will be two bash script calls in each run_0*.job file

install_code_on_tmp.bash
copy_data_to_tmp.bash 

This part is complete. I tested it against both unittestGalapagos and KokoxiliChunk30 successfully. Please test to verify it works as expected, and we can remove the former, commented, code from job_submission.py

[Ovec8hkin]

@falkamelung Should runfiles generated by create_runfiles.py without the --tmp flag include calls to copy_to_tmp.bash and install_to_tmp.bash?

[falkamelung]

I think we need 2 options: —copy_data_to_temp and —install_code_on_tmp. Then it is clear, right?

Sent from my IPhone.

On Aug 13, 2021, at 3:17 PM, Joshua Zahner @.***> wrote:

 CAUTION: This email originated from outside the organization. DO NOT CLICK ON LINKS or OPEN ATTACHMENTS unless you know and trust the sender.

@falkamelunghttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffalkamelung&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654747815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lh9koRdfg%2Fdns5IvgiuMbmZyY%2B55SoEFKM8M8q%2Byr1g%3D&reserved=0 Should runfiles generated by create_runfiles.py without the --tmp flag include calls to copy_to_tmp.bash and install_to_tmp.bash?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeodesymiami%2Frsmas_insar%2Fissues%2F496%23issuecomment-898670202&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654747815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QJuN51PGc4T%2FoI6hw2J%2BWgQSDaSg2hxGDuGJxNxZpeY%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACVFHXHD2OIMVGQGVPERL3TT4VVVLANCNFSM5BPSWEEA&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654757773%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lwhKhx83yqS0mjo%2BNYukQDzEeXUQYDps7qRv43DTa5s%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654757773%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jmCM%2BXeqHg6D4cS6ROOtVFtspinL2UCo5TygAQXC4ec%3D&reserved=0 or Androidhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26utm_campaign%3Dnotification-email&data=04%7C01%7Cfamelung%40rsmas.miami.edu%7C644fa259f6d84e97f9d208d95e8f05e5%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637644790654757773%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FIq7nB%2FyA0i4FVuZp%2F9XR9XUNN2u3tuKSwj5RfSYxYc%3D&reserved=0.

[Ovec8hkin]

@falkamelung What scripts are generating the configs and run_files directories? Is it something we wrote, or is it someone else's code (ISCE)? Is there a way we can redirect those scripts to write to a different directory?

[Ovec8hkin]

I think we need 2 options: —copy_data_to_temp and —install_code_on_tmp. Then it is clear, right?

Why would we need both? When would you ever want to copy data to tmp but not install the code there or vice-a-versa?

[falkamelung]

Sorry. I was on travel and will be on jury duty starting tomorrow. Not sure how much time I wil have.

I lost a bit track of where we are. I can imagine that this happen but it is not a big deal.

create_runfiles calls the ISCE module stackSentinel which creates one runfile for each step and the config files. create_runfiles then calls job_submission which splits the runfile for each step into multiple runfiles and creates a jobfile for each mini-runfile. At the same time it needs to modify the config_tmp files for a step (as currently done in minsarApp.bash)

[Tsmpyth32]

We want an option in minsarApp.bash to optionally copy data to /tmp: minsarApp.bash --copy_data_to_tmp. This requires:

1. Simplification of job_submission.py There will be two bash script calls in each run_0*.job file

   install_code_on_tmp.bash
   copy_data_to_tmp.bash 

2. Removal of copy_to_tmp section from minsarApp.bash
   • There will be no sed commands in minsarApp.bash
   • create_runfiles.py --copy_data_to_tmp: This would a) copy run_files to run_files_tmp and replace config by config_tmp in run_files_tmp/run_* b) copy configs to configs_tmp and replace $SCRATCHDIR by tmp as in minsarApp.bash c) modify job_submission.py calls to create jobfiles in run_files_tmp

     job_submission.py --template /home1/05861/tg851601/code/rsmas_insar/samples/unittestGalapagosSenDT128.template /scratch1/05861/tg851601/unittestGalapagosSenDT128/run_files_TMP/run_08_generate_burst_igram --outdir /scratch1/05861/tg851601/unittestGalapagosSenDT128 --numBursts 2 --writeonly

(job_submission.py assumes for --outdir run_files. This needs to be modified so that it accepts the full path and them use run_files_tmp).

Without --copy_data_to_tmp it would use run_files and configs and nothing would be modified.

Note: Actually the default is copy_data_to_tmp. So we would switch it off with minsarApp.bash --no_copy_data_to_tmp