search

geodesymiami / rsmas_insar

RSMAS InSAR code
https://rsmas-insar.readthedocs.io/
GNU General Public License v3.0
59 stars 23 forks source link

run_operations.py improvements #78

Closed falkamelung closed 3 years ago

falkamelung commented 5 years ago
falkamelung commented 5 years ago

run_operations_2.0. Regarding the last bullet, after Monday's presentation I now have a clearer idea. The information that is in the GoogleSheet could be in a database. We could have a web-based map and/or Google Sheet to modify it.

Ovec8hkin commented 5 years ago

Yes. A database would probably be ideal, but only after we get everything working appropriately with just the Google Sheets files. A database will also make life easier once we move to AWS.

Ovec8hkin commented 5 years ago

@falkamelung how do you want run_operations to deal with multiple Google Sheets? Should the sheets be specified over the command line, should they be hardcoded by default?

falkamelung commented 5 years ago

Multiple over the command line. We can put in the job file, right?

Sent from my IPhone.

On Mar 22, 2019, at 7:05 PM, Joshua Zahner notifications@github.com<mailto:notifications@github.com> wrote:

@falkamelunghttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffalkamelung&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7C95d823dd5890426bc3b108d6af2343ff%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636888963334004774&sdata=QZ7hMuHPF705LonBgE4RWzETbvSYmWGcafp2YUshgDE%3D&reserved=0 how do you want run_operations to deal with multiple Google Sheets? Should the sheets be specified over the command line, should they be hardcoded by default?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeodesymiami%2Frsmas_isce%2Fissues%2F78%23issuecomment-475820381&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7C95d823dd5890426bc3b108d6af2343ff%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636888963334014783&sdata=82vczktCU8TL4e3hDfF3ugDe%2FnNKq8UWNn%2FhcalE3f8%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKpT3D3nu08yptpYGaXwLk2O325eemboks5vZW_LgaJpZM4bQk8W&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7C95d823dd5890426bc3b108d6af2343ff%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636888963334014783&sdata=9c9in2D40PlMbtwlh7nhllsbnps5vtaWoSVIhz7z1sU%3D&reserved=0.

Ovec8hkin commented 5 years ago

So do you want it to be like this: generate_template_files.py --sheet1 ________ --sheet2 _______ --sheet3 _______ ...

falkamelung commented 5 years ago

Yes. The idea is to have people first trying with their own google sheet file. Once it works we can run them all with the principal run operations

Sent from my IPhone.

On Mar 22, 2019, at 8:17 PM, Joshua Zahner notifications@github.com<mailto:notifications@github.com> wrote:

So do you want it to be like this: generate_template_files.py --sheet1 ____ --sheet2 --sheet3 ...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeodesymiami%2Frsmas_isce%2Fissues%2F78%23issuecomment-475827060&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7C74185fe09f734f90bb6308d6af2d41fd%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636889006248438262&sdata=6gxyJuMLK%2BQyvSpfdADDFYofL8M7aysSud0M1Ovp2ys%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKpT3MUqZtbjaKoC2kSmGQ_MUWICf6yZks5vZYCOgaJpZM4bQk8W&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7C74185fe09f734f90bb6308d6af2d41fd%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636889006248438262&sdata=Ev927c%2FVLDWyE4jKdHfdBai88wPTSkcU80ZA%2Fjbt7%2Fg%3D&reserved=0.

Ovec8hkin commented 5 years ago

I don't understand what you mean. Should run_operations, by default, be run on multiple google sheets files, or is that an edge case that will only be used sometimes? If multiple sheets is the general case we only need the command line options to activate them for use. If they're the exception case, we have to be a little bit smarter about how we handle the template generations.

falkamelung commented 5 years ago

Yes, using multiple google sheet files. To be able to have thematic different sheet files. When a new area is ready we just add this in the run_operations.job file. And then we have the ability to run different areas on different platforms (I am aware that meaningful names like this one below don’t work, but maybe there is a way to do it?)

run_operations.py —sheet_id Ecuador_templates.csv. —sheet_id Hawaii_templates.csv —sheet_id India_landsubsidence_templates.csv —sheet_id EastCoast_landsubsidence.csv

Falk Amelung Professor Department of Marine Geosciences Rosenstiel School of Marine and Atmospheric Sciences University of Miami 4600 Rickenbacker Causeway Miami, FL 33149 USA Tel: 305 421 4949 E-mail: famelung@rsmas.miami.edumailto:famelung@rsmas.miami.edu Web: http://insar.rsmas.miami.edu InSAR data: http://insarmaps.miami.edu

On Mar 22, 2019, at 10:50 PM, Joshua Zahner notifications@github.com<mailto:notifications@github.com> wrote:

I don't understand what you mean. Should run_operations, by default, be run on multiple google sheets files, or is that an edge case that will only be used sometimes? If multiple sheets is the general case we only need the command line options to activate them for use. If they're the exception case, we have to be a little bit smarter about how we handle the template generations.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeodesymiami%2Frsmas_isce%2Fissues%2F78%23issuecomment-475836504&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7Ca77242e21b974bc3655308d6af42ad67%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636889098243099635&sdata=Oph1WlkVgOCyQn0V%2FuSxkGcXed2w%2FQSuv4hr8Hp6kwg%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKpT3I2fRdZ1SOdBCV49siUomuSXcFRyks5vZaR-gaJpZM4bQk8W&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7Ca77242e21b974bc3655308d6af42ad67%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636889098243099635&sdata=FNEht2zr%2BgM10afxOBDSbRi6pqK%2Br%2FNYoSHN4lAMDbo%3D&reserved=0.

Ovec8hkin commented 5 years ago

Ok, thank you for the clarification. This is easy to implement when passing the sheet id from Google Sheets directly. There would be a way to be able to do it using meaningful names, but it would require that, for each new sheet you are using, you would have to update a dictionary in run_operations (or somewhere else), that maps the meaningful name to the sheet id. Something like below:

projects = {
     "Hawaii Templates": 111111111111111,
     "India_landsubsidence_templates": 1234567890987654321
     "EastCoast_landsubsidence": 2222222222222,
     ...
}

Then, when making the command line call you could specify the project name and then run_operations can look up the sheet id from the dictionary and use that to make the request and generate the template.

falkamelung commented 5 years ago

OK. I understand. Lets not bother with this now. If we move to a database instead of GoogleSheet it will be irrelevant. You could put some pseudo code/comment in, though, so that I know where it should go if I want to add it. Falk

Falk Amelung Professor Department of Marine Geosciences Rosenstiel School of Marine and Atmospheric Sciences University of Miami 4600 Rickenbacker Causeway Miami, FL 33149 USA Tel: 305 421 4949 E-mail: famelung@rsmas.miami.edumailto:famelung@rsmas.miami.edu Web: http://insar.rsmas.miami.edu InSAR data: http://insarmaps.miami.edu

On Mar 23, 2019, at 1:50 PM, Joshua Zahner notifications@github.com<mailto:notifications@github.com> wrote:

Ok, thank you for the clarification. This is easy to implement when passing the sheet id from Google Sheets directly. There would be a way to be able to do it using meaningful names, but it would require that, for each new sheet you are using, you would have to update a dictionary in run_operations (or somewhere else), that maps the meaningful name to the sheet id. Something like below:

projects = { "Hawaii Templates": 111111111111111, "India_landsubsidence_templates": 1234567890987654321 "EastCoast_landsubsidence": 2222222222222, ... }

Then, when making the command line call you could specify the project name and then run_operations can look up the sheet id from the dictionary and use that to make the request and generate the template.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgeodesymiami%2Frsmas_isce%2Fissues%2F78%23issuecomment-475899323&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7C66a3f4d21e7b4cf8447808d6afc8d203%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636889674398206352&sdata=bS6ZJbEVV84hnB5iS7iGQNJ5caGdxhMDw0r6bOPEBHc%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKpT3IIYOuGi5HKT_a4kYf5D4zRkPw55ks5vZoWMgaJpZM4bQk8W&data=02%7C01%7Cfamelung%40rsmas.miami.edu%7C66a3f4d21e7b4cf8447808d6afc8d203%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C636889674398206352&sdata=in4tCD1Jz1ECII%2FJkgEo13Db5bnhFYEO0mVZ%2FuiqLsM%3D&reserved=0.