Desired Ingest script options

[falkamelung]

To better debug when things don't work

The ingest scripts take very long. Is there an easy way to parallelize them? It would be good to have options to

• Get a listing for all data sets (to make removal of bad ones easier). Should show last date.
• Get info on disk usage etc (rsh commands may work, or just document where to look)
• Get a proper confirmation when a dataset has been removed.
• Get a confirmation that a data set exists

We could just add options to what we have. I am using the alias below which works fine.

alias insarmapsremove='json_mbtiles2insarmaps.py -u insaradmin -p Insar123 --host insarmaps.miami.edu -P rsmastest -U rsmas@gmail.com --remove'

• Regarding “get proper confirmation” both commands give confirmation. To what I know, the second command is work

//login3/home1/05861/tg851601[777] 
insarmapsremove S1_IW1_128_0596_0597_20160605_XXXXXXXX
Removing S1_IW1_128_0596_0597_20160605_XXXXXXXX

insarmapsremove S1_IW1_128_0596_0597_20160605_XXXXXXXX.h5
Removing S1_IW1_128_0596_0597_20160605_XXXXXXXX.h5

• The stderr of the ingest scrips is weird (We are piping it in an out_insarmaps.e. Not everything what goes in there error. There are also strange characters. If we could change so that only real errors show up, and if there is no error file has zero size. (I just discovered that the piped files can bee viewed well using more command - you may want to check how it converts characters)

 0.1%  0/0/0  ^M  0.2%  0/0/0  ^M  0.4%  0/0/0  ^M  0.5%  0/0/0  ^M  0.6%  0/0/0  ^M  0.7%  0/0/0  ^M  0.8%  0/0/0  ^M  0.9%  0/0/0  ^M  1.0%  0/0/0  ^M  1.1%  0/0/0  ^M  1.3%  0/0/0  ^M  1.5%  0/0/0  ^M  1.6%  0/0/0  ^M  1.8%  0/0/0  ^M  2.0%  0/0/0  ^M  2.1%  0/0/0  ^M  2.2%  0/0/0  ^M  2.4%  0/0/0  ^M  2.5%  0/0/0  ^M  2.6%  0/0/0  ^M  2.7%  0/0/0  ^M  2.9%  0/0/0  ^M  3.0%  0/0/0  ^M  3.1%  0/0/0  ^M  3.2%  0/0/0  ^M  3.4%  0/0/0  ^M  3.5%  0/0/0  ^M  3.6%  0/0/0  ^M  3.7%  0/0/0  ^M  3.9%  0/0/0  ^M  4.0%  0/0/0  ^M  4.1%  0/0/0  ^M  4.2%  0/0/0  ^M  4.3%  0/0/0  ^M  4.4%  0/0/0  ^M  4.5%  0/0/0  ^M  4.6%  0/0/0  ^M  4.7%  0/0/0  ^M  4.8%  0/0/0  ^M  4.9%  0/0/0  ^M  5.0%  0/0/0  ^M  5.1%  0/0/0  ^M  5.2%  0/0/0  ^M  5.3%  0/0/0  ^M  5.4%  0/0/0  ^M  5.5%  0/0/0  ^M  5.6%  0/0/0  ^M  5.7%  0/0/0  ^M  5.8%  0/0/0  ^M  5.9%  0/0/0  ^M  6.0%  0/0/0  ^M  6.1%  0/0/0  ^M  6.2%  0/0/0  ^M  6.3%  0/0/0  ^M  6.4%  0/0/0  ^M  6.5%  0/0/0  ^M  6.6%  0/0/0  ^M  6.7%  1/0/0  ^M  6.8%  1/0/0  ^M  6.9%  1/0/0  ^M  7.0%  1/0/1  ^M  7.0%  1/0/0  ^M  7.1%  1/0/1  ^M  7.2%  1/0/1  ^M  7.3%  1/0/1  ^M  7.4%  1/0/1  ^M  7.5%  1/0/1  ^M  7.6%  1/0/1  ^M  7.7%  1/0/1  ^M  7.8%  1/0/1  ^M  7.9%  1/0/1  ^M  8.0%  1/0/1  ^M  8.1%  1/0/1  ^M  8.2%  1/0/1  ^M  8.3%  1/0/1  ^M  8.4%  1/0/1  ^M  8.5%  1/0/1  ^M  8.6%  1/0/1  ^M  8.7%  1/0/1  ^M  8.8%  1/0/1  ^M  8.9%  1/0/1  ^M  9.0%  1/0/1  ^M  9.1%  1/0/1  ^M  9.2%  1/0/0  ^M  9.3%  1/0/0  ^M  9.4%  1/0/0  ^M  9.5%  1/0/0  ^M  9.6%  1/0/0  ^M  9.7%  1/0/0  ^M  9.8%  1/0/0  ^M  9.9%  1/0/0  ^M  13.3%  1/0/1  ^M  13.4%  2/0/2  ^M  13.4%  2/1/2  ^M  13.5%  2/0/2  ^M  13.5%  2/1/2  ^M  13.6%  2/0/2  ^M  13.6%  2/1/2  ^M  13.7%  2/0/2  ^M  13.7%  2/1/2  ^M  13.8%  2/0/2  ^M  13.8%  2/1/2  ^M  13.9%  2/0/2  ^M  13.9%  2/1/2  ^M  14.0%  2/0/2  ^M  14.0%  2/1/2  ^M  14.1%  2/0/2  ^M  14.1%  2/1/2  ^M  14.2%  2/0/2  ^M  14.3%  2/0/2  ^M  14.4%  2/0/2  ^M  14.5%  2/0/2  ^M  14.6%  2/0/2  ^M  14.7%  2/0/2  ^M  14.8%  2/0/2  ^M  14.9%  2/0/2  ^M  16.6%  2/0/1  ^M  16.6%  2/1/1  ^M  18.2%  2/0/1  ^M  18.2%  2/1/1  ^M  18.3%  2/0/1  ^M  19.9%  2/1/1  ^M  20.0%  3/1/3  ^M  20.1%  3/2/4  ^M  20.3%  3/2/4  ^M  20.5%  3/2/4  ^M  20.6%  3/2/4  ^M  20.8%  3/2/4  ^M  21.0%  3/2/4  ^M  21.1%  3/2/4  ^M  21.3%  3/2/4  ^M  21.5%  3/2/4  ^M  21.6%  3/2/4  ^M  23.2%  3/2/3  ^M  22.4%  3/1/4  ^M  22.5%  3/1/3  ^M  23.2%  3/2/3  ^M  23.3%  3/2/3  ^M  24.2%  3/1/3  ^M  24.1%  3/1/4  ^M  24.3%  3/1/3  ^M  24.5%  3/1/3  ^M  24.6%  3/1/3  ^M  24.8%  3/1/3  ^M  26.4%  3/1/4  ^M  26.5%  3/1/4  ^M  26.6%  3/1/4  ^M  26.8%  4/4/7  ^M  27.1%  4/3/7  ^M  27.1%  4/4/8  ^M  27.0%  4/3/8  ^M  27.1%  4/4/8  ^M  27.1%  4/3/7  ^M  27.5%  4/3/7  ^M  27.3%  4/3/8  ^M  27.5%  4/3/7  ^M  27.6%  4/3/7  ^M  27.8%  4/3/7  ^M  28.0%  4/3/7  ^M  28.1%  4/3/7  ^M  28.3%  4/3/7  ^M  28.5%  4/3/7  ^M  28.6%  4/3/7  ^M  28.8%  4/3/7  ^M  29.0%  4/3/7  ^M  29.1%  4/3/7  ^M  29.3%  4/3/7  ^M  29.5%  4/3/7  ^M  29.6%  4/3/7  ^M  29.8%  4/3/7  ^M  31.0%  4/3/8  ^M  31.1%  4/3/8  ^M  31.3%  4/3/8  ^M  31.5%  4/3/8  ^M  31.6%  4/3/8  ^M  31.8%  4/3/8  ^M  32.0%  4/3/8  ^M  32.1%  4/3/8  ^M  32.2%  4/3/8  ^M  32.4%  4/3/8  ^M  32.5%  4/3/8  ^M  32.6%  4/3/8  ^M  32.7%  4/3/8  ^M  32.9%  4/3/8  ^M  33.0%  4/3/8  ^M  33.1%  4/3/8  ^M  33.2%  4/3/8  ^M  33.4%  5/7/16  ^M  33.5%  5/7/16  ^M  33.6%  5/7/16  ^M  33.7%  5/7/16  ^M  33.9%  5/7/16  ^M  34.0%  5/7/16  ^M  34.1%  5/7/16  ^M  34.2%  5/7/16  ^M  34.4%  5/7/16  ^M  34.5%  5/7/16  ^M  34.6%  5/7/16  ^M  34.7%  5/7/16  ^M  34.9%  5/7/16  ^M  35.0%  5/7/16  ^M  35.1%  5/7/16  ^M

[stackTom]

In regards to moving them out of mintpy - I currently have a copy of all the ingest scripts inside of the insarmaps repo/python_scripts. Should I make a separate repo, or just keep them here?

[falkamelung]

I can't clone just python-scripts, can I? If not then, yes, a separate repo would be the easiest to deal with.

[stackTom]

I can't clone just python-scripts, can I? If not then, yes, a separate repo would be the easiest to deal with.

Done, here is the new repo: https://github.com/geodesymiami/insarmaps_scripts/tree/master

[stackTom]

I've added the --list option. please try it.

[stackTom]

I also added more friendly messages to tell you when a remove worked or not, please also try it. I accidentally deleted S1_IW1_128_0596_0597_20160605_XXXXXXXX when testing this, sorry about that.

[stackTom]

Get a confirmation that a data set exists - is this necessary after adding the --list option and the more clear messages when removing? If so, how do I add this. A --check-exists argument where you pass a unavco name and it tells if you if it exists?

[falkamelung]

That works. Thank you @stackTom ! Did you do something on the STDERR? Will check later. Closing for now.

[stackTom]

Did not do anything with stderr. I'll have to investigate more. I think it's tippecanoe which writes to stderr.