search

SECOORA / GUTILS

🌊 🚤 Python utilities for reading, merging, and post processing Teledyne Webb Slocum Glider data
MIT License
7 stars 11 forks source link

Accept converted ASCII files #2

Closed kwilcox closed 7 years ago

kwilcox commented 7 years ago

from https://github.com/kerfoot/slocum

kerfoot commented 7 years ago

@kwilcox I cleaned up the code in https://github.com/kerfoot/slocum, added documentation and created a new repo:

https://github.com/kerfoot/slocum-utils

This repo is the one that should be used. The other repo contains a version of the script that we use internally here and has some specific pieces that are used solely for our internal use.

Please clone the repo (tag=v1.0-stable), check the wiki doco and test on a few files and let me know if everything works as expected.

kwilcox commented 7 years ago

So I will need to copy new flight/science files into a "not processed" folder, run the convert script, and then remove the files from the "not processed" directory so they don't get parsed again on the next run, correct? Is that a typical workflow?

kerfoot commented 7 years ago

Yes, that's typical. The issue that you may run into, however, during real-time ops is that the flight data file may be transferred, but the science data file may not (if iridium comms break down before both files are transferred). In that case, the script would convert only the flight file and create an ascii file missing science data.

I could add an additional command line option (-s or 'strict mode') that would only convert/merge segments with both a flight and science file an delete them if they are successfully merged. In this case, you would want to copy the original flight and science data files to a queue location and convert/merge the copies. I'll probably add this option anyways, but let me know if you want/need it and I'll do it first thing tomorrow.

kwilcox commented 7 years ago

The ability pipe the output of convertDbds.sh somewhere would be real nice and cover lots of use-cases. The output could be the file paths of all files that it used in a conversion to ASCII (something that wouldn't need to be used again). For example: piping the output to xargs/rm, xargs/mv, etc.

The strict flag would also be nice but I haven't run into that situation before. I always re-create the files at the end of a deployment to fix little issues like this so its not important at the moment.

kerfoot commented 7 years ago
  1. So you mean print the absolute paths of all files that were successfully converted and merged to STDOUT?
  2. You may not have run into this situation before, but you will. Iridium comms are notoriously bad and we've had many many cases of this happening. Just warning you now.

The script currently prints conversion/merging progress to STDOUT, but there is a -q option that prevents this. I'll add a -x option to print the list of all successfully converted/merged files, but you'll still need to specify -q or you'll have to parse this information out. Will this work?

kwilcox commented 7 years ago

That will work: -q -x.

kerfoot commented 7 years ago

Added -p option to print the paths of successfully converted & merged binary files to STDOUT. Tag is v1.1.