aodn / imos-toolbox

Graphical tool for QC'ing and NetCDF'ing oceanographic datasets
GNU General Public License v3.0
46 stars 31 forks source link

Toolbox doesn't read RDI files split across multiple files #627

Closed BecCowley closed 4 years ago

BecCowley commented 4 years ago

I have multiple files from a single deployment of an RDI instrument: 1435001.000 1435000.000

Each file contains part of the deployment. Is there a way to make the toolbox read both files and concatenate them?

sspagnol commented 4 years ago

Personally would be wary of doing that. We always use binary cat/join+ BBSub to renumber the ensembles (as outlined in RDI Tools manual). Check file before import (had one case where one of the files had something wrong with time, so had to BBMerge new timestamp first).

BecCowley commented 4 years ago

@sspagnol, thanks. I will re-make the files as you suggest. I knew someone would have a solution!

ocehugo commented 4 years ago

@BecCowley , any plumbing is required to concatenate the files or it's just a simple binary concatenation!? If you could pack some files as examples, I can have a look into the parser and see if we can provide something.

Nonetheless, a quick and simple concatenation is a piece of cake outside of the toolbox:

catfile="concatenated.000"
files=$(ls *.000)
for k in "${files}"; do
[ -f $catfile ] && dd if=$k > $catfile || dd if=$k >> $catfile

You can achieve the same in windows with the code above, but you will need bash in cygwin and the coreutils package installed.

BecCowley commented 4 years ago

@ocehugo, I have adapted the imos code in the past, and run the workhorse parser over the multiple files and concatenated them as we go (ie, run with a cell array of multiple filenames in the input argument). However, I think @sspagnol is correct, I should use the RDI tools as they will concatenate the files correctly.