try processing pipelines on newly downloaded data

[stevehadd]

currently we are usng data that was downloaded by Matt Palmer. We should check that the code works with a fresh download of the XBT from WOD.

[stevehadd]

Having looked at the freshly downloaded data, I realised that it is not organised into files per year. I updated the script to process files with an arbitrary naming convention, but then realised that the main scripts expect the CSV input files to be arranged by year, which is not the case for those CSV files produced by the revised preprocessing script. I think we need a slightly revised script to be more flexible in the input it can handle, while still producing CSV files arranged by year. So the scripts should be updated as follows:

• specify an arbitrary regex for the input netcdf files to be processed
• write out CSV files per year, where files for a year may come from an arbitrarily large number of input files.
• when writing out, the script will check whether a file for a year exists, if not create it, else append to it.
• we might consider arranging the files by some other attribute e.g. instrument, cruise, country
• the template for output files should be the same as for input to the next stage. Should we make the scripts more flexible, and not insist on CSVs arrange by year?

[stevehadd]

I have updated the wiki page to reflect the new preprocessing script, and the ability to run preprocessing as part of training. https://github.com/MetOffice/XBTs_classification/wiki/Running-XBT-experiments

[stevehadd]

Developments to address this issue:

• downloaded WOD data, found that differently porganised.
• refactored preprocessing code to be able to handle WOD data which is not organised per year, but still output per year CSV files for input to ML scripts
• added ability to run preprocessing as first step of training or inference