search

MetOffice / XBTs_classification

Project for the classification of eXpendable Bathy Thermographs
BSD 3-Clause "New" or "Revised" License
4 stars 2 forks source link

create update mode inference script #45

Closed stevehadd closed 3 years ago

stevehadd commented 4 years ago

Another use case not previously considered is running inference on the (approximately) monthly updates to the WOD XBT obs database, as new observations come in. This will involve generating classifications for the new profiles using previously trained classifiers. The old data does not need to be loaded, and the new classifications should be output in a separate file. This will make the processing quite quick, and make it easy to archive incrementally each month.

Add additional complication is that the monthly update data comes from a different source to the usual WOD source, and may be slightly differently formatted. We may need a different preprocessing script to convert to the current standard tabular data format saved as a CSV, produce by this script

stevehadd commented 3 years ago

This functionality is already in place, as one can specify the directory with the input files (either preprocessed CSV files or the netCDF files direct from WOD), and a directory for output, and an experiment file, and away you go. We don't need a separate update mode after all. This may change after further testing, but I'll close this issue for now and reopen if necessary.