Insert ML Filter into main pipeline, can be disabled in .config

[dvida]

Hi Milan, thanks! I looked at the code and if I understand correctly, an updated FTPdetectinfo file is not saved, only the list of FF files are filtered? We'd need to make a new FTPdetectinfo file and save the original as well for archiving purposes. The filtered file shouldn't have any extention, and the original file should have a suffix "_unfiltered".

[satmonkey]

Hi Denis, yes, the function itself takes care about this. The original file name gets suffix like _unfiltered.txt, the filtered file gets the original file name. If reprocessing, the unfiltered file is used. In addition, after every subsequent reprocessing, the file used is saved by appending a single digit and the threshold used, e.g. _0_0850.txt. So we always have the possibility to trace back.

[markmac99]

A complication is that some routines in RMS and in BinViewer select the active FTPdetect file using a statement like if "name.startswith("FTPdetect") and "_uncalibrated" not in name and "backup" not in name: (see for example istream.sh and RetroactiveFixup.py.

If we create additional files starting with FTPdetect, but containing relatively unconstrained strings such as _0850 then could this be a problem for these processes?

[satmonkey]

that should already be fixed by @dvida in master: 13bc4a4bc740dcbfbe6e36d7027693d9bf1a6e74

[markmac99]

cool, objection withdrawn :)