davidbaines
commented
8 months ago Hi Issac,
I think we might need to modify the preprocess.py main function, as it needs to accept a force_align parameter from the user to the preprocess function. Something like this:
def main() -> None:
parser = argparse.ArgumentParser(description="Preprocesses the parallel corpus for an NMT model")
parser.add_argument("experiment", help="Experiment name")
parser.add_argument("--stats", default=False, action="store_true", help="Output corpus statistics")
parser.add_argument("--force-align", default=False, action="store_true", help="Rerun alignments even if existing alignments already exist.")
args = parser.parse_args()
get_git_revision_hash()
exp_name = args.experiment
SIL_NLP_ENV.copy_experiment_from_bucket(exp_name)
config = load_config(exp_name)
config.set_seed()
config.preprocess(args.stats,force_align=args.force_align)
SIL_NLP_ENV.copy_experiment_to_bucket(exp_name)
The test_preprocess.py file also calls this function and will need to be updated.
Enkidu93
An alignment config can contain several source and target texts and the alignments can take hours. There are many times when an alignment run is interrupted part way through.
Currently running the same command again starts from the beginning again, even if many of the alignments have already been calculated. The default behaviour should be to check whether the file has already been created, and not re-calculate.
We could add a --recalculate option to override this default behaviour.
It would also be useful if the script could search for all the results files in the folder and include them in the final scores.csv file even if they are not expected according to the contents of the config.yml file. These two changes would allow much greater flexibility and efficiency in calculating alignment scores.