It can happen that data directories get into a broken state which breaks building the data target. It would be useful to have a "clean" target to clean data, models, or both.
eg. if files in /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/ end up empty, it happens that:
for d in /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/GNOME.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/KDE4.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/OpenSubtitles.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/QED.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/Ubuntu.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/wikimedia.br-en.clean.br.gz; do \
l=`/projappl/project_2001194/bin/pigz -cd < $d | wc -l`; \
if [ $l -gt 0 ]; then \
echo "$d" | xargs basename | \
sed -e 's#.br.gz$##' \
-e 's#.clean$##'\
-e 's#.br-en$##' | tr "\n" ' ' >> /local_scratch/hardwick/br-en/train/README.md; \
echo -n "($l) " >> /local_scratch/hardwick/br-en/train/README.md; \
fi \
done
pigz: skipping: <stdin> empty
pigz: skipping: <stdin> empty
pigz: skipping: <stdin> empty
pigz: skipping: <stdin> empty
pigz: skipping: <stdin> empty
pigz: skipping: <stdin> empty
echo "" >> /local_scratch/hardwick/br-en/train/README.md
echo "only one target language"
only one target language
zcat /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/GNOME.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/KDE4.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/OpenSubtitles.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/QED.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/Ubuntu.br-en.clean.br.gz /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/wikimedia.br-en.clean.br.gz > /local_scratch/hardwick/br-en/train/opus.src.br-en.src
gzip: /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/GNOME.br-en.clean.br.gz: unexpected end of file
make[1]: *** [add-to-local-train-data] Error 1
make[1]: Leaving directory `/scratch/clarin/hardwick/OPUS-MT-train'
It can happen that data directories get into a broken state which breaks building the data target. It would be useful to have a "clean" target to clean data, models, or both.
eg. if files in /scratch/clarin/hardwick/OPUS-MT-train/work/data/simple/ end up empty, it happens that:
I was able to fix this by
rm -rf work/data/simple