google / sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.
Apache License 2.0
10.32k stars 1.18k forks source link

entry points return non-zero exit code (at least for `--help`) #978

Closed h-vetinari closed 9 months ago

h-vetinari commented 9 months ago

Thanks a lot for the many clean-ups in 0.2.0! I was able to reduce the patches necessary for #876 quite substantially, I'll update that issue soon-ish.

In the meantime, I'm running into a new error that I need to figure out... As part of the integrity checks for the built package, we test that we can call the various CLI entry points without errors - for simplicity only with --help. For 0.1.99 and before this worked fine, but now I get:

+ spm_decode --help
spm_decode: Warning: SetProgramUsageMessage() never called

  Flags from src/spm_decode_main.cc:
    --extra_options (':' separated encoder extra options, e.g.,
      "reverse:bos:eos"); default: "";
    --input (input filename); default: "";
    --input_format (choose from piece or id); default: "piece";
    --model (model file name); default: "";
    --output (output filename); default: "";
    --output_format (choose from string or proto); default: "string";

Try --helpfull to get a list of all flags or --help=substring shows help for
flags which include specified substring in either in the name, or description or
path.

and then a non-zero exit code which generally indicates some sort of failure. This happens on all unix builds (windows is running into an unrelated problem that I'm in the process of fixing).

h-vetinari commented 9 months ago

@taku910, could you comment on whether the non-zero exit code for --help is expected in 0.2.0?

taku910 commented 9 months ago

This behavior happens when using the official abseil library with -DSPM_ABSL_PROVIDER=module. However, it turned out that it is the correct and intended behavior. The previous and default abseil-flag-compatible library was not correctly ported. We've changed the return code to be 1 not zero, which is the same as the official abseil library.