some functions mask RNP_ERROR_BAD_FORMAT error

[rrrooommmaaa]

Description

When using rnp_op_verify_execute with detached signature file in incorrect format, I receive RNP_ERROR_BAD_PARAMETERS error code instead of RNP_ERROR_BAD_FORMAT.

First, the error is masked in src/librepgp/stream-parse.cpp, lines 819-826:

    if (stream_parse_signature(readsrc, &readsig) != RNP_SUCCESS) {
        RNP_LOG("failed to parse signature");
        siginfo->unknown = true;
        if (sig) {
            *sig = NULL;
        }
        return RNP_SUCCESS;
    }

Secondly, in lines 2093-2094:

            /* no need to check the error here - we already know tag */
            signed_read_single_signature(param, readsrc, &sig);

Steps to Reproduce

    /* Load RSA signature with PGP_MPINT_BITS + 1 size (16385) MPI. Must fail. */

    assert_rnp_success(

      rnp_input_from_path(&input, "data/test_large_MPIs/rsa-pub-65535bits.pgp"));

    assert_rnp_success(rnp_input_from_path(

      &signature, "data/test_large_MPIs/rsa-pub-65535bits.pgp.16385sig.sig"));

    assert_rnp_success(rnp_op_verify_detached_create(&verify, ffi, input, signature));

    assert_int_equal(rnp_op_verify_execute(verify), RNP_ERROR_BAD_FORMAT); 

Expected Behavior

I would expect to get RNP_ERROR_BAD_FORMAT

Actual Behavior

What behavior did you observe instead?

I get RNP_ERROR_BAD_PARAMETERS error code

[antonsviridenko]

https://github.com/rnpgp/rnp/issues/941#issuecomment-565622529