BER decoding is sensitive to trailing bytes that other implementations ignore

[pihlerm]

I discovered an issue with BER_Decoder when decoding BER encoded constructed elements with indefinite length.

Since those elements use EOC element (with two 0 byte tags) to define end of construction, the function

bool BER_Decoder::more_items() const

doesn't detect the end of set correctly. I fixed this by adding detection of EOC:

bool BER_Decoder::more_items() const
   {
   if(m_source->end_of_data() && !m_pushed.is_set())
      return false;
   static byte tag[2];
   m_source->peek(tag,2,0);
   // next element is EOC 
   if(tag[0] == 0 && tag[1]==0) {
       m_source->read(tag,2);
       return false;
   }
   return true;
   }

Another issue is with DataSource_BERObject::peek function It is used by find_eoc to calculate size of indefinite elements.

This function ignores current read position (m_offset) and behaves incorrectly when element is in the middle of buffer.

I fixed it like this:

size_t peek(uint8_t out[], size_t length, size_t peek_offset) const override
         {
         BOTAN_ASSERT_NOMSG(m_offset+peek_offset <= m_obj.length());
         const size_t bytes_left = m_obj.length() - m_offset;

         if(peek_offset >= bytes_left)
            return 0;

         const size_t got = std::min(bytes_left - peek_offset, length);
         copy_mem(out, m_obj.bits() + m_offset + peek_offset, got);
         return got;
         } 

I hope this helps someone to fix the issues in the code.

[reneme]

Do you have an encoded BER object as an example?

Please feel free to open a pull request with your changes.

[mouse07410]

Would supporting decoder of BER with infinite length create an attack surface? What's the use case of it in a crypto library/application?

[reneme]

The way I understand ber_dec.h, we do support indefinite length values already. What @pihlerm is pointing out here is a bug in reading such values. Or am I mistaken?

[randombit]

IIRC we do support indefinite length encodings. Possibly we shouldn't but that's another issue.

[It may be worthwhile to explore configuration options to the decoder which limits what we are willing to accept, to minimize attack surface.]

[pihlerm]

I could post an example, but generally it's a signature embedded in PDF. Some tools use BER encoders (instead of DER), for example BouncyCastle, thus producing indefinite length constructions. For example DSS tools

Regarding pull request - I'm working with an older c++11 compliant botan version. That's why I posted my changes directly. OK, here's an example (PKCS7/CMS):

308006092A864886F70D010702A0803080020101310F300D06096086480165030402010500308006092A864886F70D0107010000A080308207073082056FA003020102020C65B2C9AC0000000059529C96300D06092A864886F70D01010B0500305D310B3009060355040613025349311F301D060355040A1316504F53544120534C4F56454E494A4520642E6F2E6F2E311730150603550461130E56415453492D3235303238303232311430120603550403130B504F535441724341204732301E170D3138313031383039353932385A170D3233313031383130323932385A3066310B3009060355040613025349312D3014060355040A130D53455443434520442E4F2E4F2E30150603550461130E56415453492D33393730353638343128300D06035504051306323438323833301706035504031310655065726F205354415254207465737430820122300D06092A864886F70D01010105000382010F003082010A0282010100A16169899D1BB6DE2D11D40A7E43E0E4DF074DEB93EC617A8D18C1807FC7CDBCCDA0AD9E2A04C77586C24B296FEE041DD9BEBB901C64CAF246EE1E94D168DFB0DCDEFC0CAA17C1A76493DFD007CF299B6CD6D7745D876676B1EA61906E45AC4B816D22DBC86E6702952AC52164E913E5A97109305420CB54F16318351386F18A74420A5B3964EF7D842E7FFE3576EE190E861A8184CCB595C16077A6FC258824B2A6F686287B2535FE0BCF2959B76FF60E3152764BC50932295B521DEA50D5CB1FA6E7AD4E4D6B84F02FE9ADB56D412209E574E983B968E2718DFFD1E776D432D60ADB01EE7029723009D21701E820B1EF1E6ECB93AEF012D627A5B72DA03FE70203010001A382033C30820338300E0603551D0F0101FF0404030205E030570603551D200450304E3041060D2B06010401F7340104020102013030302E06082B060105050702011622687474703A2F2F706F7374617263612E706F7374612E73692F646F6B756D656E74693009060704008BEC4001013081BA06082B060105050701030481AD3081AA3008060604008E460101308188060604008E460105307E303D163768747470733A2F2F706F7374617263612E706F7374612E73692F646F6B756D656E74692F706F7374617263615F7064735F656E2E7064661302656E303D163768747470733A2F2F706F7374617263612E706F7374612E73692F646F6B756D656E74692F706F7374617263615F7064735F736C2E7064661302736C3013060604008E4601063009060704008E46010602306C06082B060105050701010460305E303A06082B06010505073002862E687474703A2F2F706F7374617263612E706F7374612E73692F66696C65732F504F5354417243412D47322E636572302006082B060105050730018614687474703A2F2F6F6373702E706F7374612E7369301B0603551D11041430128210655065726F2053544152542074657374308201500603551D1F04820147308201433081C9A081C6A081C3862C687474703A2F2F706F7374617263612E706F7374612E73692F63726C2F706F7374617263612D67322E63726C8681926C6461703A2F2F706F7374617263612E706F7374612E73692F636E3D504F53544172434125323047322C6F7267616E697A6174696F6E4964656E7469666965723D56415453492D32353032383032322C6F3D504F535441253230534C4F56454E494A45253230642E6F2E6F2E2C633D53493F63657274696669636174655265766F636174696F6E4C6973743B62696E6172793075A073A071A46F306D310B3009060355040613025349311F301D060355040A1316504F53544120534C4F56454E494A4520642E6F2E6F2E311730150603550461130E56415453492D3235303238303232311430120603550403130B504F535441724341204732310E300C0603550403130543524C313030130603551D23040C300A80084B00EDF20E97A37C30110603551D0E040A04084640F5FF6DBC3B6A30090603551D1304023000300D06092A864886F70D01010B0500038201810058BB2D5BEB4E08D3FBA2009E3B96012A087BE6F136151C3BBF57F88AD41EC3E1DC37F19298FB49CCD6C569DBEBC0DB7AD90156B89902CA67307E4D82A91DD83BF7DD8E3D742D16B26216664837318A5B718F48804D9ADBF2A9CF3C431936DE5DEF1DAE01BBEB7D61DD2864F42502F649E88D6960A70DE9F6222353427370D78DFAA10DF01BA4FF4360A5823A51D91A62CB41387FCF3678028125FEB7D789570E475B8B0D55F0065B9EDF6507CFC7374EFA3B98AF2D955893AEB9A93AE36DF8F145FFA32B450C84D489AFF992CE18464260A41806DC1938111E3AB93EBAB243557768D3E137764C95645915ABE369BB0E8A4D4F31D666DBEF38F217227762E260EC92A0498764B46DFE6E74231C6A35FF5B714A9D9CDB262D6C787DF149DED7A808CA699F522AB136B837F039C2BB0A3DD89F309ABAA6B6842E89DFBBC8336B945414F8D976482B5DABFDEA8D12B8109F6A58FFCD21F21D31CF6948C9A4EE761A983A546910AE0F6508595167A2E27313CCD108761AF95365C234AF00529A5020308206A33082050BA003020102020C16D01B840000000059522321300D06092A864886F70D01010B0500305F310B3009060355040613025349311F301D060355040A1316504F53544120534C4F56454E494A4520642E6F2E6F2E311730150603550461130E56415453492D3235303238303232311630140603550403130D504F53544172434120526F6F74301E170D3137303632373039353235325A170D3335303632373130323235325A305D310B3009060355040613025349311F301D060355040A1316504F53544120534C4F56454E494A4520642E6F2E6F2E311730150603550461130E56415453492D3235303238303232311430120603550403130B504F535441724341204732308201A2300D06092A864886F70D01010105000382018F003082018A02820181009586C256B0B5060EE226E8D638DB28B7AA6A8B13217325BC1E6E0F0D77A4389488FB074DC5814CF1A14156289589130F017892D9A72B7F24A6A201333E07446E89C853232A94F7699BEFC6426780CD03ABD89441F881472C447CCA8394010E6859C0E841168A35C4CCD97709E19C25E0C61A948D0E2D771625155958D7EFA351ADDD941EA914CB0F5CC876B3428DDCB5C6D622755F8451121CB16ACD69993B872B43E4148A31662FB2CF085478D83C1A03FFF0944FF795885D3A30566815A820743760A9D574D5C6B571D828D1EDC66EBC787B6FC8A0DABEB4E0576E734DBF7CFD3FEBBC906467E0A0F6EB8F55D0C6DB7E6C0CEA5AEF0F3057FE426C895F2D5FEAEA79E34A22EF46F40D62EB64FB32EB782B53FDA0297E391C345E63FC9C82CB7F5DAFAC6E33F3BDAFE5172A1DF3EB25B5E1200B997B0911BFD852B95915A1A0D14B71AA4B0C628FF2BADF19A1C69444E6CB225AD4CD2244F239E234ACE7B53535DDE3734EA1CD2AD558880530098A44B1F28E6563381B506BF6218C948F4C350203010001A382025F3082025B30120603551D130101FF040830060101FF020100300E0603551D0F0101FF04040302010630440603551D20043D303B30390604551D20003031302F06082B060105050702011623687474703A2F2F706F7374617263612E706F7374612E73692F646F6B756D656E74692F306E06082B0601050507010104623060303C06082B060105050730028630687474703A2F2F706F7374617263612E706F7374612E73692F66696C65732F504F5354417243412D526F6F742E636572302006082B060105050730018614687474703A2F2F6F6373702E706F7374612E7369308201550603551D1F0482014C308201483081CDA081CAA081C7862E687474703A2F2F706F7374617263612E706F7374612E73692F63726C2F504F5354417243412D526F6F742E63726C8681946C6461703A2F2F706F7374617263612E706F7374612E73692F636E3D504F535441724341253230526F6F742C6F7267616E697A6174696F6E4964656E7469666965723D56415453492D32353032383032322C6F3D504F535441253230534C4F56454E494A45253230642E6F2E6F2E2C633D53493F63657274696669636174655265766F636174696F6E4C6973743B62696E6172793076A074A072A470306E310B3009060355040613025349311F301D060355040A1316504F53544120534C4F56454E494A4520642E6F2E6F2E311730150603550461130E56415453492D3235303238303232311630140603550403130D504F53544172434120526F6F74310D300B0603550403130443524C3130130603551D23040C300A80084389A6EB178A6F7E30110603551D0E040A04084B00EDF20E97A37C300D06092A864886F70D01010B050003820181005A08D49570D193B320DC4DE516E7C6C28BA8A2A7C1ED5C769267BF5135E677B3379C4BC447A49A27680B3E468C5A8EBD44B4B1D24E1AE2B4517C758E23E4D9FE0110D33A541BCC1C037861F4917C7DC440946BF45B1BA55642C3DE4E82A6FB90A73F6F2D6C8A6FA67FFF72DB562BDB1EB429D2CBA1158CDFB693F8523FBCC7873A5CEC4BD3CF9867DBA2368162C82115FB01A6766D9EC116249FABDE1C1494BED41B584B6488D7C9A5AAED6D791FDDBCEB6E7BD66FA38B69B130019A7A680EFC275B1FDD043B4C2FFF50C5CBD47D1F855214081BC89E531404C2AFC85B1939B658F6B3BE22AEF80EB905CA3C3FA437E8D8F2AF792F0DEC1CAC1C65B18317D385324839A080669EF58C04EDF4FBC4333B30A7A6B9BF7F7F25E0477BEFC7BF5588060535C9D0D85FF0D65AEAD1F8D1E40EBEC8787ACE78FAAC081896754DAB860271782FDC00F3F9C4DEDFB27DB9E0DE579B4C74A76D02119FED18BBB4497D8B346A6F654D3300C733C7F1B36511D12AD5723BEE1A932D2FAE8C9570E42EC47E183082048F308202F7A003020102020C252A24F900000000595222E6300D06092A864886F70D01010B0500305F310B3009060355040613025349311F301D060355040A1316504F53544120534C4F56454E494A4520642E6F2E6F2E311730150603550461130E56415453492D3235303238303232311630140603550403130D504F53544172434120526F6F74301E170D3137303632373038343834345A170D3337303632373039313834345A305F310B3009060355040613025349311F301D060355040A1316504F53544120534C4F56454E494A4520642E6F2E6F2E311730150603550461130E56415453492D3235303238303232311630140603550403130D504F53544172434120526F6F74308201A2300D06092A864886F70D01010105000382018F003082018A028201810081B8BCD16C412F119A8C7A1E6B489D426894233C61589AE71C993F2DDBDB61F4DD22A098D602F860B722F2066547D631FC106A9546AAC4A152C181A958602261A81AEA82D0D477354C9C81E3F0C6690442633E7F00F8013134E52EAA274F674E94D881F75FF74EAC3FACFBA39F2F885D3BE1C9976FF3A4F6926AD2D36620CC8D20690B643F7132872BCC17ADB385BE1C4CA67B2099483627A7B84B99E51D511562ED873C804B6F7C5F1299752AAB04EC22AD19FDC9C6DA4C557007C4EA6EA8594D14F151B28468417C47E940082A353ED869BEE0484C077F915066C8621BDDE69014637908AB8641214B9D85779AA120E53705A0D28C64B5786258E90FD14058159212024B6B1D59751DF8EFFA59E73C9683DB09350A8CBD522A306D06D133FD303025EC4B93C0FBF5F1015BAD5B7A223F9545D4121291DC3EC2D1A68193D914E7C8C15FE9A35DDFFDFE3BC37EB08206748EB45772524DB07FC9A45F0A04C4E985456C06C56B6C2467C455C74723C68BD0A5D8C191AA560C6B1B758B7F926DD50203010001A34B3049300F0603551D130101FF040530030101FF300E0603551D0F0101FF04040302010630130603551D23040C300A80084389A6EB178A6F7E30110603551D0E040A04084389A6EB178A6F7E300D06092A864886F70D01010B050003820181007CD78E35C1BE71BD5E865022B9996C136BEB7099A9DD0B1EC27EE56952EC0CA0151485BDA364F4765A83D023A6DAC9DA8FC764E3A9E3B7FAE6EF4136D45C24594E689E718F7E66392C88A3BE192531C4854739B6953F8A1D853DD67BA61ACDB815BFF52AA29856E7AB708AB881944E84770222F97FEB01999CE9AC03E58475A246B9C8FE2F443FE01F687F7CB7081490980D0DCB6CE739A55FFB51920D34102C1DB9E5A3A2AD123937A495B6C14E784E835FD3AFFCEFC30861E60B31B8834B38A7909A2345ED6A1D6AADB155FFA04FAD6138F139B52BC0772EBDC971AC3F39F27237C2F73494EDCAF0BA6BA1FA29B2C412B1AB0B95DC21A6756D1789292420DC090129BF34ECC320B5C775920DF477B10E5E3581C16FC20FF7060C7AA8A427AD600FF65DF5C33C3B8FEA8015EADF6E6DAF6A33009684EEDEA3E3A68F2937B30F959B3341D0161321793F4529E823C7BA435E76BB5692A27FC2D778A959AE097CDBAA504EB9A0078E4F22B3B5EF3D68F5439B6C4608C991F9457E7A6732C17A850000318202333082022F020101306D305D310B3009060355040613025349311F301D060355040A1316504F53544120534C4F56454E494A4520642E6F2E6F2E311730150603550461130E56415453492D3235303238303232311430120603550403130B504F535441724341204732020C65B2C9AC0000000059529C96300D06096086480165030402010500A08198301806092A864886F70D010903310B06092A864886F70D010701301C06092A864886F70D010905310F170D3233303530383131323033375A302D06092A864886F70D0109343120301E300D06096086480165030402010500A10D06092A864886F70D01010B0500302F06092A864886F70D010904312204205A4D9A1CACCB8C234ED57AE79A8EDF24B7EEC08394715E95E4CD2AB0378812D3300D06092A864886F70D01010B0500048201007892013AD5525C1FFD74CFA536CB33F463A8EDEC07BA15DDCCB71CC5D3817A3023AB425C3FFCDEDD092A06375DD26B192F697D708408211EA18DD0FBEEFC1FC6DB946250582EB9BCE788354C6D4E1FAF1C8D436A5EDA0776C3A9B7518E27EEC80F50D45BFCF975B3545A65E322F885ADA76875769AB5485C84787B3C82599983C04B30B3E1C6764252A127C57747317DA707D76AB028D9827891673C565224F5833277CFE555061C46FBCFA8A7100B873998D63C8F6E7CB515900F81232BA9D8B8B2BE6E1B85A5368B70441AAA53AB9E9EE81D464F749D61005E2B15442BEC23A6EB32672E95CC818C3F0F09D74BB4250B5943B2988B410BDFABB7B4B8FDC96000000000000000000000000000

[randombit]

Thanks for posting the example data. Confirmed that we can't read it while openssl asn1parse can.

[pihlerm]

Thank you all for great work.

[randombit]

So I was having a lot of problems figuring out what the issue was until I pasted your input into https://lapo.it/asn1js which dumps the structure and additionally reports "Input contains 7 more bytes to decode". If I strip those final 7 trailing zero bytes, decoding works perfectly.

I think your change to more_items is not really correct - instead it happens to detect this extra trailing zero condition.

The bug in peek is real, and in fact fixing that exposed a problem in some hand generated test data. We had some tests for comparing X.509 DNs with indefinite length encodings, but one of those DN encodings was actually completely invalid.

I suspect the difference comes because most ASN1 parsers read the initial TLV, and then will ignore any bytes that are outside that maximum TLV. Whereas we will try to continue reading until all bytes in the input stream have been exhausted.

I'm not sure what the best resolution here is.

I opened #3575 adding the test case above (with the 7 trailing zero bytes removed), along with fixing peek

[randombit]

Oh I should have figured this out earlier by noticing that the output from openssl asn1parse on the example input ends with

 5300:d=3  hl=2 l=   0 prim: EOC
 5302:d=2  hl=2 l=   0 prim: EOC
 5304:d=1  hl=2 l=   0 prim: EOC

[pihlerm]

Hello,

I tried decoding PKCS7 message from my example, minus extra 7 bytes, without the fix to BER_Decoder::more_items(). It does not work Exception : BER: Tag mismatch when decoding object got EOF expected SEQUENCE/CONSTRUCTED

With posted fix to detect EOC, it decodes correctly.

Are you decoding the PKCS7 message with real structure? The exception happens when decoding certificates, if I reference asn.1 js decoder: certificates CertificateSet [0] (3 elem)

My two fixes posted above have been thorougly tested with real examples of signed PDFs with other production tools like AdobeReader, NitroPDF, Recono qes etc. They all decode perfectly.

My 2c.

[pihlerm]

Below I'm posting the code I'm using to decode th posted CMS/PKCS7. It is not complete, since there is more code involved, but you can skip some lines and try it. It fails in while(certificates.more_items() loop.

` Botan::DataSource_Memory pkcs7_source(buf,size);

    // quick check it it's BER at all
    if (!Botan::ASN1::maybe_BER(pkcs7_source)) return false;

    Botan::BER_Decoder decoder(pkcs7_source);

    Botan::OID content_type;
    Botan::BER_Decoder content_info = decoder.start_cons(Botan::SEQUENCE);

    content_info.decode(content_type);
    if (content_type != Botan::OID("1.2.840.113549.1.7.2")) return false; // Signeddata (PKCS #7)

    // start parse signed data
    Botan::BER_Object content_obj = content_info.get_next_object(); // content[0]

    Botan::BER_Decoder signed_data(content_obj.value);
    size_t cms_version;
    Botan::BER_Decoder signed_data_seq = signed_data.start_cons(Botan::SEQUENCE);

        // version
        signed_data_seq.decode(cms_version);

        // digestAlgorithms
        Botan::BER_Decoder algorithms = signed_data_seq.start_cons(Botan::SET);
        while (algorithms.more_items())
        {
            Botan::AlgorithmIdentifier hash_algo_id;
            algorithms.decode(hash_algo_id);
            // determine hash algo

            std::string strhash = Botan::OIDS::oid2str_or_empty(hash_algo_id.get_oid());
            _hashAlg = getHashAlg(strhash);

            //Botan::BER_Object obj = algorithms.get_next_object();
        }           
        algorithms.discard_remaining();
        algorithms.end_cons();

        // encapContentInfo
        Botan::BER_Decoder encapContentInfo = signed_data_seq.start_cons(Botan::SEQUENCE);
        Botan::OID eContentType;
        encapContentInfo.decode(eContentType);
        encapContentInfo.discard_remaining();
        encapContentInfo.end_cons();

        // certificates
        Botan::X509_Certificate cert;
        Botan::BER_Object  certificates_obj = signed_data_seq.get_next_object();
        uint8_t num_certs = certificates_obj.value[0];
        Botan::BER_Decoder certificates(certificates_obj.value);

        // signing cert is not necessarily the first cert
        while (certificates.more_items()) {
            certificates.decode(cert);
            _chain.push_back(cert);
        }

        // signer infos
        Botan::BER_Decoder signer_infos = signed_data_seq.start_cons(Botan::SET);
        // decode signer info
        Botan::BER_Object  signer_info_obj = signer_infos.get_next_object();
        Botan::BER_Decoder signer_info(signer_info_obj.value);
        decode_signer_info(signer_info);
        signer_infos.end_cons();

    signed_data_seq.discard_remaining();
    signed_data_seq.end_cons();

    // find signing cert by matching serial from signer info
    for (auto itr = _chain.begin(); itr != _chain.end(); itr++) {
        if (itr->serial_number() == _Certserial) {
            ByteBuffer bbcert = itr->BER_encode();
            if (_ssigner != nullptr)
            {
                // already have signer
                Log("CMSignatureBotan: attempt to create duplicate signer cert!");
            }
            else {
                _ssigner = platform->CreateCertificate((unsigned char*)&bbcert[0], bbcert.size());
                _release_ssigner = true;
            }
        }           
    }

}
catch (const Botan::Exception &ex) {
    std::string sw = ex.what();
    return false;
}
catch (const Botan::Decoding_Error &ex) {
    std::string sw = ex.what();
    return false;
}`

[pihlerm]

Additionally, your analysis

I think your change to more_items .. happens to detect this extra trailing zero condition.

is correct. EOC detection automatically fixes the zero-padding detection as well.