testsmt / yinyang

A fuzzing framework for SMT solvers
https://testsmt.github.io/
MIT License
185 stars 23 forks source link

Parsing error #60

Open depted opened 2 years ago

depted commented 2 years ago

Greetings,

While trying to parse code written in smtlibv2 through yinyang, I found some bugs that yinyang cannot parse some codes.

For example, parsing results of yinyang are (None, None) about the below codes.

Could I know why it happened?

Thanks for your time!

(set-info :smt-lib-version 2.6) (set-logic QF_ABV) (set-info :source | Crafted benchmarks created for "Sharing is Caring: Combination of Theories" by Dejan Jovanovic and Clark Barrett. In Proceedings of the 8th International Symposium on Frontiers of Combining Systems (FroCoS '11), Oct. 2011, pp. 195-210. Based on Example 4 from Section 5. |) (set-info :category "crafted") (set-info :status sat) (declare-fun a0 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x0 () ( BitVec 128)) (declare-fun a1 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x1 () ( BitVec 128)) (declare-fun a2 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x2 () ( BitVec 128)) (declare-fun a3 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x3 () ( BitVec 128)) (declare-fun a4 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x4 () ( BitVec 128)) (declare-fun a5 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x5 () ( BitVec 128)) (declare-fun a6 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x6 () ( BitVec 128)) (declare-fun a7 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x7 () ( BitVec 128)) (declare-fun a8 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x8 () ( BitVec 128)) (declare-fun a9 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x9 () ( BitVec 128)) (declare-fun a10 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x10 () ( BitVec 128)) (declare-fun a11 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x11 () ( BitVec 128)) (declare-fun a12 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x12 () ( BitVec 128)) (declare-fun a13 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x13 () ( BitVec 128)) (declare-fun a14 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x14 () ( BitVec 128)) (declare-fun a15 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x15 () ( BitVec 128)) (declare-fun a16 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x16 () ( BitVec 128)) (declare-fun a17 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x17 () ( BitVec 128)) (declare-fun a18 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x18 () ( BitVec 128)) (declare-fun a19 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x19 () ( BitVec 128)) (declare-fun a20 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x20 () ( BitVec 128)) (declare-fun a21 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x21 () ( BitVec 128)) (declare-fun a22 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x22 () ( BitVec 128)) (declare-fun a23 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x23 () ( BitVec 128)) (declare-fun a24 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x24 () ( BitVec 128)) (declare-fun a25 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x25 () ( BitVec 128)) (declare-fun a26 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x26 () ( BitVec 128)) (declare-fun a27 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x27 () ( BitVec 128)) (declare-fun a28 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x28 () ( BitVec 128)) (declare-fun a29 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x29 () ( BitVec 128)) (declare-fun a30 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x30 () ( BitVec 128)) (declare-fun a31 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x31 () ( BitVec 128)) (declare-fun a32 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x32 () ( BitVec 128)) (declare-fun a33 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x33 () ( BitVec 128)) (declare-fun a34 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x34 () ( BitVec 128)) (declare-fun a35 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x35 () ( BitVec 128)) (declare-fun a36 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x36 () ( BitVec 128)) (declare-fun a37 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x37 () ( BitVec 128)) (declare-fun a38 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x38 () ( BitVec 128)) (declare-fun a39 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x39 () ( BitVec 128)) (declare-fun a40 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x40 () ( BitVec 128)) (declare-fun a41 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x41 () ( BitVec 128)) (declare-fun a42 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x42 () ( BitVec 128)) (declare-fun a43 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x43 () ( BitVec 128)) (declare-fun a44 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x44 () ( BitVec 128)) (declare-fun a45 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x45 () ( BitVec 128)) (declare-fun a46 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x46 () ( BitVec 128)) (declare-fun a47 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x47 () ( BitVec 128)) (declare-fun a48 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x48 () ( BitVec 128)) (declare-fun a49 () (Array ( BitVec 128) (_ BitVec 128))) (declare-fun x49 () ( BitVec 128)) (assert (not (= (select a_0 (bvmul x_0 x_1)) (select a_1 (bvmul x_1 x_2))))) (assert (not (= (select a_1 (bvmul x_1 x_2)) (select a_2 (bvmul x_2 x_3))))) (assert (not (= (select a_2 (bvmul x_2 x_3)) (select a_3 (bvmul x_3 x_4))))) (assert (not (= (select a_3 (bvmul x_3 x_4)) (select a_4 (bvmul x_4 x_5))))) (assert (not (= (select a_4 (bvmul x_4 x_5)) (select a_5 (bvmul x_5 x_6))))) (assert (not (= (select a_5 (bvmul x_5 x_6)) (select a_6 (bvmul x_6 x_7))))) (assert (not (= (select a_6 (bvmul x_6 x_7)) (select a_7 (bvmul x_7 x_8))))) (assert (not (= (select a_7 (bvmul x_7 x_8)) (select a_8 (bvmul x_8 x_9))))) (assert (not (= (select a_8 (bvmul x_8 x_9)) (select a_9 (bvmul x_9 x_10))))) (assert (not (= (select a_9 (bvmul x_9 x_10)) (select a_10 (bvmul x_10 x_11))))) (assert (not (= (select a_10 (bvmul x_10 x_11)) (select a_11 (bvmul x_11 x_12))))) (assert (not (= (select a_11 (bvmul x_11 x_12)) (select a_12 (bvmul x_12 x_13))))) (assert (not (= (select a_12 (bvmul x_12 x_13)) (select a_13 (bvmul x_13 x_14))))) (assert (not (= (select a_13 (bvmul x_13 x_14)) (select a_14 (bvmul x_14 x_15))))) (assert (not (= (select a_14 (bvmul x_14 x_15)) (select a_15 (bvmul x_15 x_16))))) (assert (not (= (select a_15 (bvmul x_15 x_16)) (select a_16 (bvmul x_16 x_17))))) (assert (not (= (select a_16 (bvmul x_16 x_17)) (select a_17 (bvmul x_17 x_18))))) (assert (not (= (select a_17 (bvmul x_17 x_18)) (select a_18 (bvmul x_18 x_19))))) (assert (not (= (select a_18 (bvmul x_18 x_19)) (select a_19 (bvmul x_19 x_20))))) (assert (not (= (select a_19 (bvmul x_19 x_20)) (select a_20 (bvmul x_20 x_21))))) (assert (not (= (select a_20 (bvmul x_20 x_21)) (select a_21 (bvmul x_21 x_22))))) (assert (not (= (select a_21 (bvmul x_21 x_22)) (select a_22 (bvmul x_22 x_23))))) (assert (not (= (select a_22 (bvmul x_22 x_23)) (select a_23 (bvmul x_23 x_24))))) (assert (not (= (select a_23 (bvmul x_23 x_24)) (select a_24 (bvmul x_24 x_25))))) (assert (not (= (select a_24 (bvmul x_24 x_25)) (select a_25 (bvmul x_25 x_26))))) (assert (not (= (select a_25 (bvmul x_25 x_26)) (select a_26 (bvmul x_26 x_27))))) (assert (not (= (select a_26 (bvmul x_26 x_27)) (select a_27 (bvmul x_27 x_28))))) (assert (not (= (select a_27 (bvmul x_27 x_28)) (select a_28 (bvmul x_28 x_29))))) (assert (not (= (select a_28 (bvmul x_28 x_29)) (select a_29 (bvmul x_29 x_30))))) (assert (not (= (select a_29 (bvmul x_29 x_30)) (select a_30 (bvmul x_30 x_31))))) (assert (not (= (select a_30 (bvmul x_30 x_31)) (select a_31 (bvmul x_31 x_32))))) (assert (not (= (select a_31 (bvmul x_31 x_32)) (select a_32 (bvmul x_32 x_33))))) (assert (not (= (select a_32 (bvmul x_32 x_33)) (select a_33 (bvmul x_33 x_34))))) (assert (not (= (select a_33 (bvmul x_33 x_34)) (select a_34 (bvmul x_34 x_35))))) (assert (not (= (select a_34 (bvmul x_34 x_35)) (select a_35 (bvmul x_35 x_36))))) (assert (not (= (select a_35 (bvmul x_35 x_36)) (select a_36 (bvmul x_36 x_37))))) (assert (not (= (select a_36 (bvmul x_36 x_37)) (select a_37 (bvmul x_37 x_38))))) (assert (not (= (select a_37 (bvmul x_37 x_38)) (select a_38 (bvmul x_38 x_39))))) (assert (not (= (select a_38 (bvmul x_38 x_39)) (select a_39 (bvmul x_39 x_40))))) (assert (not (= (select a_39 (bvmul x_39 x_40)) (select a_40 (bvmul x_40 x_41))))) (assert (not (= (select a_40 (bvmul x_40 x_41)) (select a_41 (bvmul x_41 x_42))))) (assert (not (= (select a_41 (bvmul x_41 x_42)) (select a_42 (bvmul x_42 x_43))))) (assert (not (= (select a_42 (bvmul x_42 x_43)) (select a_43 (bvmul x_43 x_44))))) (assert (not (= (select a_43 (bvmul x_43 x_44)) (select a_44 (bvmul x_44 x_45))))) (assert (not (= (select a_44 (bvmul x_44 x_45)) (select a_45 (bvmul x_45 x_46))))) (assert (not (= (select a_45 (bvmul x_45 x_46)) (select a_46 (bvmul x_46 x_47))))) (assert (not (= (select a_46 (bvmul x_46 x_47)) (select a_47 (bvmul x_47 x_48))))) (assert (not (= (select a_47 (bvmul x_47 x_48)) (select a_48 (bvmul x_48 x_49))))) (assert (not (= (select a_48 (bvmul x_48 x_49)) (select a_49 (bvmul x_49 x_0))))) (assert (not (= (select a_49 (bvmul x_49 x_0)) (select a_0 (bvmul x_0 x_1))))) (check-sat) (exit)

depted commented 2 years ago

[2022/12/05 04:30:38 AM] Strategy: typefuzz, 2 testing targets, 15084 seeds Runtime error at /home/smtfuzz/venv/lib/python3.10/site-packages/yinyang/src/parsing/Typechecker.py:524 msg: 'str' object has no attribute 'index_type' cmd: /home/smtfuzz/venv/bin/typefuzz "z3 model_validate=true;cvc4 --check-models -m -i -q" QF_ABV version: yinyang 0.3.0 unknown Please file an issue: https://github.com/testsmt/yinyang/issues

muchang commented 1 year ago

Hi @fwangdo, sorry for the late reply. Does this issue still exist on your side? It seems that yinyang has no runtime error for this case now.

depted commented 1 year ago

Greetings, @muchang. First of all, thanks for your attention.

As for your question, yes, I have this issue still in the latest commit version as well. The code below is what I used to know the result of parsing from yinyang implementation(Note that, 'bv.smt2' in the code below is what we reported before in this issue).

/yinyang# ./bin/typefuzz "z3 model_validate=true;cvc5 --check-models -m -i -q" ./bv.smt2
[2023/03/01 04:24:21 PM] Strategy: typefuzz, 2 testing targets, 1 seeds
the res of 'parse_file' is (None, None)
All seeds processed
1 seeds processed, 0 valid, 1 invalid
0 bug triggers found

/yinyang/tests# python3 ./BvTest.py
the res of 'parse_file' is (None, None)

/yinyang/tests# cat ./BvTest.py
# MIT License
#
# Copyright (c) [2020 - 2021] The yinyang authors
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

import unittest
import sys
sys.path.append("../")
import os
from yinyang.src.parsing.Parse import *
from yinyang.src.parsing.Typechecker import Context, typecheck
from yinyang.src.mutators.GenTypeAwareMutation.GenTypeAwareMutation import *
from yinyang.src.mutators.GenTypeAwareMutation.Util import *

def test():
    path = '../bv.smt2'
    parse_file(path)

if __name__ == "__main__":
    test()

I changed parse_file in Parse.py a little bit like this.

 def parse_file(fn, timeout_limit=30, silent=True):
    """
    Parse SMT-LIB file.

    :fn: path to SMT-LIB file.
    :silent: if silent=True the parser will withhold stacktrace from user
             on crash.
    :returns: Script object representing AST of SMT-LIB file. None if timeout
              or crash occurred.
    """
    res = parse(parse_filestream, fn, timeout_limit, silent)
    print("the res of 'parse_file' is", res)
    return parse(parse_filestream, fn, timeout_limit, silent)
depted commented 1 year ago

Is there anything that I did wrong? If so, I hope you tell me the point. I will use yinyang in your intention and tell you the result. :)