search

microsoft / restler-fuzzer

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
MIT License
2.52k stars 285 forks source link

Running latest fuzz version of RESTler - Results Analyzer fails #363

Open mbiuki opened 2 years ago

mbiuki commented 2 years ago

After updating RESTler to the latest version and running it with: Restler fuzz --grammar_file Compile/grammar.py --settings Compile/restler_user_settings.json --dictionary_file Compile/dict.json

I get the following error:

>>Restler/Fuzz$ cat ResultsAnalyzerStdErr.txt

Warning: could not parse request in line 3091: GET /endpoint/v1/endpoints?pageFromKey=fuzzstring&pageSize=1&pageTotal=true&sort=fuzzstring&healthStatus=bad&type=computer&tamperProtectionEnabled=true&lockdownStatus=creatingWhitelist&lastSeenBefore=09/23/2019 12:02:01&lastSeenAfter=09/23/2019 12:02:01&ids=fuzzstring&isolationStatus=isolated&hostnameContains=fuzzstring&associatedPersonContains=fuzzstring&groupNameContains=fuzzstring&search=fuzzstring&searchFields=hostname&ipAddresses=fuzzstring&cloud=&fields=fuzzstring&view=basic&assignedToGroup=true&groupIds=60c82cabca5e499696709fedf3e5a623 HTTP/1.1\r\nAccept: application/json\r\nHost: api-tk01.dev.central.company.com\r\n_OMITTED_AUTH_TOKEN_\r\nX-Tenant-ID: 566048da-ed19-4cd3-8e0a-b7e0e1ec4d72\r\nContent-Length: 0\r\nUser-Agent: restler/8.0.0\r\n\r\n
Warning: could not parse request in line 3209: GET /endpoint/v1/endpoints?pageFromKey=fuzzstring&pageSize=1&pageTotal=true&sort=fuzzstring&healthStatus=bad&type=computer&tamperProtectionEnabled=true&lockdownStatus=creatingWhitelist&lastSeenBefore=09/23/2019 12:02:01&lastSeenAfter=09/23/2019 12:02:01&ids=fuzzstring&isolationStatus=isolated&hostnameContains=fuzzstring&associatedPersonContains=fuzzstring&groupNameContains=fuzzstring&search=fuzzstring&searchFields=hostname&ipAddresses=fuzzstring&cloud=&fields=fuzzstring&view=basic&assignedToGroup=void&groupIds=66c081d61b364840859158630121f300 HTTP/1.1\r\nAccept: application/json\r\nHost: api-tk01.dev.central.company.com\r\n_OMITTED_AUTH_TOKEN_\r\nX-Tenant-ID: 566048da-ed19-4cd3-8e0a-b7e0e1ec4d72\r\nContent-Length: 0\r\nUser-Agent: restler/8.0.0\r\n\r\n
Warning: could not parse request in line 3327: GET /endpoint/v1/endpoints?pageFromKey=fuzzstring&pageSize=1&pageTotal=true&sort=fuzzstring&healthStatus=bad&type=computer&tamperProtectionEnabled=true&lockdownStatus=creatingWhitelist&lastSeenBefore=09/23/2019 12:02:01&lastSeenAfter=09/23/2019 12:02:01&ids=fuzzstring&isolationStatus=isolated&hostnameContains=fuzzstring&associatedPersonContains=fuzzstring&groupNameContains=fuzzstring&search=fuzzstring&searchFields=hostname&ipAddresses=fuzzstring&cloud=&fields=fuzzstring&view=basic&assignedToGroup=null&groupIds=3a4549f2ec8642d9a2f08a90f67bc906 HTTP/1.1\r\nAccept: application/json\r\nHost: api-tk01.dev.central.company.com\r\n_OMITTED_AUTH_TOKEN_\r\nX-Tenant-ID: 566048da-ed19-4cd3-8e0a-b7e0e1ec4d72\r\nContent-Length: 0\r\nUser-Agent: restler/8.0.0\r\n\r\n
Warning: could not parse request in line 3445: GET /endpoint/v1/endpoints?pageFromKey=fuzzstring&pageSize=1&pageTotal=true&sort=fuzzstring&healthStatus=bad&type=computer&tamperProtectionEnabled=true&lockdownStatus=creatingWhitelist&lastSeenBefore=09/23/2019 12:02:01&lastSeenAfter=09/23/2019 12:02:01&ids=fuzzstring&isolationStatus=isolated&hostnameContains=fuzzstring&associatedPersonContains=fuzzstring&groupNameContains=fuzzstring&search=fuzzstring&searchFields=hostname&ipAddresses=fuzzstring&cloud=&fields=fuzzstring&view=summary&assignedToGroup=true&groupIds=2bd181a306e94dda8588a9ccc642afc9 HTTP/1.1\r\nAccept: application/json\r\nHost: api-tk01.dev.central.company.com\r\n_OMITTED_AUTH_TOKEN_\r\nX-Tenant-ID: 566048da-ed19-4cd3-8e0a-b7e0e1ec4d72\r\nContent-Length: 0\r\nUser-Agent: restler/8.0.0\r\n\r\n
Warning: could not parse request in line 3563: GET /endpoint/v1/endpoints?pageFromKey=fuzzstring&pageSize=1&pageTotal=true&sort=fuzzstring&healthStatus=bad&type=computer&tamperProtectionEnabled=true&lockdownStatus=creatingWhitelist&lastSeenBefore=09/23/2019 12:02:01&lastSeenAfter=09/23/2019 12:02:01&ids=fuzzstring&isolationStatus=isolated&hostnameContains=fuzzstring&associatedPersonContains=fuzzstring&groupNameContains=fuzzstring&search=fuzzstring&searchFields=hostname&ipAddresses=fuzzstring&cloud=&fields=fuzzstring&view=summary&assignedToGroup=void&groupIds=8840829a38d74b57b106e1bdeb926dfa HTTP/1.1\r\nAccept: application/json\r\nHost: api-tk01.dev.central.company.com\r\n_OMITTED_AUTH_TOKEN_\r\nX-Tenant-ID: 566048da-ed19-4cd3-8e0a-b7e0e1ec4d72\r\nContent-Length: 0\r\nUser-Agent: restler/8.0.0\r\n\r\n
Warning: could not parse request in line 3681: GET /endpoint/v1/endpoints?pageFromKey=fuzzstring&pageSize=1&pageTotal=true&sort=fuzzstring&healthStatus=bad&type=computer&tamperProtectionEnabled=true&lockdownStatus=creatingWhitelist&lastSeenBefore=09/23/2019 12:02:01&lastSeenAfter=09/23/2019 12:02:01&ids=fuzzstring&isolationStatus=isolated&hostnameContains=fuzzstring&associatedPersonContains=fuzzstring&groupNameContains=fuzzstring&search=fuzzstring&searchFields=hostname&ipAddresses=fuzzstring&cloud=&fields=fuzzstring&view=summary&assignedToGroup=null&groupIds=87da9fbdf0e746bbab6e8393578f8759 HTTP/1.1\r\nAccept: application/json\r\nHost: api-tk01.dev.central.company.com\r\n_OMITTED_AUTH_TOKEN_\r\nX-Tenant-ID: 566048da-ed19-4cd3-8e0a-b7e0e1ec4d72\r\nContent-Length: 0\r\nUser-Agent: restler/8.0.0\r\n\r\n
Warning: could not parse request in line 3799: GET /endpoint/v1/endpoints?pageFromKey=fuzzstring&pageSize=1&pageTotal=true&sort=fuzzstring&healthStatus=bad&type=computer&tamperProtectionEnabled=true&lockdownStatus=creatingWhitelist&lastSeenBefore=09/23/2019 12:02:01&lastSeenAfter=09/23/2019 12:02:01&ids=fuzzstring&isolationStatus=isolated&hostnameContains=fuzzstring&associatedPersonContains=fuzzstring&groupNameContains=fuzzstring&search=fuzzstring&searchFields=hostname&ipAddresses=fuzzstring&cloud=&fields=fuzzstring&view=full&assignedToGroup=true&groupIds=70c93fdf940d4653a7e09b4835996fc1 HTTP/1.1\r\nAccept: application/json\r\nHost: api-tk01.dev.central.company.com\r\n_OMITTED_AUTH_TOKEN_\r\nX-Tenant-ID: 566048da-ed19-4cd3-8e0a-b7e0e1ec4d72\r\nContent-Length: 0\r\nUser-Agent: restler/8.0.0\r\n\r\n
Warning: could not parse request in line 3917: GET /endpoint/v1/endpoints?pageFromKey=fuzzstring&pageSize=1&pageTotal=true&sort=fuzzstring&healthStatus=bad&type=computer&tamperProtectionEnabled=true&lockdownStatus=creatingWhitelist&lastSeenBefore=09/23/2019 12:02:01&lastSeenAfter=09/23/2019 12:02:01&ids=fuzzstring&isolationStatus=isolated&hostnameContains=fuzzstring&associatedPersonContains=fuzzstring&groupNameContains=fuzzstring&search=fuzzstring&searchFields=hostname&ipAddresses=fuzzstring&cloud=&fields=fuzzstring&view=full&assignedToGroup=void&groupIds=9fc4627fe52a4304be76403a0f497e7e HTTP/1.1\r\nAccept: application/json\r\nHost: api-tk01.dev.central.company.com\r\n_OMITTED_AUTH_TOKEN_\r\nX-Tenant-ID: 566048da-ed19-4cd3-8e0a-b7e0e1ec4d72\r\nContent-Length: 0\r\nUser-Agent: restler/8.0.0\r\n\r\n
Warning: could not parse request in line 4035: GET /endpoint/v1/endpoints?pageFromKey=fuzzstring&pageSize=1&pageTotal=true&sort=fuzzstring&healthStatus=bad&type=computer&tamperProtectionEnabled=true&lockdownStatus=creatingWhitelist&lastSeenBefore=09/23/2019 12:02:01&lastSeenAfter=09/23/2019 12:02:01&ids=fuzzstring&isolationStatus=isolated&hostnameContains=fuzzstring&associatedPersonContains=fuzzstring&groupNameContains=fuzzstring&search=fuzzstring&searchFields=hostname&ipAddresses=fuzzstring&cloud=&fields=fuzzstring&view=full&assignedToGroup=null&groupIds=cbdf41e89783444bab7d155b602d07fc HTTP/1.1\r\nAccept: application/json\r\nHost: api-tk01.dev.central.company.com\r\n_OMITTED_AUTH_TOKEN_\r\nX-Tenant-ID: 566048da-ed19-4cd3-8e0a-b7e0e1ec4d72\r\nContent-Length: 0\r\nUser-Agent: restler/8.0.0\r\n\r\n

Unhandled exception. System.Exception: Unexpected response without prior request at 10/06/2021 08:41:36: { version = "HTTP/1.1"
  statusCode = 400
  statusDescription = ""
  headers =
           map
             [("Connection", "keep-alive"); ("Content-Length", "226");
              ("Content-Type", "application/json");
              ("Date", "Wed, 06 Oct 2021 15:41:36 GMT");
              ("X-Correlation-ID", "75496058-61e4-4a83-a04e-b63381c1fbe8");
              ("X-Request-ID", "11555aca-f387-4ec1-86ee-4a11cc8f28f3")]
  body =
        "{
    "error": "badRequest",
    "correlationId": "75496058-61e4-4a83-a04e-b63381c1fbe8",
    "requestId": "11555aca-f387-4ec1-86ee-4a11cc8f28f3",
    "createdAt": "2021-10-06T15:41:36.345Z",
    "message": "Invalid request"
}"
  str = None }
   at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail@1639.Invoke(String message) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\printf.fs:line 1639
   at Microsoft.FSharp.Collections.Internal.IEnumerator.choose@171.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 179
   at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 77
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator`2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268
   at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 77
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64
   at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 77
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator`2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268
   at Microsoft.FSharp.Collections.Internal.IEnumerator.next@193[T](FSharpFunc`2 f, IEnumerator`1 e, FSharpRef`1 started, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 194
   at Microsoft.FSharp.Collections.Internal.IEnumerator.filter@188.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 196
   at Microsoft.FSharp.Collections.SeqModule.oneStepTo@987[T](IEnumerable`1 source, List`1 prefix, FSharpRef`1 enumeratorR, Int32 i) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 996
   at Microsoft.FSharp.Collections.SeqModule.action@4164-1[T](IEnumerable`1 source, List`1 prefix, FSharpRef`1 enumeratorR, Int32 i, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1011
   at Microsoft.FSharp.Collections.SeqModule.result@1003.Invoke(Int32 i)
   at Microsoft.FSharp.Collections.Internal.IEnumerator.unfold@205.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 207
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 64
   at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc`2 action, IEnumerable`1 source) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 496
   at Microsoft.FSharp.Collections.SeqModule.groupByValueType[T,TKey](FSharpFunc`2 keyf, IEnumerable`1 seq) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1068
   at Microsoft.FSharp.Collections.SeqModule.GroupBy@1076.Invoke(Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1076
   at Microsoft.FSharp.Collections.SeqModule.mkDelayedSeq@471.Invoke(Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 471
   at Restler.ResultsAnalyzer.Analyze.Buckets.getBuckets@98.GenerateNext(IEnumerable`1& next) in /usr/local/bin/restler-fuzzer/src/ResultsAnalyzer/Analyze/Buckets.fs:line 98
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.MoveNextImpl() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 371
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 403
   at Microsoft.FSharp.Collections.SeqModule.ToList[T](IEnumerable`1 source)
   at Restler.ResultsAnalyzer.Analyze.Main.main(AnalyzeArgs args) in /usr/local/bin/restler-fuzzer/src/ResultsAnalyzer/Analyze/Main.fs:line 85
   at Restler.ResultsAnalyzer.Program.main(String[] argv) in /usr/local/bin/restler-fuzzer/src/ResultsAnalyzer/Program.fs:line 223

Thanks.

mbiuki commented 2 years ago

Note, I tried this on 3 different platforms and I receive the same behavior after updating RESTler to the latest version. Please advise, thanks!

marina-p commented 2 years ago

Hi @mbiuki,

Results analyzer has not been updated in a while, however, it is possible this error appeared due to a change in your service's behavior, which triggered the bug because the logs are different. We will investigate. Are you using results analyzer output (the 'ResponseBuckets' directory)? If not, and you want to suppress this error, you can use the --no_results_analyzer switch until this is fixed.

mbiuki commented 2 years ago

Nope, at our front, I tested w/ Postman and it looks like our service is running bug-free. I can see the responses.

c7h commented 2 years ago

Hi @marina-p. I'm running into the same issue. I'm running restler on ubuntu 20.04. This is my stacktrace:

...
Unhandled exception. System.Exception: Unexpected response without prior request at 12/6/2021 7:59:26 PM: { version = "HTTP/1.1"
  statusCode = 200
  statusDescription = "OK"
  headers =
           map
             [("Connection", "Close"); ("Content-Length", "62");
              ("Content-Type", "application/json")]
  body = "{"nfInstances":[],"searchId":"227420","validityPeriod":100000}"
  str = None }
   at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail@1639.Invoke(String message) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\printf.fs:line 1639
   at Microsoft.FSharp.Collections.Internal.IEnumerator.choose@171.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 179
   at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 81
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 68
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator`2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268
   at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 81
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 68
   at Microsoft.FSharp.Collections.Internal.IEnumerator.map@75.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 81
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 68
   at Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers.takeInner@266[T,TResult](ConcatEnumerator`2 x, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 268
   at Microsoft.FSharp.Collections.Internal.IEnumerator.next@193[T](FSharpFunc`2 f, IEnumerator`1 e, FSharpRef`1 started, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 194
   at Microsoft.FSharp.Collections.SeqModule.oneStepTo@987[T](IEnumerable`1 source, List`1 prefix, FSharpRef`1 enumeratorR, Int32 i) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 996
   at Microsoft.FSharp.Collections.SeqModule.action@4164-1[T](IEnumerable`1 source, List`1 prefix, FSharpRef`1 enumeratorR, Int32 i, Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1008
   at Microsoft.FSharp.Collections.SeqModule.result@1003.Invoke(Int32 i)
   at Microsoft.FSharp.Collections.Internal.IEnumerator.unfold@205.DoMoveNext(b& curr) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 208
   at Microsoft.FSharp.Collections.Internal.IEnumerator.MapEnumerator`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 68
   at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc`2 action, IEnumerable`1 source) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 496
   at Microsoft.FSharp.Collections.SeqModule.groupByValueType[T,TKey](FSharpFunc`2 keyf, IEnumerable`1 seq) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1068
   at Microsoft.FSharp.Collections.SeqModule.GroupBy@1076.Invoke(Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 1076
   at Microsoft.FSharp.Collections.SeqModule.mkDelayedSeq@471.Invoke(Unit unitVar0) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 471
   at Restler.ResultsAnalyzer.Analyze.Buckets.getBuckets@98.GenerateNext(IEnumerable`1& next) in /home/c7h/workspace/restler-fuzzer/src/ResultsAnalyzer/Analyze/Buckets.fs:line 98
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.MoveNextImpl() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 371
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.System-Collections-IEnumerator-MoveNext() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 403
   at Microsoft.FSharp.Collections.SeqModule.ToList[T](IEnumerable`1 source)
   at Restler.ResultsAnalyzer.Analyze.Main.main(AnalyzeArgs args) in /home/c7h/workspace/restler-fuzzer/src/ResultsAnalyzer/Analyze/Main.fs:line 85
   at Restler.ResultsAnalyzer.Program.main(String[] argv) in /home/c7h/workspace/restler-fuzzer/src/ResultsAnalyzer/Program.fs:line 223

Does this help to understand the problem?