search

microsoft / restler-fuzzer

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
MIT License
2.62k stars 303 forks source link

Compile Error with valid swagger api file #477

Open dexterinbk opened 2 years ago

dexterinbk commented 2 years ago

Hello:

I have installed restler on windows with dotnet 5.0 and python 3.8.

I am getting the following error whenever I try to run the command as specified in the documentation to compile the file. Does this indicate the wrong version of dotnet?

C:\Users\rwidham\Documents\GitHub\restler_bin\Compile>type StdErr.txt Unhandled exception. System.NotImplementedException: Objects in query parameters are not supported yet. at Restler.CodeGenerator.Python.formatNestedQueryParameter@301(RequestParameter requestParameter, String parameterName, NestedType propertyType, FSharpOption1 namePayloadSeq, IEnumerable1 innerProperties) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 310 at Restler.CodeGenerator.Python.generatePythonParameter@408-1.Invoke(Int32 level, InnerProperty p, IEnumerable1 innerProperties) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 408 at Restler.Grammar.Tree.processTree@34-1.Invoke(IEnumerable1 ts) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\Grammar.fs:line 35 at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr callersRetAddrSlot, IntPtr callTarget, IntPtr retVal) at Restler.CodeGenerator.Python.generatePythonParameter(Boolean includeOptionalParameters, ParameterKind parameterKind, RequestParameter requestParameter) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 136 at Restler.CodeGenerator.Python.parameters@490-2.Invoke(RequestParameter p) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 490 at Microsoft.FSharp.Primitives.Basics.List.map[T,TResult](FSharpFunc2 mapping, FSharpList1 x) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\local.fs:line 247 at Restler.CodeGenerator.Python.generatePythonFromRequestElement(Boolean includeOptionalParameters, RequestId requestId, RequestElement e) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 488 at Restler.CodeGenerator.Python.generatePythonFromRequest@754.Invoke(RequestElement requestElement) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 755 at Microsoft.FSharp.Primitives.Basics.List.mapToFreshConsTail[a,b](FSharpList1 cons, FSharpFunc2 f, FSharpList1 x) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\local.fs:line 241 at Microsoft.FSharp.Primitives.Basics.List.map[T,TResult](FSharpFunc2 mapping, FSharpList1 x) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\local.fs:line 247 at Restler.CodeGenerator.Python.generatePythonFromRequest(Request request, Boolean includeOptionalParameters, Boolean mergeStaticStrings) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 753 at Restler.CodeGenerator.Python.generatePythonRequest@1207(Boolean includeOptionalParameters, FSharpFunc2 formatRestlerPrimitive, Request request) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 1208 at Restler.CodeGenerator.Python.getRequests@1235.Invoke(Request r) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 1235 at Microsoft.FSharp.Primitives.Basics.List.mapToFreshConsTail[a,b](FSharpList1 cons, FSharpFunc2 f, FSharpList1 x) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\local.fs:line 241 at Microsoft.FSharp.Primitives.Basics.List.map[T,TResult](FSharpFunc2 mapping, FSharpList1 x) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\local.fs:line 247 at Restler.CodeGenerator.Python.generatePythonGrammar@1249.GenerateNext(IEnumerable1& next) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 1256 at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase1.MoveNextImpl() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seqcore.fs:line 371 at Microsoft.FSharp.Collections.SeqModule.ToList[T](IEnumerable1 source) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 546 at Restler.CodeGenerator.Python.generateCode(GrammarDefinition grammar, Boolean includeOptionalParameters, FSharpFunc2 write) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\CodeGenerator.fs:line 1281 at Restler.Workflow.generatePython(String grammarOutputDirectoryPath, Config config, GrammarDefinition grammar) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\Workflow.fs:line 27 at Restler.Workflow.generateRestlerGrammar(FSharpOption1 swaggerDoc, Config config) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.Compiler\Workflow.fs:line 276 at Program.main(String[] argv) in C:\Users\rwidham\Documents\GitHub\src\compiler\Restler.CompilerExe\Program.fs:line 37

Thank you!

marina-p commented 2 years ago

Hello @dexterinbk,

RESTler is not able to handle one of the constructs in the spec (objects in query parameters), but it should not be crashing in this case.

Depending on the spec, you may be able to work around this by making the particular query parameter a string instead of an object, compiling it, then adding a restler_custom_payload for the parameter to plug in the correctly typed value.

Would you be able to share a simplified snippet of the query parameter definition from the spec, to make sure we test it works correctly when fixing this issue?

Thanks,

Marina

painsAgains commented 2 years ago

Hello： when I compile my OpenAPI file I also got a compile . /home/target/Desktop/Restler/restler-fuzzer/restler_working_dir/Compile/:cat StrErr.txt

Unhandled exception. System.AggregateException: One or more errors occurred. (Unexpected initial token 'String' when populating object. Expected JSON object or array. Path 'definitions['io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps'].properties.__referencePath', line 1, position 3231994.)

---> Newtonsoft.Json.JsonSerializationException: Unexpected initial token 'String' when populating object. Expected JSON object or array. Path 'definitions['io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps'].properties.__referencePath', line 1, position 3231994. at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Populate(JsonReader reader, Object target) at Newtonsoft.Json.Serialization.JsonSerializerProxy.PopulateInternal(JsonReader reader, Object target) at Newtonsoft.Json.JsonSerializer.Populate(JsonReader reader, Object target) at NJsonSchema.ExtensionDataDeserializationConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Populate(JsonReader reader, Object target) at Newtonsoft.Json.Serialization.JsonSerializerProxy.PopulateInternal(JsonReader reader, Object target) at Newtonsoft.Json.JsonSerializer.Populate(JsonReader reader, Object target) at NJsonSchema.ExtensionDataDeserializationConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Populate(JsonReader reader, Object target) at Newtonsoft.Json.Serialization.JsonSerializerProxy.PopulateInternal(JsonReader reader, Object target) at Newtonsoft.Json.JsonSerializer.Populate(JsonReader reader, Object target) at NJsonSchema.ExtensionDataDeserializationConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) at NJsonSchema.Infrastructure.JsonSchemaSerialization.FromJson[T](String json, IContractResolver contractResolver) at NJsonSchema.Infrastructure.JsonSchemaSerialization.FromJsonAsync[T](String json, SchemaType schemaType, String documentPath, Func2 referenceResolverFactory, IContractResolver contractResolver) at NSwag.OpenApiDocument.FromJsonAsync(String data, String documentPath, SchemaType expectedSchemaType, Func2 referenceResolverFactory) at NSwag.OpenApiDocument.FromFileAsync(String filePath) --- End of inner exception stack trace --- at Microsoft.FSharp.Control.AsyncResult1.Commit() in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 349 at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronouslyInCurrentThread[a](CancellationToken cancellationToken, FSharpAsync1 computation) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\async.fs:line 882 at Microsoft.FSharp.Control.AsyncPrimitives.RunSynchronously[T](CancellationToken cancellationToken, FSharpAsync1 computation, FSharpOption1 timeout) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\async.fs:line 890 at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync1 computation, FSharpOption1 timeout, FSharpOption1 cancellationToken) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 1154 at Restler.Workflow.swaggerDocs@91-1.Invoke(String fp) in /home/target/Desktop/Restler/restler-fuzzer/src/compiler/Restler.Compiler/Workflow.fs:line 91 at Microsoft.FSharp.Primitives.Basics.List.map[T,TResult](FSharpFunc2 mapping, FSharpList1 x) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\local.fs:line 248 at Microsoft.FSharp.Collections.ListModule.Map[T,TResult](FSharpFunc2 mapping, FSharpList1 list) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\list.fs:line 75 at Restler.Workflow.generateGrammarFromSwagger(String grammarOutputDirectoryPath, FSharpOption1 swaggerDoc, FSharpOption1 specMetadata, Config config) in /home/target/Desktop/Restler/restler-fuzzer/src/compiler/Restler.Compiler/Workflow.fs:line 91 at Restler.Workflow.generateRestlerGrammar(FSharpOption1 swaggerDoc, Config config) in /home/target/Desktop/Restler/restler-fuzzer/src/compiler/Restler.Compiler/Workflow.fs:line 275 at Program.main(String[] argv) in /home/target/Desktop/Restler/restler-fuzzer/src/compiler/Restler.CompilerExe/Program.fs:line 37

But when I checked the path in my OpenAPI file ,I did not find the key:__referencePath in 'io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps'].properties.__referencePath".

My compile enviroment is dotnet 5.0 and python 3.8.

Really hope someone can help me

Thank you!

marina-p commented 2 years ago

Hello @painsAgains,

I will open a different issue for your error, since the error is different. Would it be possible for you to share a snippet of your openAPI spec to help reproduce the problem?

Thanks,

Marina

painsAgains commented 2 years ago

Thanks,this is my openapi spec,It may be a little complex.

------------------ 原始邮件 ------------------ 发件人: "microsoft/restler-fuzzer" @.>; 发送时间: 2022年6月8日(星期三) 中午12:31 @.>; @.**@.>; 主题: Re: [microsoft/restler-fuzzer] Compile Error with valid swagger api file (Issue #477)

Hello @painsAgains,

I will open a different issue for your error, since the error is different. Would it be possible for you to share a snippet of your openAPI spec to help reproduce the problem?

Thanks,

Marina

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

painsAgains commented 2 years ago

Hello,@marina-p,this is the api spec path in my StdErr.txt output

{
    "io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps":{
        "description":"JSONSchemaProps is a JSON-Schema following Specification Draft 4 (http://json-schema.org/).",
        "type":"object",
        "properties":{
            "$ref":{
                "type":"string"
            },
            "$schema":{
                "type":"string"
            },
            "additionalItems":{
                "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaPropsOrBool"
            },
            "additionalProperties":{
                "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaPropsOrBool"
            },
            "allOf":{
                "type":"array",
                "items":{
                    "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps"
                }
            },
            "anyOf":{
                "type":"array",
                "items":{
                    "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps"
                }
            },
            "default":{
                "description":"....",
                "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSON"
            },
            "definitions":{
                "type":"object",
                "additionalProperties":{
                    "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps"
                }
            },
            "dependencies":{
                "type":"object",
                "additionalProperties":{
                    "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaPropsOrStringArray"
                }
            },
            "description":{
                "type":"string"
            },
            "enum":{
                "type":"array",
                "items":{
                    "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSON"
                }
            },
            "example":{
                "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSON"
            },
            "exclusiveMaximum":{
                "type":"boolean"
            },
            "exclusiveMinimum":{
                "type":"boolean"
            },
            "externalDocs":{
                "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.ExternalDocumentation"
            },
            "format":{
                "description":"...",
                "type":"string"
            },
            "id":{
                "type":"string"
            },
            "items":{
                "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaPropsOrArray"
            },
            "maxItems":{
                "type":"integer",
                "format":"int64"
            },
            "maxLength":{
                "type":"integer",
                "format":"int64"
            },
            "maxProperties":{
                "type":"integer",
                "format":"int64"
            },
            "maximum":{
                "type":"number",
                "format":"double"
            },
            "minItems":{
                "type":"integer",
                "format":"int64"
            },
            "minLength":{
                "type":"integer",
                "format":"int64"
            },
            "minProperties":{
                "type":"integer",
                "format":"int64"
            },
            "minimum":{
                "type":"number",
                "format":"double"
            },
            "multipleOf":{
                "type":"number",
                "format":"double"
            },
            "not":{
                "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps"
            },
            "nullable":{
                "type":"boolean"
            },
            "oneOf":{
                "type":"array",
                "items":{
                    "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps"
                }
            },
            "pattern":{
                "type":"string"
            },
            "patternProperties":{
                "type":"object",
                "additionalProperties":{
                    "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps"
                }
            },
            "properties":{
                "type":"object",
                "additionalProperties":{
                    "$ref":"#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.JSONSchemaProps"
                }
            },
            "required":{
                "type":"array",
                "items":{
                    "type":"string"
                }
            },
            "title":{
                "type":"string"
            },
            "type":{
                "type":"string"
            },
            "uniqueItems":{
                "type":"boolean"
            },
            "x-kubernetes-embedded-resource":{
                "description":"...",
                "type":"boolean"
            },
            "x-kubernetes-int-or-string":{
                "description":"...",
                "type":"boolean"
            },
            "x-kubernetes-list-map-keys":{
                "description":"...",
                "type":"array",
                "items":{
                    "type":"string"
                }
            },
            "x-kubernetes-list-type":{
                "description":"...",
                "type":"string"
            },
            "x-kubernetes-map-type":{
                "description":"...",
                "type":"string"
            },
            "x-kubernetes-preserve-unknown-fields":{
                "description":"...",
                "type":"boolean"
            }
        }
    }
}