search

codingsoo / nlp2rest

11 stars 5 forks source link

Difficulty Replicating Artifact Results #1

Open kjain14 opened 1 year ago

kjain14 commented 1 year ago

Hi, was trying to replicate the artifact results on the countries enhanced specification and was running into the following set of issues:

Services:

**bboxrt:** 
- given the format of tool.sh, I assume that bboxrt expects a YAML file, however enhanced_spec.json is only a JSON file
- tried converting the OpenAPI spec from JSON to YAML using swagger-cli, however running bboxrt on the enhanced_specification.yaml yields the following error:

Error: java.lang.NullPointerException java.lang.NullPointerException at io.swagger.v3.parser.OpenAPIV3Parser.read(OpenAPIV3Parser.java:117) at io.swagger.v3.parser.OpenAPIV3Parser.read(OpenAPIV3Parser.java:105) at test.Load(test.java:19)


**evomaster:**
- seems to be working, specifying the SPEC_HERE, and TARGET_URL (assume this is the reverse proxy URL, ie. http://localhost:9007) parameters

**morest:**
- Running command: `python3 fuzzer.py /root/projects/nlp2rest/rule_validator/output/rest-countries-20230616193418850/enhanced_spec.json`
- Getting the error:
`prance.ValidationError: ("'3.1.0' does not match '^3\\\\.0\\\\.\\\\d(-.+)?$'", 'pattern', deque(['openapi']), None, [], '^3\\.0\\.\\d(-.+)?$', '3.1.0', {'type': 'string', 'pattern': '^3\\.0\\.\\d(-.+)?$'}, deque(['properties', 'openapi', 'pattern']), None)`

**restest:**
- changed api.properties to point to the spec: oas.path=/root/projects/nlp2rest/rule_validator/output/rest-countries-20230616193418850/enhanced_spec.json
- getting an error saying: src/test/resources/testConf.yaml does not exist, can this file be added to the repository?

**schemathesis:**
- seems to be working, with specifying SPEC_HERE and TARGET_URL
- however the initial checks seem to be failing
not_a_server_error                              356 / 356 passed          PASSED 
status_code_conformance                         0 / 356 passed            FAILED 
content_type_conformance                        356 / 356 passed          PASSED 
response_headers_conformance                    356 / 356 passed          PASSED 
response_schema_conformance                     356 / 356 passed          PASSED 
max_response_time                               356 / 356 passed          PASSED 

**tcases:**
- running tool.sh with the API spec file yields the following error

org.cornutum.tcases.openapi.reader.OpenApiReaderException: 2 conformance problem(s) found in Open API definition=file:/root/projects/nlp2rest/rule_validator/output/rest-countries-20230616193418850/enhanced_spec.json [0] attribute paths.'/v2/alpha'(get).parameters.[['unknown']].in is missing [1] attribute paths.'/v3/alpha'(get).parameters.[['unknown']].in is missing


**resttestgen:**
- modified rtg_config.json to point to enhanced_spec
- parsing seems to fail:

java.lang.NullPointerException at io.resttestgen.core.openapi.Operation.(Operation.java:123) at io.resttestgen.core.openapi.OpenAPIParser.parse(OpenAPIParser.java:116) at io.resttestgen.core.Environment.setUp(Environment.java:57) at io.resttestgen.core.cli.App.main(App.java:53)


**restler:**
- tried to compile to enhanced_spec to generate a RESTLER grammar
- got the following error:

Starting task Compile...

ERROR: Compiler failed. See logs in /root/projects/nlp2rest/tools/restler/Compile directory for more information.



For all of these fuzzers am using the following as API spec file and BASE_URL:
- spec file: /root/projects/nlp2rest/rule_validator/output/rest-countries-20230616193418850/enhanced_spec.json
- base url: http://localhost:9007
davidecorradini commented 1 year ago

Hello, could you please provide the enhanced specification so I can check its integrity? Thank you

kjain14 commented 1 year ago

Hi, here is the enhanced spec:

{
  "openapi": "3.1.0",
  "info": {
    "title": "REST countries API",
    "description": "REST countries API",
    "version": "v2.0.5"
  },
  "servers": [
    {
      "url": "http://localhost:9007"
    }
  ],
  "paths": {
    "/v2/lang/{lang}": {
      "get": {
        "operationId": "v2Lang",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "lang",
            "description": "Search by ISO 639-1 language code",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/lang/{lang}": {
      "get": {
        "operationId": "v3Lang",
        "parameters": [
          {
            "name": "lang",
            "description": "Search by language code or name",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          },
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/demonym/{demonym}": {
      "get": {
        "operationId": "v3Demonym",
        "parameters": [
          {
            "name": "demonym",
            "description": "Now you can search by how a citizen is called.",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          },
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v2/all": {
      "get": {
        "operationId": "v2All",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/translation/{translation}": {
      "get": {
        "operationId": "v3Translation",
        "parameters": [
          {
            "name": "translation",
            "description": "You can search by any translation name",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          },
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/subregion/{subregion}": {
      "get": {
        "operationId": "v3Subregion",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "subregion",
            "description": "Search by Subregion: South America, Southern Europe, Central America, Eastern Asia, etc. The search can be using the full subregion’s name or just part of it",
            "in": "path",
            "example": "South America",
            "schema": {
              "type": "string"
            },
            "required": true
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v2/callingcode/{callingcode}": {
      "get": {
        "operationId": "v2Callingcode",
        "parameters": [
          {
            "name": "callingcode",
            "description": "Search by calling code",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          },
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3.1/region/{region}": {
      "get": {
        "operationId": "v3Region",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "region",
            "description": "Search by Region: Africa, Americas, Asia, Europe, Oceania. The search can be using the full region’s name or just part of it",
            "in": "path",
            "example": "Americas",
            "schema": {
              "type": "string"
            },
            "required": true
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/name/{name}": {
      "get": {
        "operationId": "v3Name",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "name",
            "description": "Search by country name. It can be the common or official value",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          },
          {
            "name": "fullText",
            "in": "query",
            "schema": {
              "type": "boolean"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v2/alpha": {
      "get": {
        "operationId": "v2Alphacodes",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "schema": {
              "type": "array",
              "items": {
                "name": "codes",
                "description": "Search by list of ISO 3166-1 2-letter or 3-letter country codes",
                "in": "query",
                "schema": {
                  "type": "string"
                },
                "required": true
              }
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v2/regionalbloc/{regionalbloc}": {
      "get": {
        "operationId": "v2Regionalbloc",
        "parameters": [
          {
            "name": "regionalbloc",
            "description": "Search by regional bloc:\n  - EU (European Union)\n  - EFTA (European Free Trade Association)\n  - CARICOM (Caribbean Community)\n  - PA (Pacific Alliance)\n  - AU (African Union)\n  - USAN (Union of South American Nations)\n  - EEU (Eurasian Economic Union)\n  - AL (Arab League)\n  - ASEAN (Association of Southeast Asian Nations)\n  - CAIS (Central American Integration System)\n  - CEFTA (Central European Free Trade Agreement)\n  - NAFTA (North American Free Trade Agreement)\n  - SAARC (South Asian Association for Regional Cooperation)\n",
            "in": "path",
            "example": "CEFTA",
            "schema": {
              "type": "string"
            },
            "required": true
          },
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/alpha": {
      "get": {
        "operationId": "v3Alphacodes",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "schema": {
              "type": "array",
              "items": {
                "name": "codes",
                "description": "Search by list of ISO 3166-1 2-letter or 3-letter country codes",
                "in": "query",
                "schema": {
                  "type": "string"
                },
                "required": true
              }
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/capital/{capital}": {
      "get": {
        "operationId": "v3Capital",
        "parameters": [
          {
            "name": "capital",
            "description": "Search by capital city",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          },
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/alpha/{alphacode}": {
      "get": {
        "operationId": "v3Alphacode",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "alphacode",
            "description": "Search by ISO 3166-1 2-letter or 3-letter country code",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v2/alpha/{alphacode}": {
      "get": {
        "operationId": "v2Alphacode",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "alphacode",
            "description": "Search by ISO 3166-1 2-letter or 3-letter country code",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/all": {
      "get": {
        "operationId": "v3All",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v2/name/{name}": {
      "get": {
        "operationId": "v2Name",
        "parameters": [
          {
            "name": "name",
            "description": "Search by country name. It can be the native name or partial name",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          },
          {
            "name": "fullText",
            "in": "query",
            "schema": {
              "type": "boolean"
            }
          },
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v2/capital/{capital}": {
      "get": {
        "operationId": "v2Capital",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "capital",
            "description": "Search by capital city",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v3/currency/{currency}": {
      "get": {
        "operationId": "v3Currency",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "currency",
            "description": "Search by currency code or name",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v2/currency/{currency}": {
      "get": {
        "operationId": "v2Currency",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "currency",
            "description": "Search by ISO 4217 currency code",
            "in": "path",
            "schema": {
              "type": "string"
            },
            "required": true
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    },
    "/v2/region/{region}": {
      "get": {
        "operationId": "v2Region",
        "parameters": [
          {
            "name": "fields",
            "description": "You can filter the output of your request to include only the specified fields.",
            "in": "query",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "region",
            "description": "In version 2 regions are continent and subregions are region Search by continent: Africa, Americas, Asia, Europe, Oceania",
            "in": "path",
            "example": "Americas",
            "schema": {
              "type": "string"
            },
            "required": true
          }
        ],
        "responses": {},
        "x-dependencies": []
      }
    }
  }
}
codingsoo commented 1 year ago

Thank you for reporting the issues you've been experiencing in trying to replicate the artifact results on the enhanced specification. Your detailed feedback is greatly appreciated, and I've looked into the issues you've mentioned.

Regarding the Spotify API token issue, it seems that Spotify has recently changed the token access scope and overhauled their website, now providing REST API tokens with more limited access. We will engage with the Spotify team to see if there is a way to obtain "user access" scope so we can use the same access token as used in our paper.

As for the enhanced specification compatibility, we recently updated our enhanced specification version to 3.1.0 from 3.0.1, which appears to have generated some issues as certain tools do not support 3.1.0. To address this, I will provide a script to revert to 3.0.1 for the tools that are incompatible with the latest version.

Lastly, on the matter of the running REST API testing tool problem, we provided the testing tool running manual sourced from the developers. However, it seems more specific instructions are needed. I will add more detailed steps to use the tool. This should make it easier for users to utilize the tool.

Once all these issues have been addressed, I will respond to the issue. I appreciate your patience and cooperation as we work through these challenges.

kjain14 commented 1 year ago

A couple of other tweaks that would help with artifact:

  1. The setup.sh script does not wget the required JaCoCo jars in the services directory. This causes none of the local services to start.
  2. In run_service.py the genome-nexus line should be changed as follows:

Old line: "tmux new -d -s genome-nexus-server '. ./java8.env && java " + cov + "9002.exec" + " -jar ./services/genome-nexus/web/target/web-0-unknown-version-SNAPSHOT.war'" New line: "tmux new -d -s genome-nexus-server '. ./java8.env && java " + cov + "9002.exec" + " -jar ./genome-nexus/web/target/web-0-unknown-version-SNAPSHOT.war'"

  1. src/test/resources/testConf.yaml should be included for RESTest
  2. I think for OMDB I got blocked because the fuzzers send too many requests

With those tweaks more of the fuzzers seem to be working.

codingsoo commented 1 year ago

Thank you for your suggestions, they're greatly appreciated. I've addressed each one as follows:

  1. I've updated the setup.sh script to include the necessary wget command for fetching the required JaCoCo jars in the services directory.

  2. I've revised the run_service.py to reflect the correct path in the genome-nexus line as you suggested.

Regarding your other points:

  1. For generating testConf for RESTest, this process is outlined in our repository at "https://github.com/codingsoo/nlp2rest/tree/main/tools/restest". I agree that more detailed instructions would be beneficial, so I'll ensure that these are added soon.

  2. Regarding the OMDB blocking issue, it's possible that the number of requests sent by the fuzzers exceeded their rate limit. I'll investigate this matter further.

kjain14 commented 1 year ago

Sorry one more question. I don't see a way to get coverage for all APIs, specifically ohsome, fdic, language-tool, and rest-countries. The reverse proxy points to actual APIs rather than local ones, so wondering how we would get coverage for those (or is coverage just averaged over the other 4 APIs)? I don't jacoco.agent.runtime setup for these APIs specifically.

codingsoo commented 1 year ago

Thanks for your question. Local APIs, like Language Tool, OCVN, Genome-Nexus, and YouTube-Mock services are easier to measure since they reside on our own servers. Thus, we can set up and use tools like Jacoco to track the code execution paths during tests and determine coverage.

On the other hand, for remote services, coverage measurement is more challenging because the code isn't local to our environment and therefore cannot be directly analyzed. These services are hosted on their respective servers and our tests can only interact with them through the provided endpoints.

Regarding the OMDB issue you raised before, we are indeed using the paid version to circumvent the request limit. This allows us to run more extensive tests against this API (https://www.patreon.com/join/omdb).

I hope this helps! Let me know if you have any other questions.