search

apimastery / APISimulator

API Simulator - configuration-driven tool for modeling and running of API simulations
https://apisimulator.io
3 stars 1 forks source link

The first data line in a CSV is not matched/retrieved (data provided through simlet's yaml configuration) #21

Open felipetofoli opened 7 months ago

felipetofoli commented 7 months ago

Hi! First of all, APISimulator is a very interesting project and I am checking its features to see if I can integrate it into a new project that I am part of!

Issue

I am having an issue when using the CSV data provided in the simlet's YAML configuration: I can't match/retrieve the first (data) line of the CSV. It seems the first data line is ignored when matching the results.

Steps to reproduce

  1. I am using a docker-compose that looks like the following: 
    
version: "3.9"

services: csv-apisimulator: container_name: "csv-apisimulator" image: apimastery/apisimulator:1.11 ports:

  1. The templates/csv/apisim.yaml looks like the following:

Note: The file is a copy + paste from the content (zip file) downloaded from the apisimulator website: apisimulator-http-1.11-distro\apisimulator\apisimulator-http-1.11\examples\parameters-from-csv\simlets\read-product-from-config\simlet.yaml The only change I made was adding the line: simlet: read-products-csv

#----------------------------------------------------------------------
# Simlet-specific configuration.
#----------------------------------------------------------------------
simlet: read-products-csv

ProductID:
  is: parameter
  from: uriPathPattern
  pattern: "/v1.2/product/{ProductID}/**"

# Parameter names must match the name of the columns in the CSV data
ProductColor:
  is: parameter
  from: uriPathPattern
  pattern: "/v1.2/product/*/{Color}/*"

# Parameter names must match the name of the columns in the CSV data
ProductSize:
  is: parameter
  from: uriPathPattern
  pattern: "/v1.2/product/*/*/{Size}"

ProductData:
  is: parameter
  from: csv

  # The first row in the CSV file is expected to contain unique column names
  # with no spaces or special characters in the names other than underscore.
  # A column value that contains a comma or double quote(s) is to be surrounded 
  # by double quotes. Each double quote inside of a column value is to be escaped
  # with another double quote. This is what file export to CSV from MS Excel does
  data: `
ProductID,ProductCategory,ProductSubcategory,ProductName,ProductColor,ProductSize,ProductSKU,Original_Price,Sale_Price
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Black Charcoal,S,769409213,55.00,39.99
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Black Charcoal,M,769413463,55.00,39.99
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Black Charcoal,L,769413609,55.00,39.99
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Black Charcoal,XL,769413531,55.00,39.99
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Black Charcoal,XXL,769413579,55.00,39.99
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Solid Gray,S,769417959,55.00,35.95
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Solid Gray,M,769417966,55.00,36.96
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Solid Gray,L,769413623,55.00,37.97
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Solid Gray,XL,769413692,55.00,38.98
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Solid Gray,XXL,769413678,55.00,39.99
2592228,Men,Hoodies & Sweatshirts,Full-Zip Hooded Fleece Jacket,Dark Gray,S,769458440,65.00,52.00
2592228,Men,Hoodies & Sweatshirts,Full-Zip Hooded Fleece Jacket,Dark Gray,M,769458501,65.00,52.00
2592228,Men,Hoodies & Sweatshirts,Full-Zip Hooded Fleece Jacket,Dark Gray,L,769458327,65.00,52.00
2592228,Men,Hoodies & Sweatshirts,Full-Zip Hooded Fleece Jacket,Dark Gray,XL,769458303,65.00,52.00
2592228,Men,Hoodies & Sweatshirts,Full-Zip Hooded Fleece Jacket,Dark Gray,XXL,769458426,65.00,52.00
6218877,Children,Shoes,The "Rockets",White,4,669032116,35.59,31.99
`

  # List of keys from parameters to use in querying the CSV data. 
  # A parameter as key may appear more than once in the list.
  # Parameter names must match the column names in the CSV data
  keys:
  - ProductID
  - ProductColor
  - ProductSize

request:
- method: GET

- where: uriPathPattern
  matches: "/v1.2/product/{id}/{color}/{size}"

- where: parameter
  named: ProductData
  exists: true

response:
  from: template
  template: Simula

  status: 200

  headers:
  - "Content-Type: application/json; charset=UTF-8"

  body: `
{
  "product": {
    "id": "${ ProductData['ProductID'] }",
    "sku": "${ ProductData['ProductSKU'] }",
    "name": "${  jsonEncode( ProductData['ProductName'] ) }",
    "category": "${ ProductData['ProductCategory'] }",
    "subCategory": "${ ProductData['ProductSubcategory'] }",
    "color": "${ ProductData['ProductColor'] }",
    "size": "${ ProductData['ProductSize'] }",
    "orig_price": ${ ProductData['Original_Price'] },
    "sale_price": ${ ProductData['Sale_Price'] }
  }
}
`

  1. Run docker-compose command:

    docker-compose up

  2. Call the endpoint:

    4.1. Checking for the product in the first line - Result: 404 Not Found

curl:

curl --location 'http://localhost:6090/v1.2/product/2706414/Black Charcoal/S'

HTTP Request and Response

GET /v1.2/product/2706414/Black%20Charcoal/S HTTP/1.1
User-Agent: PostmanRuntime/7.35.0
Accept: */*
Cache-Control: no-cache
Postman-Token: [REDACTED]
Host: localhost:6090
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

HTTP/1.1 404 Not Found
Content-Type: application/text; charset=UTF-8
Server: API Simulator, v1.11
Date: Mon, 27 Nov 2023 14:29:09 GMT
Content-Length: 62

API Simulator couldn't find a matching simlet for the request.

4.2 Checking for the product in the second line - Result: 200 OK curl:

curl --location 'http://localhost:6090/v1.2/product/2706414/Black Charcoal/M'

HTTP Request and Response:

GET /v1.2/product/2706414/Black%20Charcoal/M HTTP/1.1
User-Agent: PostmanRuntime/7.35.0
Accept: */*
Cache-Control: no-cache
Postman-Token: [REDACTED]
Host: localhost:6090
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Server: API Simulator, v1.11
Date: Mon, 27 Nov 2023 14:31:36 GMT
Content-Length: 270

{
"product": {
"id": "2706414",
"sku": "769413463",
"name": "Fleece Pullover Hoodie",
"category": "Men",
"subCategory": "Hoodies & Sweatshirts",
"color": "Black Charcoal",
"size": "M",
"orig_price": 55.00,
"sale_price": 39.99
}
}

Observation

If I change the places of the "S" product to the second line, and the "M" product to the first line, then the "M" (medium) product is the one that is not matched:

ProductID,ProductCategory,ProductSubcategory,ProductName,ProductColor,ProductSize,ProductSKU,Original_Price,Sale_Price
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Black Charcoal,M,769413463,55.00,39.99
2706414,Men,Hoodies & Sweatshirts,Fleece Pullover Hoodie,Black Charcoal,S,769409213,55.00,39.99

So, the behavior seems tied to the first data line of the csv configuration.

Logs

2023-11-27T14:27:50.284+0000|INFO|10|main|[+--------------------------+]
2023-11-27T14:27:50.285+0000|INFO|10|main|[| A P I  S i m u l a t o r |]
2023-11-27T14:27:50.285+0000|INFO|10|main|[+--------------------------+]
2023-11-27T14:27:50.291+0000|INFO|10|main|[API Simulator initialization in progress...]
2023-11-27T14:27:50.291+0000|INFO|10|main|[API Simulator: {java version:"11.0.19+7", jvm name:"OpenJDK 64-Bit Server VM", jvm version:"11.0.19+7", jvm pid:10}]
2023-11-27T14:27:50.291+0000|INFO|10|main|[API Simulator: implementation {title:"API Simulator",version:"1.11"}]
2023-11-27T14:27:50.291+0000|INFO|10|main|[API Simulator: args=[-c, /apisimulator/apisimulator-http-1.11/config, -s, /templates/simlets, /templates]]
2023-11-27T14:27:50.305+0000|INFO|10|main|[Loading configuration from '/apisimulator/apisimulator-http-1.11/config/apisim.yaml'...]
2023-11-27T14:27:50.338+0000|INFO|10|main|[Loaded admin configuration]
2023-11-27T14:27:50.342+0000|INFO|10|main|[Loaded simulator configuration]
2023-11-27T14:27:50.346+0000|INFO|10|main|[Loaded simulation configuration]
2023-11-27T14:27:50.374+0000|INFO|10|main|[Loaded 2 simlets in 69 ms]
2023-11-27T14:27:50.376+0000|INFO|10|main|[Loading configuration from '/templates/apisim.yaml'...]
2023-11-27T14:27:51.410+0000|INFO|10|main|[Loaded 1 simlet in 1034 ms]
2023-11-27T14:27:51.411+0000|INFO|10|main|[Loading simlets from '/templates/simlets'...]
2023-11-27T14:27:51.413+0000|INFO|10|main|[Loaded 0 simlets in 1 ms]
2023-11-27T14:27:51.783+0000|INFO|10|API-Simulator-Boss-1-1|[API Simulator Server is listening on 0.0.0.0:6090]
2023-11-27T14:27:51.783+0000|INFO|10|main|[API Simulator started]
2023-11-27T14:27:51.802+0000|INFO|10|AdminServer-Boss-6-1|[Admin HTTP Server is listening on 0.0.0.0:6190]

Please let me know if any other information is needed! Thanks!

apisim commented 7 months ago

Thank you, @felipetofoli, for the detailed information! I wished each bug report were like this 👍

A workaround is to place the CSV data in an external file.

This is a confirmed bug. The fix will be included in the upcoming version 1.12 along with a few others queued up for a release ("Cyrillic characters in a template may cause incorrect response output", "fixed log4j2 config for API Recorder", "upgraded Netty to fix a vulnerability in the HTTP/2 protocol known as "HTTP/2 Rapid Reset""...).

Thanks!

felipetofoli commented 7 months ago

@apisim Thank you for the response! I will try the workaround while I wait for the fix!

Thanks!