manticoresoftware / manticoresearch

Easy to use open source fast database for search | Good alternative to Elasticsearch now | Drop-in replacement for E in the ELK soon
https://manticoresearch.com
GNU General Public License v3.0
9.08k stars 510 forks source link

Log http requests / responses for development / QA purposes #2711

Open sanikolaev opened 3 weeks ago

sanikolaev commented 3 weeks ago

Proposal:

We want to log all HTTP requests and responses for development and QA purposes. Here's how it should work:

  1. New setting log_http = /path/to/file in the searchd section. This should remain undocumented for now.
  2. If this requires making Manticore slower due to additional locks, that's okay.
  3. Log each request and response on separate lines.
  4. Include an ID to link each request with its corresponding response.
  5. Add timestamps with millisecond precision.
  6. Log full raw HTTP requests and responses.
[2023-10-31 14:23:45.123] [Request-ID: 12345] - Incoming HTTP Request:
═══════════════════════════════════════════════════════════════════════
POST /api/search HTTP/1.1
Host: manticoresearch.local
Content-Type: application/json
User-Agent: ManticoreClient/1.0
Content-Length: 75

{
    "query": {
        "match": {
            "field": "example"
        }
    },
    "size": 10
}

[2023-10-31 14:23:45.678] [Request-ID: 12345] - Outgoing HTTP Response:
═══════════════════════════════════════════════════════════════════════
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 135

{
    "took": 5,
    "hits": {
        "total": 1,
        "hits": [
            {
                "_id": "1",
                "_score": 1.0,
                "_source": {
                    "field": "example"
                }
            }
        ]
    }
}

Checklist:

To be completed by the assignee. Check off tasks that have been completed or are not applicable.

- [ ] Implementation completed - [ ] Tests developed - [ ] Documentation updated - [x] Documentation reviewed - [ ] [Changelog](https://docs.google.com/spreadsheets/d/1mz_3dRWKs86FjRF7EIZUziUDK_2Hvhd97G0pLpxo05s/edit?pli=1&gid=1102439133#gid=1102439133) updated - [x] OpenAPI YAML updated and issue created to rebuild clients