search

synopse / mORMot2

OpenSource RESTful ORM/SOA/MVC Framework for Delphi and FreePascal
https://synopse.info
Other
485 stars 122 forks source link

TSimpleClient can't get the correct response #256

Closed zen010101 closed 3 weeks ago

zen010101 commented 3 weeks ago

turn off the USELIBCURL switch, and with TSimpleHttpClient.Create(true), we get 404 error , here is the log:

20240604 05305145  +    marsBmcApi.TMarsBMCAPI(7fba375bc0).SendRequest
20240604 05305145 info          https://192.168.7.200:18080/login
20240604 05305145 debug         Request Header: ["Accept: application/json; charset=UTF-8"]
20240604 05305154 debug         Response: Code: 200, Body: {   "code": 200,   "data": {     "auth_token": "VV6idNT18YMcpthGlxux"   } }
20240604 05305154  -    00.135.061
20240604 05305154 debug Auth-Token: VV6idNT18YMcpthGlxux
20240604 05305154  +    marsBmcApi.TMarsBMCAPI(7fba375bc0).SendRequest
20240604 05305154 info          https://192.168.7.200:18080/arm_server/info
20240604 05305154 debug         Request Header: ["Accept: application/json; charset=UTF-8","Auth-Token: VV6idNT18YMcpthGlxux"]
20240604 05305154 debug         Response: Code: 404, Body:
20240604 05305154  -    00.001.326

turn on the USELIBCURL swith, and with TSimpleHttpClient.Create(), all thing are right! here is the log:

20240604 05242722  +    marsBmcApi.TMarsBMCAPI(7f87529b40).SendRequest
20240604 05242722 info          https://192.168.7.200:18080/login
20240604 05242722 debug         Request Header: ["Accept: application/json; charset=UTF-8"]
20240604 05242732 debug         Response: Code: 200, Body: {   "code": 200,   "data": {     "auth_token": "qJcuLbqOCuJ1qAGUpghI"   } }
20240604 05242732  -    00.152.034
20240604 05242732 debug Auth-Token: qJcuLbqOCuJ1qAGUpghI
20240604 05242732  +    marsBmcApi.TMarsBMCAPI(7f87529b40).SendRequest
20240604 05242732 info          https://192.168.7.200:18080/arm_server/info
20240604 05242732 debug         Request Header: ["Accept: application/json; charset=UTF-8","Auth-Token: qJcuLbqOCuJ1qAGUpghI"]
20240604 05242733 debug         Response: Code: 200, Body: {   "code": 0,   "content": {     "model": "JUPITER6",     "os_version": "1.0.0",     "sn": "JUPITER-0006"   },   "message": "succes" }
20240604 05242733  -    00.023.813
synopse commented 3 weeks ago

Which compiler are you using? On which Operating System?

zen010101 commented 3 weeks ago

Which compiler are you using? On which Operating System?

FPC 3.2.2 on Windows 11 Target : Linux aarch64

synopse commented 3 weeks ago

I am not able to reproduce it here.

Can you get what is the content returned by the server? (headers + response body stream)

Is really 404 returned by the server, or returned internally by the class?

Is there any 30x redirection involved?

Can you try not with TSimpleClient but with plain THttpClientSocket?

zen010101 commented 3 weeks ago

I am not able to reproduce it here.

Can you get what is the content returned by the server? (headers + response body stream) only headers can be recieved but no body conten:

20240604 13424119  +    marsBmcApi.TMarsBMCAPI(7faa849bc0).SendRequest
20240604 13424119 info          https://192.168.7.200:18080/login
20240604 13424119 debug         Request Headers: ["Accept: application/json; charset=UTF-8"]
20240604 13424126 debug         Response: Code: 200, Headers: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload  X-Frame-Options: DENY  Pragma: no-cache  Cache-Control: no-Store,no-Cache  X-XSS-Protection: 1; mode=block  X-Content-Type-Options: nosniff  Content-Security-Policy: default-src 'none'; img-src 'self' data:; font-src 'self'; style-src 'self'; script-src 'self'; connect-src 'self' wss:; form-action 'none'; frame-ancestors 'none'; plugin-types 'none'; base-uri 'none'  Date: Tue, 04 Jun 2024 13:42:41 GMT  , Body: {   "code": 200,   "data": {     "auth_token": "6I6B6DsYd9rRSVeSgBVP"   } }
20240604 13424126  -    00.119.394
20240604 13424126 debug Auth-Token: 6I6B6DsYd9rRSVeSgBVP
20240604 13424126  +    marsBmcApi.TMarsBMCAPI(7faa849bc0).SendRequest
20240604 13424126 info          https://192.168.7.200:18080/arm_server/info
20240604 13424126 debug         Request Headers: ["Accept: application/json; charset=UTF-8","Auth-Token: 6I6B6DsYd9rRSVeSgBVP"]
20240604 13424126 debug         Response: Code: 404, Headers: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload  X-Frame-Options: DENY  Pragma: no-cache  Cache-Control: no-Store,no-Cache  X-XSS-Protection: 1; mode=block  X-Content-Type-Options: nosniff  Content-Security-Policy: default-src 'none'; img-src 'self' data:; font-src 'self'; style-src 'self'; script-src 'self'; connect-src 'self' wss:; form-action 'none'; frame-ancestors 'none'; plugin-types 'none'; base-uri 'none'  Date: Tue, 04 Jun 2024 13:42:41 GMT  , Body:
20240604 13424126  -    00.001.556

Is really 404 returned by the server, or returned internally by the class? I am not sure. I wondered why the first api I called that Login can return the correct conten and the remains can't

Is there any 30x redirection involved? no

Can you try not with TSimpleClient but with plain THttpClientSocket? with TSimpleHttpClient.Create(true), it seems to use THttpClientSocket only . I am not sure

when I using proxy just like below all things works

constructor TMarsBMCAPI.Create(const BaseURL: string);
begin
  inherited Create;
  FHttpClient := TSimpleHttpClient.Create(True);
  FHttpClient.IgnoreTlsCertificateErrors := True;
  FHttpClient.Proxy := 'http://192.168.7.163:8888';   /// this points to the Fiddler to capture the data
  FBaseURL := BaseURL;
end;

But when I disabled the proxy , it not works from the second api ('/arm_server/info') .

I added some log to TSampleHttpClient.RawRequest, and I found it use fHttp to send https request:

20240604 14172308  +    marsBmcApi.TMarsBMCAPI(7f8cd75b80).SendRequest
20240604 14172308 info          https://192.168.7.200:18080/login
20240604 14172308 debug         Request Headers: ["Accept: application/json; charset=UTF-8"]
20240604 14172308  +            mormot.net.client.TSimpleHttpClient(7f8ce353a0).RawRequest
20240604 14172308 debug                 Uri.Address: login, Method: POST, Header: Accept: application/json; charset=UTF-8 , Data: {"user":"root","password":"marsserver.cm"}, DataMineType: application/json; charset=UTF-8
20240604 14172316 debug                 fHttp: {"THttpClientSocket(7f8ce19e10)":{SocketFamily:"nfIP4",Server:"192.168.7.200",Port:"18080",RawSocket:4,TimeOut:20000,BytesIn:1250,BytesOut:324,UserAgent:"Mozilla/5.0 (Linux aarch64; mORMot) HCS/2.2 test",ContentLength:75,ContentType:"application/json"}}, Uri: login, Result: 200, fHeaders: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload  X-Frame-Options: DENY  Pragma: no-cache  Cache-Control: no-Store,no-Cache  X-XSS-Protection: 1; mode=block  X-Content-Type-Options: nosniff  Content-Security-Policy: default-src 'none'; img-src 'self' data:; font-src 'self'; style-src 'self'; script-src 'self'; connect-src 'self' wss:; form-action 'none'; frame-ancestors 'none'; plugin-types 'none'; base-uri 'none'  Date: Tue, 04 Jun 2024 14:17:23 GMT  , fBody: {   "code": 200,   "data": {     "auth_token": "gs5bYYDYDTyuVLp8o4HA"   } }
20240604 14172316  -            00.129.678
20240604 14172316 debug         Response: Code: 200, Headers: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload  X-Frame-Options: DENY  Pragma: no-cache  Cache-Control: no-Store,no-Cache  X-XSS-Protection: 1; mode=block  X-Content-Type-Options: nosniff  Content-Security-Policy: default-src 'none'; img-src 'self' data:; font-src 'self'; style-src 'self'; script-src 'self'; connect-src 'self' wss:; form-action 'none'; frame-ancestors 'none'; plugin-types 'none'; base-uri 'none'  Date: Tue, 04 Jun 2024 14:17:23 GMT  , Body: {   "code": 200,   "data": {     "auth_token": "gs5bYYDYDTyuVLp8o4HA"   } }
20240604 14172316  -    00.129.808
20240604 14172316 debug Auth-Token: gs5bYYDYDTyuVLp8o4HA
20240604 14172316  +    marsBmcApi.TMarsBMCAPI(7f8cd75b80).SendRequest
20240604 14172316 info          https://192.168.7.200:18080/arm_server/info
20240604 14172316 debug         Request Headers: ["Accept: application/json; charset=UTF-8","Auth-Token: gs5bYYDYDTyuVLp8o4HA"]
20240604 14172316  +            mormot.net.client.TSimpleHttpClient(7f8ce353a0).RawRequest
20240604 14172316 debug                 Uri.Address: arm_server/info, Method: GET, Header: Accept: application/json; charset=UTF-8 Auth-Token: gs5bYYDYDTyuVLp8o4HA , Data: , DataMineType: application/json; charset=UTF-8
20240604 14172316 debug                 fHttp: {"THttpClientSocket(7f8ce19e10)":{SocketFamily:"nfIP4",Server:"192.168.7.200",Port:"18080",RawSocket:-1,TimeOut:20000,BytesIn:1250,BytesOut:581,UserAgent:"Mozilla/5.0 (Linux aarch64; mORMot) HCS/2.2 test",ContentLength:75,ContentType:"application/json"}}, Uri: arm_server/info, Result: 404, fHeaders: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload  X-Frame-Options: DENY  Pragma: no-cache  Cache-Control: no-Store,no-Cache  X-XSS-Protection: 1; mode=block  X-Content-Type-Options: nosniff  Content-Security-Policy: default-src 'none'; img-src 'self' data:; font-src 'self'; style-src 'self'; script-src 'self'; connect-src 'self' wss:; form-action 'none'; frame-ancestors 'none'; plugin-types 'none'; base-uri 'none'  Date: Tue, 04 Jun 2024 14:17:23 GMT  , fBody:
20240604 14172316  -            00.001.433
20240604 14172316 debug         Response: Code: 404, Headers: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload  X-Frame-Options: DENY  Pragma: no-cache  Cache-Control: no-Store,no-Cache  X-XSS-Protection: 1; mode=block  X-Content-Type-Options: nosniff  Content-Security-Policy: default-src 'none'; img-src 'self' data:; font-src 'self'; style-src 'self'; script-src 'self'; connect-src 'self' wss:; form-action 'none'; frame-ancestors 'none'; plugin-types 'none'; base-uri 'none'  Date: Tue, 04 Jun 2024 14:17:23 GMT  , Body:
20240604 14172316  -    00.001.509
zen010101 commented 3 weeks ago

I think the bug is TLS related.

with TSimpleHttpClient.Create(True), it will raise exception :

      raise ENetSock.Create('%s.DoTlsAfter: TLS support not compiled ' +
        '- try including mormot.lib.openssl11 in your project',

when I added mormot.lib.openssl11 in my project, no exception raised but the response is soemthing wrong.

zen010101 commented 3 weeks ago

the workaround for it is turn on USELIBCURL swtich and using TSimpleHttpClient.Create(False) to tell the mormot to use curl lib for https request.

synopse commented 3 weeks ago

What if you free and re-create a TSimpleHttpClient instance for the 2nd call to https://192.168.7.200:18080/arm_server/info ?

How is implemented the server?

What is the exact URI value sent at RawRequest on THttpClientSocket ? (perhaps a missing / at the beginning?)

zen010101 commented 3 weeks ago

What if you free and re-create a TSimpleHttpClient instance for the 2nd call to https://192.168.7.200:18080/arm_server/info ?

How is implemented the server?

What is the exact URI value sent at RawRequest on THttpClientSocket ? (perhaps a missing / at the beginning?)

when I skip login and call "/arm_server/info" directly, I got the correct unauthorized body, it is correct.

but when I pass the correct auth-token returned by login to /arm_server/info call， not work again, it returns EMPTY body.

synopse commented 3 weeks ago

Please answer the remaining questions:

How is implemented the server? What is the exact URI value sent at RawRequest on THttpClientSocket ? (perhaps a missing / at the beginning?)

zen010101 commented 3 weeks ago
  1. server is not written by me , but just return a auth-token when called login and all other calling must take it in the header.
  2. the exact URI value is ok and is the same as called by Curl Lib. You can see the log: 20240606 11590531 ! debug THttpClientSocket.Open, Uri.server: 192.168.7.200, Uri.Port: 18080, Uri.Https: 1, Uri.Address: arm_server/info
20240606 11590523  !  +    marsBmcApi.TMarsBMCAPI(7fa1a536a0).SendRequest
20240606 11590523  ! http       https://192.168.7.200:18080/login
20240606 11590523  ! http       Request Headers: ["Accept: application/json; charset=UTF-8"]
20240606 11590523  ! debug      THttpClientSocket.Open, Uri.server: 192.168.7.200, Uri.Port: 18080, Uri.Https: 1, Uri.Address: login
20240606 11590531  ! http       Response: Code: 200, Headers: Strict-Transport-Security: max-age=31536000; includeSubdomains; preload  X-Frame-Options: DENY  Pragma: no-cache  Cac>
20240606 11590531  !  -    00.126.576
20240606 11590531  ! debug Auth-Token: BCnlQZROc18eJYch2zpl
20240606 11590531  !  +    marsBmcApi.TMarsBMCAPI(7fa1a536a0).SendRequest
20240606 11590531  ! http       https://192.168.7.200:18080/arm_server/info
20240606 11590531  ! http       Request Headers: ["Accept: application/json; charset=UTF-8","Auth-Token: BCnlQZROc18eJYch2zpl"]
20240606 11590531  ! debug      THttpClientSocket.Open, Uri.server: 192.168.7.200, Uri.Port: 18080, Uri.Https: 1, Uri.Address: arm_server/info
20240606 11590533  ! http       Response: Code: 404, Headers: , Body:
20240606 11590533  !  -    00.032.879
zen010101 commented 3 weeks ago

Here is the test code:


    api := TMarsBMCAPI.Create('https://192.168.7.200:18080');
    api.Login('root', 'marsserver.cm');
    key := api.AuthToken;
    api.Free;

    api := TMarsBMCAPI.Create('https://192.168.7.200:18080');
    api.AuthToken := key;
    try
      //api.Login('root', 'marsserver.cm');
      add(api.GetServerInfo, 'GetServerInfo');
      add(api.GetServerMonitorInfo, 'GetServerMonitorInfo');
      add(api.GetServerPsuInfo, 'GetServerPsuInfo');
      add(api.GetServerFanInfo, 'GetServerFanInfo');
      add(api.GetNodeSummary('0'), 'GetNodeSummary');
      add(api.GetAllNodeStatus, 'GetAllNodeStatus');
      add(api.GetAllNodeInfo, 'GetAllNodeInfo');
      add(api.GetAllNodeNetworkInfo, 'GetAllNodeNetworkInfo');
      add(api.GetAllNodeMonitorInfo, 'GetAllNodeMonitorInfo');
      add(api.GetAllNodeInfoV2, 'GetAllNodeInfoV2');
      add(api.GetAllNodePortInfo, 'GetAllNodePortInfo');
      writeln('press [enter] to quit');
      readln;
    finally
      api.Free;
    end;
synopse commented 3 weeks ago

Please debug a little and try to check about my question:

What is the exact URI value sent at RawRequest on THttpClientSocket ? (perhaps a missing / at the beginning?)

zen010101 commented 3 weeks ago

I think it already insert a '/' at the beginning in the code Request:

function THttpClientSocket.Request(const url, method: RawUtf8;
  KeepAlive: cardinal; const Header: RawUtf8; const Data: RawByteString;
  const DataMimeType: RawUtf8; retry: boolean; InStream, OutStream: TStream): integer;
var
  ctxt: THttpClientRequest;
  newuri: TUri;
  log: ISynLog;
begin
  log := TSynLog.Add.Enter(Self, 'Request');
  ctxt.Url := url;
  if (url = '') or
     (url[1] <> '/') then
    insert('/', ctxt.Url, 1); // normalize URI as in RFC (e.g. for Digest auth)

Here is more output in the log:

https://i.imgur.com/6p2rx5R.png

the log code is here:

https://i.imgur.com/hiJNYBI.png

zen010101 commented 3 weeks ago

SOLVED: at last, I found the reason that I add a redundant header into the request:

  Uri := FBaseUrl + Endpoint;
  Headers := TStringList.Create;

  Headers.Add('Accept: ' + mimetype);  // HERE is redundant, when delete this line, everything is working properly

  if FAuthToken <> '' then
    Headers.Add('Auth-Token: ' + FAuthToken);
  Safe.Lock;
  try
    case Method of
      'GET':
        Response := FHttpClient.Request(Uri, 'GET', Headers.Text, Params, mimetype);
      'POST':
        Response := FHttpClient.Request(Uri, 'POST', Headers.Text, Params, mimetype);
      'PUT':
        Response := FHttpClient.Request(Uri, 'PUT', Headers.Text, Params, mimetype);
      'DELETE':
        Response := FHttpClient.Request(Uri, 'DELETE', Headers.Text, Params, mimetype);
    end;
  finally
    Safe.UnLock;
    Headers.Free;
  end;
synopse commented 3 weeks ago

My guess is more that you should not use TSTRingList for headers, because on POSIX I suspect it uses LF for line feeds, instead of CR+LF as expected by HTTP.

zen010101 commented 3 weeks ago

My guess is more that you should not use TSTRingList for headers, because on POSIX I suspect it uses LF for line feeds, instead of CR+LF as expected by HTTP.

So, I should use RawUtf8 and manually add LF for every line ?

zen010101 commented 3 weeks ago

When adding something to headers using TStringList.Add, the request fails. I'm wondering why even when I set Header.LineBreak to #10, it still doesn't work. I'm also curious why the curl library works fine.

synopse commented 3 weeks ago

I guess libcurl is re-formatting the headers content.

The LineBreak should be #13#10.

But you can call AppendLine() from mormot.core.text.pas to generate the HTTP headers directly in a RawUtf8 variable.

zen010101 commented 3 weeks ago

You are right! It works when I use #13#10 as TStringList.LineBreak . It is a better way that use AppendLine to headers RawUtf8 varaible.