Websocket not connected to server in native

SviatoslavKomkov commented 4 years ago

#include <drogon/WebSocketClient.h>
#include <drogon/WebSocketController.h>
#include <drogon/HttpAppFramework.h>
#include <trantor/net/EventLoopThread.h>

#include <iostream>

using namespace drogon;
using namespace std::chrono_literals;

class WebSocketTest : public drogon::WebSocketController<WebSocketTest> {
public:
    virtual void handleNewMessage(const WebSocketConnectionPtr &,
                                  std::string &&,
                                  const WebSocketMessageType &) override;

    virtual void handleConnectionClosed(
            const WebSocketConnectionPtr &) override;

    virtual void handleNewConnection(const HttpRequestPtr &,
                                     const WebSocketConnectionPtr &) override;

    WS_PATH_LIST_BEGIN
    WS_PATH_ADD("/chat", "drogon::LocalHostFilter", Get);
    WS_PATH_LIST_END
};

void WebSocketTest::handleNewMessage(const WebSocketConnectionPtr &wsConnPtr,
                                     std::string &&message,
                                     const WebSocketMessageType &type) {
    // write your application logic here
    LOG_DEBUG << "new websocket message:" << message;
    if (type == WebSocketMessageType::Ping) {
        LOG_DEBUG << "recv a ping";
    }
}

void WebSocketTest::handleConnectionClosed(const WebSocketConnectionPtr &) {
    LOG_DEBUG << "websocket closed!";
}

void WebSocketTest::handleNewConnection(const HttpRequestPtr &,
                                        const WebSocketConnectionPtr &conn) {
    LOG_DEBUG << "new websocket connection!";
    conn->send("haha!!!");
}

int main(int argc, char *argv[]) {

    std::string host("0.0.0.0");
    int port(8848);
    app()
    .setLogLevel(trantor::Logger::kDebug)
    .addListener(host, port);

    auto wsPtr = WebSocketClient::newWebSocketClient(host, port);
    auto req = HttpRequest::newHttpRequest();
    req->setPath("/chat");
    wsPtr->setMessageHandler([](const std::string &message,
                                           const WebSocketClientPtr &wsPtr,
                                           const WebSocketMessageType &type) {
        std::cout << "new message:" << message << std::endl;
        if (type == WebSocketMessageType::Pong) {
            std::cout << "recv a pong" << std::endl;
        }
    });
    wsPtr->setConnectionClosedHandler([](const WebSocketClientPtr &wsPtr) {
        std::cout << "ws closed!" << std::endl;
    });
    wsPtr->connectToServer(req,
                           [](ReqResult r,
                                         const HttpResponsePtr &resp,
                                         const WebSocketClientPtr &wsPtr) {
                               if (r == ReqResult::Ok) {
                                   std::cout << "ws connected!" << std::endl;
                                   wsPtr->getConnection()->setPingMessage("",
                                                                          2s);
                                   wsPtr->getConnection()->send("hello!");
                               } else {
                                   std::cout << "ws failed!" << std::endl;
                               }
                           });
    app().run();
}

Output:

ws failed!

But with:

<!DOCTYPE html>
<pre id="log"></pre>
<script>
  // helper function: log message to screen
  function log(msg) {
    document.getElementById('log').textContent += msg + '\n';
  }

  // setup websocket with callbacks
  var ws = new WebSocket("ws://0.0.0.0:8848/chat");
  ws.onopen = function() {
    log('CONNECT');
    ws.send("hello!!!");
  };
  ws.onclose = function() {
    log('DISCONNECT');
  };
  ws.onmessage = function(event) {
    log('MESSAGE: ' + event.data);
    ws.send(event.data);
    ws.send(event.data);
  };
</script>

Works totaly fine;

CONNECT
MESSAGE: haha!!!

Please explain this behavior.

Regards.

an-tao commented 4 years ago

You can't connect to a server with IP '0.0.0.0', please use 127.0.0.1 instead.

auto wsPtr = WebSocketClient::newWebSocketClient("127.0.0.1", port);

SviatoslavKomkov commented 4 years ago

Oh thank you.

Here is new unclear situation.

#include <drogon/WebSocketController.h>
#include <drogon/HttpAppFramework.h>
#include <iostream>

#include "WsInit.h"

using namespace drogon;
using namespace std::chrono_literals;

class WebSocketTest : public drogon::WebSocketController<WebSocketTest, false> {
public:
    virtual void handleNewMessage(const WebSocketConnectionPtr &,
                                  std::string &&,
                                  const WebSocketMessageType &) override;

    virtual void handleConnectionClosed(
            const WebSocketConnectionPtr &) override;

    virtual void handleNewConnection(const HttpRequestPtr &,
                                     const WebSocketConnectionPtr &) override;

    std::vector<WebSocketConnectionPtr> connList;
    void Connections() {
        LOG_DEBUG << "Connections from function: " << connList.size();
    }
    WS_PATH_LIST_BEGIN
    WS_PATH_ADD("/chat", "drogon::LocalHostFilter", Get);
    WS_PATH_LIST_END
};

void WebSocketTest::handleNewMessage(const WebSocketConnectionPtr &wsConnPtr,
                                     std::string &&message,
                                     const WebSocketMessageType &type) {
    // write your application logic here
    LOG_DEBUG << "new websocket message:" << message;
    LOG_DEBUG << "Connections from handler: " << connList.size();
    if (type == WebSocketMessageType::Ping) {
        LOG_DEBUG << "recv a ping";
    }
}

void WebSocketTest::handleConnectionClosed(const WebSocketConnectionPtr &) {
    LOG_DEBUG << "websocket closed!";
}

void WebSocketTest::handleNewConnection(const HttpRequestPtr &,
                                        const WebSocketConnectionPtr &conn) {
    LOG_DEBUG << "new websocket connection!";
    connList.push_back(conn);
    conn->send("haha!!!");

}

int main(int argc, char *argv[]) {
    WebSocketTest wst;
    std::string host("127.0.0.1");
    int port(8848);
    app()
    .setLogLevel(trantor::Logger::kDebug)
    .addListener(host, port);

    std::thread t([&wst] {
        std::this_thread::sleep_for(1s);
        WsClient::Init();
        std::this_thread::sleep_for(3s);
        wst.Connections();
    });

    wst.initPathRouting();
    app().run();
}

What interests me here is connList. I've added to connList every new ws connection. I've expected it will contain all new connections, but after running this example I get:

20200818 17:38:38.224419 UTC 11026 DEBUG [handleNewConnection] new websocket connection! - main.cpp:49
ws connected!
new message:haha!!!
20200818 17:38:38.224755 UTC 11026 DEBUG [handleNewMessage] new websocket message:hello! - main.cpp:36
20200818 17:38:38.224780 UTC 11026 DEBUG [handleNewMessage] Connections from handler: 1 - main.cpp:37
20200818 17:38:40.224923 UTC 11026 DEBUG [handleNewMessage] new websocket message: - main.cpp:36
20200818 17:38:40.224972 UTC 11026 DEBUG [handleNewMessage] Connections from handler: 1 - main.cpp:37
20200818 17:38:40.224982 UTC 11026 DEBUG [handleNewMessage] recv a ping - main.cpp:39
new message:
recv a pong
20200818 17:38:41.223464 UTC 11025 DEBUG [Connections] Connections from function: 0 - main.cpp:24
20200818 17:38:42.224988 UTC 11026 DEBUG [handleNewMessage] new websocket message: - main.cpp:36
20200818 17:38:42.225036 UTC 11026 DEBUG [handleNewMessage] ConnOh thank you.ections from handler: 1 - main.cpp:37
new message:20200818 17:38:42.225046 UTC 11026 DEBUG [handleNewMessage] recv a ping - main.cpp:39

Connections from handler: 1 Connections from function: 0

Why does it happens?

P.S. Is it ok to ask another questions in one thread or every time create new one?

Regards.

an-tao commented 4 years ago

Drogon automatically create WebSocketTest instance for application. the wst is not the one created by drogon and it's not working at all. Please feel free to ask questions here.

an-tao commented 4 years ago

If you really want create the controller by yourself. Set the second template parameter to false, and register it manually into the framework.

auto wsCtlPtr=std::make_shared<WebSocketTest>();
app().registerController(wsCtlPtr);

You don't need to call the initPathRouting method.

SviatoslavKomkov commented 4 years ago

I've try to reproduce next problem on test, but I failed.

The problem appears, when I've sends a message from client to server. Server waits for client response. The problem starts, after few client responses. When client send message, they just don't reach server. When the next message came to server, the previous one arrived to server without any issue.

20200819 19:42:10.303646 UTC 51785 DEBUG [SendMessage] Sending 32494 bytes - MessageWrapper.cpp:141
20200819 19:42:10.303649 UTC 51785 TRACE [sendWsData] send 32494 bytes - WebSocketConnectionImpl.cc:66
20200819 19:42:10.303652 UTC 51785 TRACE [sendWsData] bytes[2]=126 - WebSocketConnectionImpl.cc:85
20200819 19:42:10.303654 UTC 51785 TRACE [sendWsData] bytes[3]=18446744073709551598 - WebSocketConnectionImpl.cc:86
20200819 19:42:14.234111 UTC 51780 TRACE [sendWsData] send 84 bytes - WebSocketConnectionImpl.cc:66
20200819 19:42:19.234286 UTC 51780 TRACE [sendWsData] send 84 bytes - WebSocketConnectionImpl.cc:66
20200819 19:42:24.234375 UTC 51780 TRACE [sendWsData] send 84 bytes - WebSocketConnectionImpl.cc:66
20200819 19:42:29.234446 UTC 51780 TRACE [sendWsData] send 84 bytes - WebSocketConnectionImpl.cc:66
20200819 19:42:34.234518 UTC 51780 TRACE [sendWsData] send 84 bytes - WebSocketConnectionImpl.cc:66
20200819 19:42:39.234600 UTC 51780 TRACE [sendWsData] send 84 bytes - WebSocketConnectionImpl.cc:66
20200819 19:42:44.234599 UTC 51780 TRACE [sendWsData] send 84 bytes - WebSocketConnectionImpl.cc:66
20200819 19:42:49.234630 UTC 51780 TRACE [sendWsData] send 84 bytes - WebSocketConnectionImpl.cc:66
20200819 19:42:54.234639 UTC 51780 TRACE [sendWsData] send 84 bytes - WebSocketConnectionImpl.cc:66

84 bytes - is a ping message every 5 seconds. They stuck up too.

I've set next parametrs

        server->setThreadNum(16);
        server->setClientMaxWebSocketMessageSize(1024 * 1024);
        server->setPipeliningRequestsNumber(64);
        server->enableGzip(true);
        server->enableBrotli(true);
        server->registerController(WsServer);

It seems like ws connection locked up by something, Why could possibly this thing happened? What problem can cause such behavior?

Regards.

an-tao commented 4 years ago

Please show me the complete code, so that I can test it locally. Thanks.

SviatoslavKomkov commented 4 years ago

Ok, it seems I could reproduce it.

Here we go,

#include <drogon/WebSocketController.h>
#include <drogon/WebSocketClient.h>
#include <drogon/HttpAppFramework.h>
#include <iostream>
#include <random>

//#include "WsInit.h"

using namespace drogon;
using namespace std::chrono_literals;
std::atomic<uint64_t> connCounter;
bool stop = false;

struct Ctx {
    uint64_t Idx = 0;
    std::shared_ptr<std::string> Req = nullptr;
    std::shared_ptr<std::string> Resp = nullptr;
    std::mutex Mtx;
    std::condition_variable Cv;
};

uint64_t getCounter() {
    return connCounter++;
}

using namespace drogon;

namespace Json {
    Json::Value Parse(const string_view &str) {
        Json::Value root;   // will contains the root value after parsing.
        Json::CharReaderBuilder readerBuilder;
        Json::String errs;
        auto reader = readerBuilder.newCharReader();

        reader->parse(str.data(), str.data() + str.size(), &root, &errs);
        return root;
    };

    Json::Value Parse(const std::string &str) {
        return Parse(string_view(str));
    };

    std::string ToString(const Json::Value &v) {
        Json::StreamWriterBuilder fastWriter;
        auto sv = fastWriter.newStreamWriter();
        std::stringstream ss;
        sv->write(v, &ss);
        return ss.str();
    }

}
namespace WsClient {
    std::queue<WebSocketClientPtr> ptrs;

    void Init() {
        for (size_t i = 0; i < std::thread::hardware_concurrency(); i++) {
//        for (size_t i = 0; i < 1; i++) {
            std::string host("127.0.0.1");
            int port(8848);

            auto wsPtr = WebSocketClient::newWebSocketClient(host, port);
            auto req = HttpRequest::newHttpRequest();
            req->setPath("/chat");
            wsPtr->setMessageHandler([](const std::string &message,
                                        const WebSocketClientPtr &wsPtr,
                                        const WebSocketMessageType &type) {
//            std::cout << "new message:" << message << std::endl;
                if (type == WebSocketMessageType::Pong) {
                    std::cout << "recv a pong" << std::endl;
                } else if (type == WebSocketMessageType::Text) {
                    Json::Value j =  Json::Parse(message);
                    wsPtr->getConnection()->send(Json::ToString(j));
                }
            });

            wsPtr->setConnectionClosedHandler([](const WebSocketClientPtr &wsPtr) {
                std::cout << "ws closed!" << std::endl;
            });
            wsPtr->connectToServer(req,
                                   [](ReqResult r,
                                      const HttpResponsePtr &resp,
                                      const WebSocketClientPtr &wsPtr) {
                                       if (r == ReqResult::Ok) {
                                           LOG_DEBUG << "ws connected!";
                                       } else {
                                           LOG_DEBUG << "ws failed!";
                                       }
                                   });
            ptrs.push(wsPtr);
        }
    }
}

const std::string vocabulary = "abcdefghijklmnopqrstuvwxyz";
std::string GenerateRnd(size_t num) {
    std::random_device dev;
    std::mt19937 rng(dev());
    std::uniform_int_distribution<std::mt19937::result_type> dist(0, vocabulary.size() - 1);

    std::string ret;
    for (size_t i(0); i < num; i++) {
        ret.push_back(vocabulary[dist(rng)]);
    }

    return ret;
}

class WebSocketTest : public drogon::WebSocketController<WebSocketTest, false> {
public:
    virtual void handleNewMessage(const WebSocketConnectionPtr &,
                                  std::string &&,
                                  const WebSocketMessageType &) override;

    virtual void handleConnectionClosed(
            const WebSocketConnectionPtr &) override;

    virtual void handleNewConnection(const HttpRequestPtr &,
                                     const WebSocketConnectionPtr &) override;

    std::queue<WebSocketConnectionPtr> availableConnections;
    std::mutex availableConnectionsMtx;
    std::mutex availableConnectionsDelMtx;
    std::condition_variable availableConnectionsCv;

    std::function<Json::Value()> Echo(const Json::Value &request) {
        std::unique_lock<std::mutex> lck(this->availableConnectionsMtx);
        LOG_DEBUG << "this->availableConnections: " << this->availableConnections.size();
        availableConnectionsCv.wait(lck, [this] { return !this->availableConnections.empty(); });
        std::unique_lock<std::mutex> lck2(this->availableConnectionsDelMtx);
        auto conn = this->availableConnections.front();
        this->availableConnections.pop();

        auto ctx = conn->getContext<Ctx>();
//        auto bigString = GenerateRnd(ctx->Idx + 10);
//        auto bigString = GenerateRnd(1024 * 32);
        auto bigString(request);
        Json::Value val = Json::ToString(request);

        std::string jsonStr;
        jsonStr = Json::ToString(request);
        ctx->Req = std::make_shared<std::string>(jsonStr);
        LOG_DEBUG << "Send message #" << jsonStr << " to #" << ctx->Idx;
        conn->send(jsonStr);

        return [this, conn, ctx]() -> Json::Value {
            std::unique_lock<std::mutex> lck(ctx->Mtx);
            ctx->Cv.wait_for(lck, std::chrono::seconds(5), [&ctx] { return ctx->Resp != nullptr; });
            bool flag = true;
            if (ctx->Resp == nullptr) {
                LOG_DEBUG << "Not received #" << ctx->Idx;
                flag = false;
            } else if (*ctx->Resp != *ctx->Req) {
                LOG_DEBUG << "Not equal #" << ctx->Idx << "\n"
                          << "ctx->Resp.size(): " << ctx->Resp->size() << "\n"
                          << "ctx->Req.size(): " << ctx->Req->size() << "\n"
                          << "ctx->Resp: " << *ctx->Resp << "\n"
                          << "ctx->Req: " << *ctx->Req;

                flag = false;
            }

            Json::Value resp;
            if (flag) {
                LOG_DEBUG << "OK #" << ctx->Idx;
                std::lock_guard<std::mutex> lck1(this->availableConnectionsMtx);
                resp = Json::Parse(*ctx->Resp);
            }
            ctx->Resp = nullptr;
            ctx->Req = nullptr;
            this->availableConnections.push(conn);
            this->availableConnectionsCv.notify_one();

            return resp;
        };
    }

    WS_PATH_LIST_BEGIN
        WS_PATH_ADD("/chat", "drogon::LocalHostFilter", Get);
    WS_PATH_LIST_END
};

void WebSocketTest::handleNewMessage(const WebSocketConnectionPtr &wsConnPtr,
                                     std::string &&message,
                                     const WebSocketMessageType &type) {
    auto ctx = wsConnPtr->getContext<Ctx>();
    LOG_DEBUG << "Get message from #" << ctx->Idx;
//    if (message.size() < 100) {
//        return;
//    }
    LOG_DEBUG << __FILE__ << ":" << __LINE__;
    std::lock_guard<std::mutex> lck(ctx->Mtx);
    ctx->Resp = std::make_shared<std::string>(message);
    ctx->Cv.notify_one();
}

void WebSocketTest::handleConnectionClosed(const WebSocketConnectionPtr &) {
    LOG_DEBUG << "websocket closed!";

}

void WebSocketTest::handleNewConnection(const HttpRequestPtr &,
                                        const WebSocketConnectionPtr &conn) {

    auto ctx = std::make_shared<Ctx>();
    ctx->Idx = getCounter();
    conn->setContext(ctx);
    LOG_DEBUG << "new websocket connection!";
    std::lock_guard<std::mutex> lck1(this->availableConnectionsMtx);
    std::unique_lock<std::mutex> lck2(this->availableConnectionsDelMtx);
    availableConnections.push(conn);
    availableConnectionsCv.notify_one();

//    conn->send("Connection ok");
}

std::shared_ptr<WebSocketTest> wsCtlPtr;

int main(int argc, char *argv[]) {
    wsCtlPtr = std::make_shared<WebSocketTest>();
    std::string host("127.0.0.1");
    int port(8848);
    app()
            .setLogLevel(trantor::Logger::kDebug)
            .addListener(host, port);

    WsClient::Init();

    std::atomic<uint64_t> counter;
    app().addListener("127.0.0.1", 8848);
    app().setThreadNum(16);
    app().setClientMaxWebSocketMessageSize(1024 * 1024);
    app().setPipeliningRequestsNumber(64);
    app().enableGzip(true);
    app().enableBrotli(true);
    app().registerHandler("/echo", [&counter](const HttpRequestPtr &req,
                                      std::function<void(const HttpResponsePtr &)> &&callback) {
        auto reqUid = counter++;
        std::string_view msg = req->body();
        Json::Value reqJson = Json::Parse(msg);
        reqJson["uid"] = reqUid;
        LOG_DEBUG << "Send message from http #" << msg << " uid #" << reqUid;
        auto future = wsCtlPtr->Echo(reqJson);
        auto resp = future();
        LOG_DEBUG << "Response message from http #" << msg << " uid #" << reqUid;
        auto res = drogon::HttpResponse::newHttpJsonResponse(resp);
        callback(res);

    });
    app().registerController(wsCtlPtr);
    app().run();
}

Testing script:

#!/usr/bin/env bash

COUNTER=${1}
COMMAND=${2}
echo "Running '${COMMAND}' for ${COUNTER} times"
for i in `seq 1 ${COUNTER}`;
do
  eval "${COMMAND}"
done

Testing script arguments:

./Looper.sh 100 "curl --location --request POST '127.0.0.1:8848/echo' \
--header 'Content-Type: application/json' \
--data-raw '{\"test\":\"tuturu\"}

If it will not reproduce, I will send debug list.

Regards.

an-tao commented 4 years ago

@SviatoslavKomkov I saw there are some blocking operations in your code (the wait() and wait_for() method). This can lead some problems:

the IO threads in drogon should never be blocked, because blocking IO threads would decrease performance and throughput very much. Remember that there are only 16 threads(in your code), if a thread block for 1 second for every request, the total throughput is only 16 QPS.
In some cases, for example, the event you are waiting for depends on the subsequent data received from network and processed in the same IO thread, a logical deadlock will occur. (current thread can't handle subsequent data when blocked). This is a more serious problem.

I'm not sure if above things happened in your case, please check and confirm it.

SviatoslavKomkov commented 4 years ago

It seems that Drogon use one big context(lets call it Drogon main thread or DMT), that responsible for routing between Drogon subproceses, so when you locked something up in DMT, it cause stopping of every routing process.

The problem above is clearly lock DMT. Server sends message to client, and server create next condition: I will unlock only when receive the answer from client. Client try to send message, but DMT is locked up because the subprocess that must receive message is locked up because of waiting this answer.

To prevent this you must not locking DMT subprocess with conditions, that can be reached only by using another DMT suprocess. But you clearly can use locks inside one DMT subprocess.

Sorry if my thought is not clear enough to explain this example.

Thanks for routing my mind in right direction.

an-tao commented 4 years ago

What you said is relatively close, but not completely correct. Drogon has a main Event Loop, but it is not responsible for network processing and routing. The working threads is called IO Event Loops ( there are 16 IO Event loops in your case). Every IO event loop manager and handle multiple connections (a particular connection always be processed in a particular event loop). We call it event loop because it is a true loop in programming. When nothing happens, it is blocked at epoll_wait() function, and if some networking events occurs, the epoll_wait() returns these events, we handle them and then call epoll_wait() again in the loop. Your description of DMT is correct for IO event loops. For an IO event loop, if it is blocked by user application code, other events and upcoming events that occur on the connections managed by the same event loop will never be processed.

SviatoslavKomkov commented 4 years ago

Here is the problem with httpClient.

Example:

stringstream ss;
std::string url("https://api.telegram.org/");

std::string botId("1164954682:AAFMTqiB7kFU73gJyCICcCITqQqFZQjQ3Xq");
std::string chatId("1448412012");

ss << "bot" << botId << "/sendMessage";
string path(ss.str());

Json::Value obj;
obj["chat_id"] = "-100" + chatId;
obj["text"] = "Test";

auto client = HttpClient::newHttpClient(url);
auto req = HttpRequest::newHttpRequest();
req->setMethod(drogon::Post);
req->setPath(ss.str());

client->sendRequest(
  req,
  [](ReqResult result, const HttpResponsePtr &response) {
LOG_DEBUG << "Telegram receive response.";
LOG_DEBUG << "Body: " << response->body();

if (result == ReqResult::Ok) {
    LOG_DEBUG << "Success";
} else {
    LOG_DEBUG << "Failed";
}
});

Here is must send to:

https://api.telegram.org/bot1164954682:AAFMTqiB7kFU73gJyCICcCITqQqFZQjQ3Xq/sendMessage

Instead send to(relying to trace logs):

TRACE [sendReq] Send request:POST bot1164954682%3AAFMTqiB7kFU73gJyCICcCITqQqFZQjQ3Xq/sendMessage HTTP/1.1

And I got error code 400 as response. Is there some possibility to send httpRequest without url been converted to url-encoded format?

Regards.

an-tao commented 4 years ago

ss << "/bot" << botId << "/sendMessage";

The path must be prefixed by '/'

SviatoslavKomkov commented 4 years ago

Thanks. It make some fixes but main problem is still exists.

    std::string ToString(const Value &val) {
        StreamWriterBuilder fastWriter;
        auto svPtr = fastWriter.newStreamWriter();
        auto sv = std::shared_ptr<StreamWriter>(svPtr);
        svPtr = nullptr;
        std::stringstream ss;
        sv->write(val, &ss);
        return ss.str();
    }

  stringstream ss;
  std::string url("https://api.telegram.org/");

  std::string botId("1334656530:AAG4T1tImQ-TRdFDwErzJpqg2oAS-hERknA");
  std::string chatId("1163222674");

  ss << "/bot" << botId << "/sendMessage";
  string path(ss.str());

  auto msg = "Message from native";
  Json::Value obj;
  obj["chat_id"] = "-100" + chatId;
  obj["text"] = msg;

  auto client = HttpClient::newHttpClient(url);
  auto req = HttpRequest::newHttpRequest();
  req->setContentTypeCode(CT_APPLICATION_JSON);
  req->setMethod(drogon::Post);
  req->setBody(Json::ToString(obj));
  req->setPath(path);

  client->sendRequest(
    req,
    [](ReqResult result, const HttpResponsePtr &response) {
        LOG_DEBUG << "Telegram receive response.";
        LOG_DEBUG << "Body: " << response->body();

        if (result == ReqResult::Ok) {
       LOG_DEBUG << "Success";
        } else {
       LOG_DEBUG << "Failed";
        }
    });

Trace:

TRACE [sendReq] Send request:POST /bot1334656530%3AAAG4T1tImQ-TRdFDwErzJpqg2oAS-hERknA/sendMessage HTTP/1.1
Content-Length: 66
Content-Type: text/plain; charset=utf-8
User-Agent: DrogonClient
Host: api.telegram.org
Connection: Keep-Alive

{
    "chat_id" : "-1001163222674",
    "text" : "Message from native"
} - HttpClientImpl.cc:412

Error:

{"ok":false,"error_code":400,"description":"Bad Request: message text is empty"}

Curl working example:

curl --location --request POST 'https://api.telegram.org/bot1334656530:AAG4T1tImQ-TRdFDwErzJpqg2oAS-hERknA/sendMessage' \
--header 'Content-Type: application/json' \
--data-raw '{
    "chat_id":"-1001163222674",
    "text":"Message from curl"
}'

And why request still has 'text/plain' even if json was set?

Content-Type: text/plain; charset=utf-8

Feel free to send some messages to this telegram channel if needed.

an-tao commented 4 years ago

20200831 12:59:32.775118 UTC 6778309 TRACE [sendReq] Send request:POST /bot1334656530%3AAAG4T1tImQ-TRdFDwErzJpqg2oAS-hERknA/sendMessage HTTP/1.1
Content-Length: 66
Content-Type: application/json; charset=utf-8
host: api.telegram.org
user-agent: DrogonClient
connection: Keep-Alive

{
    "chat_id" : "-1001163222674",
    "text" : "Message from native"
} - HttpClientImpl.cc:434
20200831 12:59:32.775140 UTC 6778309 TRACE [writeInLoop] send in loop - TcpConnectionImpl.cc:1288
20200831 12:59:33.206634 UTC 6778309 TRACE [readCallback] read Callback - TcpConnectionImpl.cc:329
20200831 12:59:33.206747 UTC 6778309 TRACE [readCallback] ssl read:-1 bytes - TcpConnectionImpl.cc:348
...
20200831 12:59:33.730609 UTC 6778309 DEBUG [operator()] Telegram receive response. - main.cc:40
20200831 12:59:33.730616 UTC 6778309 DEBUG [operator()] Body: {"ok":true,"result":{"message_id":6,"chat":{"id":-1001163222674,"title":"Testing","type":"channel"},"date":1598878773,"text":"Message from native"}} - main.cc:41
20200831 12:59:33.730622 UTC 6778309 DEBUG [operator()] Success - main.cc:44

This is the logs of running your code on my machine. It works fine. Please recompile drogon (make clean && make -j install) and your application completely and try again. BTW: please use req = HttpRequest::newHttpJsonRequest(json); instead of setBody() and setContentTypeCode(). Although they are equivalent, the former is more concise.

SviatoslavKomkov commented 4 years ago

Have already rebuild several times with different drogon versions v1.0.0-beta[20 - 21].

My full log looks like:

TRACE [sendReq] Send request:POST /bot1334656530%3AAAG4T1tImQ-TRdFDwErzJpqg2oAS-hERknA/sendMessage HTTP/1.1
Content-Length: 66
Content-Type: text/plain; charset=utf-8
User-Agent: DrogonClient
Host: api.telegram.org
Connection: Keep-Alive

{
    "chat_id" : "-1001163222674",
    "text" : "Message from native"
} - HttpClientImpl.cc:412
20200831 16:20:05.291743 UTC 41331 TRACE [writeInLoop] send in loop - TcpConnectionImpl.cc:1288
20200831 16:20:05.331601 UTC 41331 TRACE [readCallback] read Callback - TcpConnectionImpl.cc:329
20200831 16:20:05.331640 UTC 41331 TRACE [readCallback] ssl read:-1 bytes - TcpConnectionImpl.cc:348
20200831 16:20:05.372947 UTC 41331 TRACE [readCallback] read Callback - TcpConnectionImpl.cc:329
20200831 16:20:05.372958 UTC 41331 TRACE [readCallback] ssl read:431 bytes - TcpConnectionImpl.cc:348
20200831 16:20:05.372963 UTC 41331 TRACE [processResponseLine] 1 - HttpResponseParser.cc:44
20200831 16:20:05.372966 UTC 41331 TRACE [processResponseLine] 400 Bad Request - HttpResponseParser.cc:65
20200831 16:20:05.372986 UTC 41331 TRACE [parseResponse] post got all:len=0 - HttpResponseParser.cc:200
20200831 16:20:05.372991 UTC 41331 TRACE [parseResponse] content(END) - HttpResponseParser.cc:202
20200831 16:20:05.373015 UTC 41331 DEBUG [operator()] Telegram receive response. - Telegram.cpp:40
20200831 16:20:05.373020 UTC 41331 DEBUG [operator()] Body: {"ok":false,"error_code":400,"description":"Bad Request: message text is empty"} - Telegram.cpp:41
20200831 16:20:05.373022 UTC 41331 DEBUG [operator()] Success - Telegram.cpp:44

the thing here is probably:

my: Content-Type: text/plain; charset=utf-8
yours: Content-Type: application/json; charset=utf-8

I've change this:

  auto client = HttpClient::newHttpClient(url);
  auto req = HttpRequest::newHttpRequest();
  req->setContentTypeCode(CT_APPLICATION_JSON);
  req->setMethod(drogon::Post);
  req->setBody(Json::ToString(obj));
  req->setPath(path);

to:

auto client = HttpClient::newHttpClient(url);
req = HttpRequest::newHttpJsonRequest(obj);
req->setPath(path);

I've tried on manjaro and ubuntu16.04. Result always the same. Why does text/plain won't change to application/json in my cases? Checking on drogon versions v1.0.0-beta[20-21].

an-tao commented 4 years ago

please set a breakpoint at req->setContentTypeCode(CT_APPLICATION_JSON); and run into it step by step in debug mode.

an-tao commented 4 years ago

@SviatoslavKomkov I tested your code on Ubuntu16.04, It works fine. I suspect that there are multiple drogon libraries in your system. Please use LOG_DEBUG << drogon::getVersion() to confirm whether your application is linked to the correct library.

SviatoslavKomkov commented 4 years ago

You are probably right.

I've tested several times. The thing is I have debug and release drogon builds, somehow cmake using two different libs and this happened. I've placed logs in release and debug builds of project and clearly in different functions were used both libraries. So this is obviously my bad with cmake scripting but still.

Thanks for help.

SviatoslavKomkov commented 3 years ago

When you started Drogon http, the folder "uploads" created. It's not a problem, but good to know, is there some way to manage max size of this folder with standart drogon methods?

I have large request sometimes, and 2 weeks later folder had ~20GB size. Is there some way to prevent this?

Regards.

an-tao commented 3 years ago

No, there is no universal way to avoid running out of disk space. Some users may delete the oldest files or largest files, while other users may give up creating new files. I think you should deal with this problem yourself.

SviatoslavKomkov commented 3 years ago

Hello. Near a year past and all works like a charm :)

I want to ask again about "uploads" folder, how can I prevent it's creation and just parse request as is. And what is the purpose of this folder? There are some situations of high load performance, when my hdd just suffering from too many files.

Also the problem of oversizing is still there. Current solution, just delete files where lifespan above N minutes. But it still an issue for me. In addition, writing\delete a lot amount of files at ssd is a bad thing.

So the questions are: 1) What a purpose of uploads folder, rather than parse requests as is? 2) Is it possible to parse request without saving it to uploads?

Regards.

an-tao commented 3 years ago

We cache a request body to a file when the body size is bigger than a value (64k by default), The purpose of this is to avoid occupying too much memory. If a lot of connections send large requests and their network are very slow, a lot of memory is occupied for a long time (Say 10000 connections send requests with 1M bytes images at the same time, this means 10G bytes memory is used). And the temparary files are deleted automatically when the related requests are destroyed. you don't need menually remove them. if you don't want to use temparary files, set the client_max_memory_body_size option to "" to disable the feature.

drogonframework / drogon

Websocket not connected to server in native #544