eclipse-iceoryx / iceoryx

Eclipse iceoryx™ - true zero-copy inter-process-communication
https://iceoryx.io
Apache License 2.0
1.68k stars 391 forks source link

Request server received no valid server port from RouDi #2315

Open nevernevergu opened 3 months ago

nevernevergu commented 3 months ago

Hey, guys: I am using a request-response model, with the client and server in an n-n mode. When the server receives a trigger message, it creates n threads. Each thread will use a different Wrapper instance to call InitServer to create an UntypedServer, which is used to receive messages from a specified client.

// in server
void Wrapper::InitServer() {
    server_.reset(new iox::popo::UntypedServer({service, instance, event}));
}

However, sometimes when my server creates multiple threads simultaneously, the following error occurs:

2024-07-11 14:44:42.449 [Error]: Request server received no valid server port from RouDi.
2024-07-11 14:44:42.449 [Warn ]: Could not create server with service description 'Service: myservice, Instance: myserver, Event: QHGGBv06h12475329139651441481472' as we are out of memory for servers.
2024-07-11 14:44:42.449 [Error]: iceoryx_posh/source/runtime/posh_runtime_impl.cpp:533 [POSH__RUNTIME_ROUDI_OUT_OF_SERVERS (code = 13)] in module [iceoryx_posh (id = 2)]

I would like to ask : 1) what causes this error? Is it problematic to create multiple servers simultaneously with multiple threads? 2) Should the created UntypedServer be manually released when the thread exits?

Thank you very much :-)

elBoberido commented 3 months ago

It seems you are running out of server resources. I can only speculating about the reason. Usually the dtor takes care of releasing the resources. Unfortunately the introspection client does not yet support client and servers.

Creating multiple servers simultaneously is not a problem.

You could try to run iox-roudi -l debug and check whether Destroy server port from runtime ... is printed on the terminal. By default, there should be resources for 512 servers.

nevernevergu commented 3 months ago

Thank you @elBoberido , I'll use the iox-roudi -l debug trying to find the reason. When I use request-response pattern, I meet another problem that the no available chunk resources:

2024-07-12 17:40:28.876 [Warn ]: Mempool [m_chunkSize = 1200000048, numberOfChunks = 10, used_chunks = 10 ] has no more space left
2024-07-12 17:40:28.876 [Error]: MemoryManager: unable to acquire a chunk with a chunk-payload size of 1073741848The following mempools are available:  MemPool [ ChunkSize = 1072, ChunkPayloadSize = 1024, ChunkCount = 100000 ]  MemPool [ ChunkSize = 4144, ChunkPayloadSize = 4096, ChunkCount = 100000 ]  MemPool [ ChunkSize = 65584, ChunkPayloadSize = 65536, ChunkCount = 50000 ]  MemPool [ ChunkSize = 153648, ChunkPayloadSize = 153600, ChunkCount = 100000 ]  MemPool [ ChunkSize = 320248, ChunkPayloadSize = 320200, ChunkCount = 20000 ]  MemPool [ ChunkSize = 3202048, ChunkPayloadSize = 3202000, ChunkCount = 1000 ]  MemPool [ ChunkSize = 1200000048, ChunkPayloadSize = 1200000000, ChunkCount = 10 ]
2024-07-12 17:40:28.876 [Error]: iceoryx_posh/source/mepoo/memory_manager.cpp:197 [MEPOO__MEMPOOL_GETCHUNK_POOL_IS_RUNNING_OUT_OF_CHUNKS (code = 84)] in module [iceoryx_posh (id = 2)]
Could not allocate Read Response! Error: AllocationError::RUNNING_OUT_OF_CHUNKS

My code is as follows: After I use loan to allocate 1GB of space, I send a request to the server. Once I receive the response, I release the resources. However, I still encounter error messages indicating resource exhaustion. What could be the reason for this?

void IceoryxWrapper::OnRequest(ReqRes& reqRes) {

    int reqsize = reqRes.GetRequestSize();
    int alignsize = reqRes.GetRequestAlignSize();
    int64_t expectedResponseSequenceId = requestSequenceId_;

    {
    client_->loan(reqsize, alignsize)
                .and_then([&](auto& requestPayload) {

                    auto requestHeader = iox::popo::RequestHeader::fromPayload(requestPayload);
                                        requestHeader->setSequenceId(requestSequenceId_);
                    expectedResponseSequenceId = requestSequenceId_;
                    requestSequenceId_ += 1;
                    char* request = static_cast<char*>(requestPayload);

                    reqRes.CopyRequestDataToBuf((void*)request);
                    client_->send(request).or_else(
                        [&](auto& error) { std::cout << "Could not send Request! Error: " << error << std::endl; });
            })
            .or_else([](auto& error) { std::cout << "Could not allocate Request! Error: " << error << std::endl; });

    }
    //! [take response]
    {
    bool hasReceivedResponse{false};
    do{
        client_->take().and_then([&](const auto& responsePayload) {
            auto responseHeader = iox::popo::ResponseHeader::fromPayload(responsePayload);
            if (responseHeader->getSequenceId() == expectedResponseSequenceId)
            {
                reqRes.SetResponse((void*)responsePayload);
                client_->releaseResponse(responsePayload);
                std::cout << "Got Response with expected sequence ID! -> continue" << std::endl;
            }
            else
            {
                spdlog::error("Got Response with outdated sequence ID! Expected = {}; Actual = {} ! -> skip",
                            expectedResponseSequenceId, responseHeader->getSequenceId());
            }
            hasReceivedResponse = true;
        });
    } while (!hasReceivedResponse);
    }

}
elBoberido commented 1 week ago

You have to move client_->releaseResponse(responsePayload); out of the if condition, else you will lose chunks when the sequence ID of the response does not match the expected sequence ID.

Can you check if that solves your problem?