Sendfile works faster for larger files?

[ahmedyarub]

I've been doing benchmarks for multiple web-servers/REST frameworks inlcuding rwasa, asmttpd, C++ REST SDK, Spring MVC, Spring Webflux and Restinio. The performance for sending files with Restinio was among the best. Especially that it didn't have the deadlocking problem serving files with multiple requests that the C++ REST SDK had. However, I've noticed a strange behavior with Restinio: I make 4 comparative benchmarks with Gatling: 2 with 300 simultaneous users and 2 with 3000 users, each one is done with 70 bytes and 700kb files respectively. The smaller file is an HTTP file and the larger one is a binary file. For the larger file the response time has been like this:

and for the smaller file like this:

As you can see, the smaller file has a larger average time for some reason! I've done that multiple times and always had the same results. Any explanation for that? All the tests were done on the sendfile sample on Linux

[eao197]

Hi!

Thanks for reporting this.

We need some time to investigate this issue. However, there are holidays from 5 to 9 of May in our country, so I can't guarantee we can do it quickly.

[ahmedyarub]

no rush thank you for the prompt reply

[mipac]

hi, very interresting! do you plan to investigate other lib like restbed, pistache and etc... do you plan to share this bench code? best regards

[ahmedyarub]

@mipac I'm working exactly on that. I even compared asmttpd, rwasa (written in assembly language), Spring MVC, Spring Webflux (written in Java) and Microsoft C++ REST SDK. The series of articles are currently only available in Portuguese but I'll be soon translating them to English with a more thorough explanation. You can find my articles here: https://www.linkedin.com/in/ahmed-yarub-hani-al-nuaimi-77186267/detail/recent-activity/posts/

I'm working on a boiler-plate micro-service written in C++ (and may be some assembly parts). The project is still in alpha-stage but rest assured that it will reach a fully functioning micro-service with CI/CD soon: https://github.com/ahmedyarub/micro-service

[eao197]

@ahmedyarub I want to ask: how many different files were used in your benchmark? Did all your requests ask for the same file or every request asks a unique file?

[ahmedyarub]

@ahmedyarub I want to ask: how many different files were used in your benchmark? Did all your requests ask for the same file or every request asks a unique file?

2 files only: one of them is a 700 bytes HTML file and the other is a 700KB binary file. All the requests of each of the benchmarks asked for the same file. It is done this way in order to be comparable to other frameworks that I'm testing.

[eao197]

All the requests of each of the benchmarks asked for the same file.

RESTinio does the same actions and the same system calls for all files regardless of their sizes/attributes. So I think there is some OS-specific thing.

I have a hypothesis but don't check it yet: the transfer of a small files takes so little time that RESTinio opens a file, calls sendfile, and then closes file just before next request arrived (or its processing started). Because of that OS reopens and closes the file again and again.

But in the case of a large file, the transfer time can be significantly bigger. It means that when the next request is accepted the transfer of file for the previous request is not finished yet. And OS can reuse already created data structures for the file. I suppose that in that case open and close calls work much faster.

It is just a hypothesis.

[ahmedyarub]

do you suggest a way to troubleshoot this issue? because it didn't happen to me with other frameworks.

[eao197]

@ahmedyarub Can you show the code you used for the benchmark?

[ahmedyarub]

https://github.com/ahmedyarub/boot2-load-demo/blob/master/applications/load-scripts/src/test/scala/simulations/BootLoadSimulation.scala

Tested with this command: ./gradlew -p applications/load-scripts -DTARGET_URL=http://localhost:8088 -DSIM_USERS=3000 gatlingRun

And the Restinio code:

#include <iostream>
#include <restinio/all.hpp>
#include <fmt/format.h>
#include <clara/clara.hpp>

struct app_args_t
{
    bool m_help{ false };
    std::string m_address{ "localhost" };
    std::uint16_t m_port{ 8088 };
    std::size_t m_pool_size{ 40 };
    std::string m_file = "index.html";
    restinio::file_offset_t m_data_offset{ 0 };
    restinio::file_size_t m_data_size{ std::numeric_limits< restinio::file_size_t >::max() };
    std::string m_content_type{ "text/html" };
    bool m_trace_server{ false };

    static app_args_t
    parse( int argc, const char * argv[] )
    {
        using namespace clara;

        app_args_t result;

        auto cli =
            Opt( result.m_address, "address" )
                    ["-a"]["--address"]
                    ( fmt::format( "address to listen (default: {})", result.m_address ) )
            | Opt( result.m_port, "port" )
                    ["-p"]["--port"]
                    ( fmt::format( "port to listen (default: {})", result.m_port ) )
            | Opt( result.m_pool_size, "thread-pool size" )
                    [ "-n" ][ "--thread-pool-size" ]
                    ( fmt::format(
                        "The size of a thread pool to run server (default: {})",
                        result.m_pool_size ) )
            | Opt( result.m_data_offset, "offset" )
                    ["-o"]["--data-offset"]
                    ( fmt::format(
                        "Offset of the data portion in file (default: {})",
                        result.m_data_offset ) )
            | Opt( result.m_data_size, "size" )
                    ["-s"]["--data-size"]
                    ( "size of the data portion in file (default: to the end of file)" )
            | Opt( result.m_content_type, "content-type" )
                    ["--content-type"]
                    ( fmt::format(
                        "A value of 'Content-Type' header field (default: {})",
                        result.m_content_type ) )
            | Opt( result.m_trace_server )
                    [ "-t" ][ "--trace" ]
                    ( "Enable trace server" )
            | Arg( result.m_file, "file" ).required()
                    ( "Path to a file that will be served as response" )
            | Help(result.m_help);

        auto parse_result = cli.parse( Args(argc, argv) );
        if( !parse_result )
        {
            throw std::runtime_error{
                fmt::format(
                    "Invalid command-line arguments: {}",
                    parse_result.errorMessage() ) };
        }

        if( result.m_help )
        {
            std::cout << cli << std::endl;
        }

        return result;
    }
};

template < typename Server_Traits >
void run_server( const app_args_t & args )
{
    restinio::run(
        restinio::on_thread_pool< Server_Traits >( args.m_pool_size )
            .port( args.m_port )
            .address( args.m_address )
            .concurrent_accepts_count( args.m_pool_size )
            .request_handler(
                [&]( auto req ){
                    if( restinio::http_method_get() == req->header().method())
                    {
                        try
                        {
                            auto sf = restinio::sendfile( args.m_file );
                            sf.offset_and_size(
                                args.m_data_offset,
                                args.m_data_size );

                            return
                                req->create_response()
                                    .append_header( restinio::http_field::server, "RESTinio hello world server" )
                                    .append_header_date_field()
                                    .append_header(
                                        restinio::http_field::content_type,
                                        args.m_content_type )
                                    .set_body( std::move( sf ) )
                                    .done();
                        }
                        catch( const std::exception & )
                        {
                            return
                                req->create_response(
                                        restinio::status_not_found() )
                                    .connection_close()
                                    .append_header_date_field()
                                    .done();
                        }
                    }

                    return restinio::request_rejected();
            } ) );
}

int main( int argc, const char * argv[] )
{
    try
    {
        const auto args = app_args_t::parse( argc, argv );

        if( !args.m_help )
        {
            if( args.m_trace_server )
            {
                using traits_t =
                    restinio::traits_t<
                        restinio::asio_timer_manager_t,
                        restinio::shared_ostream_logger_t >;

                run_server< traits_t >( args );
            }
            else
            {
                run_server< restinio::default_traits_t >( args );
            }
        }
    }
    catch( const std::exception & ex )
    {
        std::cerr << "Error: " << ex.what() << std::endl;
        return 1;
    }

    return 0;
}

[eao197]

I think there can be an influence of Nagle's algorithm. Can you modify RESTinio code that way:

template < typename Server_Traits >
void run_server( const app_args_t & args )
{
    restinio::run(
        restinio::on_thread_pool< Server_Traits >( args.m_pool_size )
            .port( args.m_port )
            .address( args.m_address )
            .concurrent_accepts_count( args.m_pool_size )
            .socket_options_setter( []( auto & options ) {
                    restinio::asio_ns::ip::tcp::no_delay no_delay{ true };
                    options.set_option( no_delay );
                } )
            .request_handler(
                [&]( auto req ){

You need to add a call to socket_options_setter from the code above.

[ahmedyarub]

well that was it! before:

---- Global Information --------------------------------------------------------

request count 9000 (OK=9000 KO=0 ) min response time 0 (OK=0 KO=- ) max response time 56 (OK=56 KO=- ) mean response time 44 (OK=44 KO=- ) std deviation 8 (OK=8 KO=- ) response time 50th percentile 44 (OK=44 KO=- ) response time 75th percentile 47 (OK=47 KO=- ) response time 95th percentile 48 (OK=48 KO=- ) response time 99th percentile 49 (OK=49 KO=- ) mean requests/sec 281.25 (OK=281.25 KO=- ) ---- Response Time Distribution ------------------------------------------------ t < 800 ms 9000 (100%) 800 ms < t < 1200 ms 0 ( 0%) t > 1200 ms 0 ( 0%) failed 0 ( 0%)

After:

---- Global Information --------------------------------------------------------

request count 9000 (OK=9000 KO=0 ) min response time 0 (OK=0 KO=- ) max response time 37 (OK=37 KO=- ) mean response time 0 (OK=0 KO=- ) std deviation 1 (OK=1 KO=- ) response time 50th percentile 0 (OK=0 KO=- ) response time 75th percentile 1 (OK=1 KO=- ) response time 95th percentile 1 (OK=1 KO=- ) response time 99th percentile 2 (OK=2 KO=- ) mean requests/sec 300 (OK=300 KO=- ) ---- Response Time Distribution ------------------------------------------------ t < 800 ms 9000 (100%) 800 ms < t < 1200 ms 0 ( 0%) t > 1200 ms 0 ( 0%) failed 0 ( 0%)

Today a learned a new thing! Thanks a lot for the help.