search

lpereira / lwan

Experimental, scalable, high performance HTTP server
https://lwan.ws
GNU General Public License v2.0
5.94k stars 548 forks source link

Error when response large content using strbuf_set_static #77

Closed whatvn closed 9 years ago

whatvn commented 9 years ago

We are working on a lwan module that read images from an external service (written using Apache thrift). The specify content is images that user upload, then save to thrift backend, response to web's user by lwan.

Every object can be served fine when its size is smaller than 2.4MB. But when it's larger that than, lwan response with content size = 2.4 MB, even when we sent content length > 2.4M, that's cause content-length mis-match, image received is not enough (it was cut) to 2.4MB and client have to wait until timeout. response->mime_type = lwan_determine_mime_type_for_file_name(file_name); strbuf_set(response->buffer,(char*) fi->metadata.content, fi->metadata.size);

when metadata.content has size larger than 2.4MB, problem occurs. Is this a known issue? I tested when lwan serve files module and it works fine.

lpereira commented 9 years ago

This is a pretty curious issue. I've never encountered it before. Is the metadata size correct before strbuf_set() is called? I ask this because, if fi->metadata.size is zero, strlen() will be used inside strbuf_set() to calculate the length; and this being arbitrary binary data, will of course not work. The same thing happens with strbuf_set_static().

whatvn commented 9 years ago

Yes, metadata size is correct. Even after doing strbuf_set_static, I save metadata.size into local file, I can read it well, content is enough.

I write a test code with static file, open local file and serve by lwan, with content size 3MB (this image: http://www.zastavki.com/pictures/originals/2013/Love__040998_.jpg), it fail.

Source code: 

static size_t
get_file_size(const char * file_name) {
    struct stat sb;
    if (stat(file_name, & sb) != 0) {
        fprintf(stderr, "'stat' failed for '%s': %s.\n",
                file_name, strerror(errno));
        exit(EXIT_FAILURE);
    }
    return sb.st_size;
}

// ham nay load file tu dia len memory

size_t ae_load_file_to_memory(const char *filename, char **result) {
    size_t size = 0;
    FILE *f = fopen(filename, "rb");
    if (f == NULL) {
        *result = NULL;
        return 0; // -1 means file opening fail 
    }
    fseek(f, 0, SEEK_END);
    size = (size_t) ftell(f);
    fseek(f, 0, SEEK_SET);
    *result = (char *) malloc(size + 1);
    if (size != fread(*result, sizeof (char), size, f)) {
        free(*result);
        return 0; // -2 means file reading fail 
    }
    fclose(f);
    (*result)[size] = 0;
    return size;
}

// handler trong file config 

lwan_http_status_t
static_handler(lwan_request_t *request,
        lwan_response_t *response,
        void *data __attribute__((unused))) {
    struct file_info_t *fi;
    fi = calloc(1, sizeof (fi));
    lwan_url_map_t *url_map;
    url_map = lwan_trie_lookup_prefix(l.url_map_trie, request->original_url.value);
    char physical_path[200];
    printf("prefix: %s\n", url_map->prefix);
    strcpy(physical_path, l.config.root);
    strcat(physical_path, request->url.value);
    fi = (struct file_info_t*) cache_coro_get_and_ref_entry(cache,
            request->conn->coro, (const char*) physical_path);
    response->mime_type = lwan_determine_mime_type_for_file_name("a.jpg");
    // tra ra cho browser
    strbuf_set_static(response->buffer, fi->metadata.content, fi->metadata.size);
    return HTTP_OK;
}

// read file and initialize cache object

static struct cache_entry_t *
create_fileinfo(const char *file_path,
        void *context __attribute__((unused))) {
    struct file_info_t *fileinfo_;
    fileinfo_ = calloc(1, sizeof (struct file_info_t));
    size_t file_size = 0;
    char *content = NULL;
    ae_load_file_to_memory(file_path, &content);
    if (content) {
        file_size = get_file_size(file_path);
        fileinfo_->key = file_path;
        fileinfo_->metadata.content = content;
        fileinfo_->metadata.size = file_size;
        return (struct cache_entry_t *) fileinfo_;
    }
    return NULL;
}

// callback to destroy cache

static void
destroy_fileinfo(struct cache_entry_t *entry,
        void *context __attribute__((unused))) {
    struct file_info_t *fileinfo = (struct file_info_t *) entry;
    if (!fileinfo) return;
    free(fileinfo->metadata.content);
    //free(fileinfo->key);
    free(fileinfo);
}

screen shot 2014-12-22 at 9 55 35 am

lpereira commented 9 years ago

From a quick glance, the code looks fine. I'll play with it tonight and see if I can reproduce the issue.

lpereira commented 9 years ago

Could not reproduce the bug here, using the JPG file you've provided and similar code (available here). From the perspective of Lwan, the code is working as it should; even strace agrees with that:

[pid 14063] writev(15, [{"HTTP/1.1 200 OK\r\nContent-Length:"..., 185}, {"\377\330\377\340\0\20JFIF\0\1\1\1\0d\0d\0\0\377\333\0C\0\1\1\1\1\1\1\1"..., 3139506}], 2) = 3139691

The transferred content is the same according to md5sum:

$ curl http://localhost:8080/static | md5sum -
1912f66032a1a6520ec9c8abf683f787  -
$ cat love.jpg | md5sum -
1912f66032a1a6520ec9c8abf683f787  -

Could you please try the linked version and see if it works? There were a few changes to the example code you've posted and some of these changes might do the trick.

whatvn commented 9 years ago

Which platform did you make that test? We clone newest lwan source code and use your gist, it fail. (we make tests on ubuntu).

lpereira commented 9 years ago

I do all my tests using an up-to-date Arch Linux box, but I'm not sure if that'll make a difference.

I've updated the gist with print statements; could you please test it again and paste the output of your program running under strace, tracing only the writev, open and lseek system calls? (Basically, strace -f -e trace=writev,lseek,open /path/to/executable). The relevant output here is this one:

[pid  1036] open("love.jpg", O_RDONLY)  = 17
[pid  1036] lseek(17, 3137536, SEEK_SET) = 3137536
[pid  1036] lseek(17, 0, SEEK_SET)      = 0
loaded from disk: key=[love.jpg], metadata.size=[3139506].
loaded from cache: key=[love.jpg], metadata.size=[3139506].
[pid  1036] writev(15, [{"HTTP/1.1 200 OK\r\nContent-Length:"..., 185}, {"\377\330\377\340\0\20JFIF\0\1\1\1\0d\0d\0\0\377\333\0C\0\1\1\1\1\1\1\1"..., 3139506}], 2) = 3139691

This is just a hunch, but might give some clue as well. Here, stat love.jpg gives me these numbers; could you please run the same command so we can compare the outputs?

$ stat love.jpg
  File: ‘love.jpg’
  Size: 3139506     Blocks: 6144       IO Block: 4096   regular file
Device: 806h/2054d  Inode: 5924815     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ leandro)   Gid: (  100/   users)
Access: 2014-12-22 20:12:33.000000000 -0200
Modify: 2013-06-11 03:32:51.000000000 -0300
Change: 2014-12-22 20:15:40.451387588 -0200
 Birth: -
whatvn commented 9 years ago

Output of strace command

[pid 29243] open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 18 [pid 29243] lseek(18, -139, SEEK_CUR) = 116 [pid 29243] open("/home/vunt/Desktop/99_Delete/Love_040998.jpg", O_RDONLY) = 18 [pid 29243] lseek(18, 3137536, SEEK_SET) = 3137536 [pid 29243] lseek(18, 0, SEEK_SET) = 0 29243 lwan-response.c:114 log_request() 127.0.0.1 "GET /thrift/Love_040998.jpg HTTP/1.1" 200 image/jpeg. [pid 29243] writev(16, [{"HTTP/1.1 200 OK\r\nContent-Length:"..., 185}, {"\377\330\377\340\0\20JFIF\0\1\1\1\0d\0d\0\0\377\333\0C\0\1\1\1\1\1\1\1"..., 3139506}], 2) = 2531840

You can see writev returns wrong content size. Output of stat command:

stat /home/vunt/Desktop/99_Delete/Love040998_.jpg File: ‘/home/vunt/Desktop/99_Delete/Love040998_.jpg’ Size: 3139506 Blocks: 6144 IO Block: 4096 regular file Device: 803h/2051d Inode: 8525471 Links: 1 Access: (0640/-rw-r-----) Uid: ( 1000/ vunt) Gid: ( 1000/ vunt) Access: 2014-12-23 09:42:10.185145824 +0700 Modify: 2014-12-23 09:41:41.253144745 +0700 Change: 2014-12-23 09:41:47.437144975 +0700 Birth: -

lpereira commented 9 years ago

That's really odd. Writes performed by writev() are supposed to be atomic. I'm surprised it's returning a value less than the sum of all parts that it should write; this makes error handling tricky.

As a last question for today: I'm testing on localhost here as I don't have another machine nearby to test; are you transferring over the network?

whatvn commented 9 years ago

I am testing on localhost too. I do this test on 2 different machines (both are Ubuntu) and it makes no difference. Can you make a test a Ubuntu?

lpereira commented 9 years ago

I know what's the problem. Will give a try at fixing it tonight.

whatvn commented 9 years ago

Thank :)

lpereira commented 9 years ago

@whatvn, could you please test Lwan with commit 65ac284 and see if the problem persists? You can just close the issue if it does.

whatvn commented 9 years ago

Confirm this patch fixed this issue.

ancuop commented 9 years ago

Hi Ipereira,

I got an issue with new version of lwan_writev() function.

When I test Lwan web server by running: http_load -rate 1000 -sencond 10 http.txt

=> I guess that new implementation of lwan_writev() may affect to CPU, could you please help me to confirm? Thank you very much.

lpereira commented 9 years ago

What's the content of your http.txt, so I can reproduce this here?

ancuop commented 9 years ago

Hi Ipereira,

Here is the content of http.txt:

http://0.0.0.0:8080/photo/Android_robot.png http://0.0.0.0:8080/photo/images.png http://0.0.0.0:8080/photo/abcd.jpg http://0.0.0.0:8080/photo/12345.png http://0.0.0.0:8080/photo/abcd.jpg http://0.0.0.0:8080/photo/12345.jpg http://0.0.0.0:8080/photo/lion.jpg http://0.0.0.0:8080/photo/3.jpg http://0.0.0.0:8080/photo/3.jpg http://0.0.0.0:8080/photo/asd.jpg http://0.0.0.0:8080/photo/Android_robot.png http://0.0.0.0:8080/photo/dfasdf.jpg http://0.0.0.0:8080/photo/download.jpg http://0.0.0.0:8080/photo/abcd.jpg http://0.0.0.0:8080/photo/asd.jpg http://0.0.0.0:8080/photo/qef.jpg http://0.0.0.0:8080/photo/flower123.jpg http://0.0.0.0:8080/photo/12345.jpg

ancuop commented 9 years ago

For your information.

After 10 seconds, http_load ends, Lwan still runs well, CPU % is normal.

Then, while Lwan is running free, I do nothing and look at CPU%. Suddenly, 4 CPU% increase to 100%.

So if I enter an url, Lwan can not return anything.

Finally, I stop Lwan and see that source code is waiting in accept4.

=> I don't know if this issue comes from writev() system call. I meant if writev has to write a buffer more than twice, there may be some issues with CPU??. I try putting "printf("loop"); after "try_again: " label in lwan_writev() function and see that:

looplooplooploop17445 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. loop17445 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. looplooploop17443 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html.

----- Thank you very much for your strong support ----- Have a nice weekend.

lpereira commented 9 years ago

Couldn't reproduce here. Could you please share the handler code as well?

The original code by @whatvn had a few issues, and if your code is based on that, that could be the reason. I've cleaned it up a bit; it should be more stable, although not completely correct (reading the file to memory, for instance, won't work if short reads happens).

However, unless you're obtaining these images from somewhere else than a file system, it's actually better to just use the built-in file serving module.

ancuop commented 9 years ago

Hi Ipereira, Please see my code in: https://gist.github.com/ancuop/55b554b547748538896b I have revised my source code to be similar to yours so that it is better and clearer for checking. (I need to disable "lwan_status_info()" in line 65, 83 because it slows down the operation and makes http_load dump core)

The issue still occurs.

ancuop commented 9 years ago

Hi Ipereira, I made some changes in lwan_writev() to use write() instead of writev(). Fortunately, today the issue no longer occurs. Could you please help to check my code and give advice? Thank you. https://gist.github.com/ancuop/00e05a52497efee80d7f

lpereira commented 9 years ago

The issue is probably the while loop that discovers which curr_iov that the short write happened. Using write() will work but it's a workaround; writev() offers reasonably better performance. I don't have much time to look at this issue this week, so if using write() works, you can just go forward with your implementation; this will eventually get fixed and you'll be able to go back to the original lwan_writev() implementation when this happens.

ancuop commented 9 years ago

Thanks for explanation. Hope that day will come soon. :) I continue discovering other operations of Lwan. It's really interesting and useful.

lpereira commented 9 years ago

@ancuop Could you please build Lwan in debug mode and run it under Valgrind? Please post the entire log here (or in a gist/pastebin).

ancuop commented 9 years ago

Hi Ipereira,

So sorry that i don't know how to use gist/pastebin. I put log in next comment.

ancuop commented 9 years ago

~/Desktop/1_Workspace/lwan$ valgrind ./lwan/lwan ==6098== Memcheck, a memory error detector ==6098== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==6098== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info ==6098== Command: ./lwan/lwan ==6098== 6098 lwan-job.c:76 lwan_job_thread_init() Initializing low priority job thread. 6098 lwan-response.c:74 lwan_response_init() Initializing default response. 6098 lwan-tables.c:40 lwan_tables_init() Uncompressing MIME type table. 6098 lwan.c:57 lwan_module_init() Initializing module registry. 6098 lwan.c:72 lwan_module_register() Registering module "serve_files". 6098 lwan.c:72 lwan_module_register() Registering module "redirect". 6098 lwan.c:72 lwan_module_register() Registering module "lua". 6098 lwan.c:333 setup_from_config() Loading configuration file: lwan.conf. 6098 lwan.c:474 lwan_init() Initializing lwan web server. 6098 lwan.c:485 lwan_init() Using 4 threads, maximum 256 sockets per thread. 6098 lwan-thread.c:392 lwan_thread_init() Initializing threads. 6101 lwan-thread.c:289 thread_io_loop() Starting IO loop on thread #2. 6100 lwan-thread.c:289 thread_io_loop() Starting IO loop on thread #1. 6102 lwan-thread.c:289 thread_io_loop() Starting IO loop on thread #3. 6103 lwan-thread.c:289 thread_io_loop() Starting IO loop on thread #4. 6098 lwan-socket.c:226 lwan_socket_init() Initializing sockets. 6098 lwan-socket.c:146 listen_addrinfo() Listening on http://0.0.0.0:8080. 6098 lwan.c:559 lwan_main_loop() Ready to serve. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. ==6098== Thread 3: ==6098== Conditional jump or move depends on uninitialised value(s) ==6098== at 0x41A9C5: lwan_writev (lwan-io-wrappers.c:120) ==6098== by 0x40DD17: lwan_response (lwan-response.c:173) ==6098== by 0x40DD96: lwan_default_response (lwan-response.c:199) ==6098== by 0x40D090: lwan_response (lwan-response.c:138) ==6098== by 0x40CB0D: lwan_process_request (lwan-request.c:759) ==6098== by 0x4154F8: process_request_coro (lwan-thread.c:153) ==6098== by 0x408BA5: coro_entry_point (lwan-coro.c:142) ==6098== 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. ==6098== Thread 6: ==6098== Conditional jump or move depends on uninitialised value(s) ==6098== at 0x41A9C5: lwan_writev (lwan-io-wrappers.c:120) ==6098== by 0x40DD17: lwan_response (lwan-response.c:173) ==6098== by 0x40CB0D: lwan_process_request (lwan-request.c:759) ==6098== by 0x4154F8: process_request_coro (lwan-thread.c:153) ==6098== by 0x408BA5: coro_entry_point (lwan-coro.c:142) ==6098== 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. ==6098== Thread 4: ==6098== Syscall param writev(vector) points to uninitialised byte(s) ==6098== at 0x5A06370: writev (writev.c:54) ==6098== by 0x41A948: lwan_writev (lwan-io-wrappers.c:104) ==6098== by 0x40DD17: lwan_response (lwan-response.c:173) ==6098== by 0x40CB0D: lwan_process_request (lwan-request.c:759) ==6098== by 0x4154F8: process_request_coro (lwan-thread.c:153) ==6098== by 0x408BA5: coro_entry_point (lwan-coro.c:142) ==6098== Address 0x8868c50 is not stack'd, malloc'd or (recently) free'd ==6098== 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. ==6098== Thread 6: ==6098== Syscall param writev(vector) points to uninitialised byte(s) ==6098== at 0x5A06370: writev (writev.c:54) ==6098== by 0x41A948: lwan_writev (lwan-io-wrappers.c:104) ==6098== by 0x40DD17: lwan_response (lwan-response.c:173) ==6098== by 0x40DD96: lwan_default_response (lwan-response.c:199) ==6098== by 0x40D090: lwan_response (lwan-response.c:138) ==6098== by 0x40CB0D: lwan_process_request (lwan-request.c:759) ==6098== by 0x4154F8: process_request_coro (lwan-thread.c:153) ==6098== by 0x408BA5: coro_entry_point (lwan-coro.c:142) ==6098== Address 0xafc8900 is not stack'd, malloc'd or (recently) free'd ==6098== 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /dfasdf.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.png HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /qef.jpg HTTP/1.0" 404 text/html. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /flower123.jpg HTTP/1.0" 404 text/html. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /abcd.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /download.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /images.png HTTP/1.0" 200 image/png. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /3.jpg HTTP/1.0" 200 image/jpeg. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /lion.jpg HTTP/1.0" 200 image/jpeg. 6102 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. 6103 lwan-response.c:111 log_request() 127.0.0.1 "GET /asd.jpg HTTP/1.0" 200 image/jpeg. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. 6101 lwan-response.c:111 log_request() 127.0.0.1 "GET /12345.jpg HTTP/1.0" 200 image/jpeg. 6100 lwan-response.c:111 log_request() 127.0.0.1 "GET /Android_robot.png HTTP/1.0" 200 image/png. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ==6098== Warning: invalid file descriptor 1031 in syscall accept() 6098 lwan.c:565 lwan_main_loop() accept: Too many open files. ^C6098 lwan.c:570 lwan_main_loop() Signal 2 (Interrupt) received. 6098 lwan.c:498 lwan_shutdown() Shutting down. 6098 lwan-job.c:95 lwan_job_thread_shutdown() Shutting down job thread. 6098 lwan-thread.c:405 lwan_thread_shutdown() Shutting down threads. 6098 lwan.c:506 lwan_shutdown() Shutting down URL handlers. 6098 lwan-response.c:83 lwan_response_shutdown() Shutting down response. 6098 lwan-cache.c:151 cache_destroy() Cache stats: 0 hits, 0 misses, 0 evictions. 6098 lwan-cache.c:151 cache_destroy() Cache stats: 260 hits, 106 misses, 8 evictions. ==6098== ==6098== HEAP SUMMARY: ==6098== in use at exit: 0 bytes in 0 blocks ==6098== total heap usage: 0 allocs, 0 frees, 0 bytes allocated ==6098== ==6098== All heap blocks were freed -- no leaks are possible ==6098== ==6098== For counts of detected and suppressed errors, rerun with: -v ==6098== Use --track-origins=yes to see where uninitialised values come from ==6098== ERROR SUMMARY: 1087 errors from 4 contexts (suppressed: 0 from 0) :~/Desktop/1_Workspace/test/Hung/lwan$

whatvn commented 9 years ago

this issue is still remain even when with latest source. I believe it happens in lwan_writev which try to handle writev fails, due to some situation (which I don't know why it fail). Writev event was blocked somewhere, I did strace lwan to figure out what happen, sometime writev can write all of iovec at the first time, but sometimes it fail. When we test with httpload, it always fail, cpu keeps running at 100% until we restart lwan.

lpereira commented 9 years ago

@whatvn I couldn't reproduce here, despite having tried it for a while. Could you please test your hypothesis by using gdb and providing a backtrace?

whatvn commented 9 years ago

Strange, I tried to reproduce this issue this morning but cannot, but I confirm it did happen several times in our environment, even with normal local file serving (not lwan_serv_files, because your module using sendfile() instead of writev).

lpereira commented 9 years ago

I'm closing the issue then. If it happens again and it turns out to be reproducible, please open another issue.