search

leo-project / leo_gateway

🚫 DEPRECATED: leo_gateway was merged into leo-project/leofs
https://github.com/leo-project/leofs/tree/master/apps/leo_gateway
Apache License 2.0
13 stars 11 forks source link

Downloading large files fails #12

Closed Licenser closed 10 years ago

Licenser commented 10 years ago

When downloading a large(ish) file i.e. ~ 150MB the gateway suffers a timeout here are some logs:

[E]     gateway_0@127.0.0.1     2013-12-03 23:57:02.488574 +0000        1386115022      leo_gateway_s3_api:handle_2/6   475     path:fifo-images/base64-13.2.1.zfs.gz, cause:{timeout,{gen_server,call,[<0.5292.0>,{get,152,{http_req,{sslsocket,{gen_tcp,#Port<0.1996>},<0.5288.0>},ranch_ssl,keepalive,<0.5289.0>,<<"GET">>,'HTTP/1.1',{{172,16,9,100},59038},<<"fifo-images.leofs.licenser.net">>,undefined,443,<<"/base64-13.2.1.zfs.gz">>,undefined,<<>>,[],[],[{<<"host">>,<<"fifo-images.leofs.licenser.net">>},{<<"date">>,<<"Tue, 03 Dec 2013 23:56:04 GMT">>},{<<"authorization">>,<<"AWS 20da8ea55424d40cae01:Dycyi/8RuarBDpDVIFavWspJsW4=">>},{<<"connection">>,<<"Keep-Alive">>},{<<"user-agent">>,<<"DragonDisk 1.05 ( http://www.dragondisk.com )">>},{<<"accept-encoding">>,<<"gzip">>}],[{<<"connection">>,[<<"keep-alive">>]}],undefined,[],waiting,undefined,<<>>,false,waiting,[],<<>>,undefined},{metadata,<<"fifo-images/base64-13.2.1.zfs.gz">>,116641657252207905397432121950179924924,32,159279359,0,1048576,152,0,48242741,1386114784066847,63553333984,281949768489412648962353822266799178366,3629116367,0}},30000]}}
[E]     gateway_0@127.0.0.1     2013-12-03 23:57:03.747205 +0000        1386115023      leo_gateway_large_object_handler:handle_loop/7  241     key:fifo-images/base64-13.2.1.zfs.gz
113, index:112, cause:closed
[E]     gateway_0@127.0.0.1     2013-12-03 23:58:03.917595 +0000        1386115083      leo_gateway_s3_api:handle_2/6   475     path:fifo-images/base64-13.2.1.zfs.gz, cause:{timeout,{gen_server,call,[<0.6029.0>,{get,152,{http_req,{sslsocket,{gen_tcp,#Port<0.1998>},<0.6025.0>},ranch_ssl,keepalive,<0.6026.0>,<<"GET">>,'HTTP/1.1',{{172,16,9,100},59045},<<"fifo-images.leofs.licenser.net">>,undefined,443,<<"/base64-13.2.1.zfs.gz">>,undefined,<<>>,[],[],[{<<"host">>,<<"fifo-images.leofs.licenser.net">>},{<<"date">>,<<"Tue, 03 Dec 2013 23:57:06 GMT">>},{<<"authorization">>,<<"AWS 20da8ea55424d40cae01:pdntMahMU7cgqm7RQxLYD0zootA=">>},{<<"connection">>,<<"Keep-Alive">>},{<<"user-agent">>,<<"DragonDisk 1.05 ( http://www.dragondisk.com )">>},{<<"accept-encoding">>,<<"gzip">>}],[{<<"connection">>,[<<"keep-alive">>]}],undefined,[],waiting,undefined,<<>>,false,waiting,[],<<>>,undefined},{metadata,<<"fifo-images/base64-13.2.1.zfs.gz">>,116641657252207905397432121950179924924,32,159279359,0,1048576,152,0,48242741,1386114784066847,63553333984,281949768489412648962353822266799178366,3629116367,0}},30000]}}
[E]     gateway_0@127.0.0.1     2013-12-03 23:58:04.678013 +0000        1386115084      leo_gateway_large_object_handler:handle_loop/7  241     key:fifo-images/base64-13.2.1.zfs.gz
129, index:128, cause:closed
[E]     gateway_0@127.0.0.1     2013-12-03 23:59:04.840107 +0000        1386115144      leo_gateway_s3_api:handle_2/6   475     path:fifo-images/base64-13.2.1.zfs.gz, cause:{timeout,{gen_server,call,[<0.6754.0>,{get,152,{http_req,{sslsocket,{gen_tcp,#Port<0.2000>},<0.6750.0>},ranch_ssl,keepalive,<0.6751.0>,<<"GET">>,'HTTP/1.1',{{172,16,9,100},59049},<<"fifo-images.leofs.licenser.net">>,undefined,443,<<"/base64-13.2.1.zfs.gz">>,undefined,<<>>,[],[],[{<<"host">>,<<"fifo-images.leofs.licenser.net">>},{<<"date">>,<<"Tue, 03 Dec 2013 23:58:07 GMT">>},{<<"authorization">>,<<"AWS 20da8ea55424d40cae01:DpVR6RXEC4DYom2MFyH1P7olDP0=">>},{<<"connection">>,<<"Keep-Alive">>},{<<"user-agent">>,<<"DragonDisk 1.05 ( http://www.dragondisk.com )">>},{<<"accept-encoding">>,<<"gzip">>}],[{<<"connection">>,[<<"keep-alive">>]}],undefined,[],waiting,undefined,<<>>,false,waiting,[],<<>>,undefined},{metadata,<<"fifo-images/base64-13.2.1.zfs.gz">>,116641657252207905397432121950179924924,32,159279359,0,1048576,152,0,48242741,1386114784066847,63553333984,281949768489412648962353822266799178366,3629116367,0}},30000]}}
[E]     gateway_0@127.0.0.1     2013-12-03 23:59:04.926386 +0000        1386115144      leo_gateway_large_object_handler:handle_loop/7  241     key:fifo-images/base64-13.2.1.zfs.gz
122, index:121, cause:closed
[E]     gateway_0@127.0.0.1     2013-12-03 23:59:16.573734 +0000        1386115156      leo_gateway_large_object_handler:handle_loop/7  241     key:fifo-images/base64-13.2.1.zfs.gz
2, index:1, cause:closed
[E]     gateway_0@127.0.0.1     2013-12-03 23:59:16.575913 +0000        1386115156      null:null       0       [leo_gateway_s3_api,handle,2,error,{case_clause,{error,closed}},[{1,64},true,{http_options,leo_gateway_s3_api,8080,8443,"./etc/server_cert.pem","./etc/server_key.pem",128,4096,inner,16,1073741824,524288000,1048576,"./cache/data","./cache/journal",300,1048576,[],[],1000,524288000,1048576,5767168}],[{socket,#Port<0.2002>},{transport,ranch_tcp},{connection,keepalive},{pid,<0.7584.0>},{method,<<"GET">>},{version,'HTTP/1.1'},{peer,{{172,16,9,100},59056}},{host,<<"fifo-images.s3.amazonaws.com">>},{host_info,undefined},{port,80},{path,<<"/base64-13.2.1.zfs.gz">>},{path_info,undefined},{qs,<<>>},{qs_vals,undefined},{bindings,[]},{headers,[{<<"host">>,<<"fifo-images.s3.amazonaws.com">>},{<<"accept-encoding">>,<<"identity">>},{<<"authorization">>,<<"AWS 20da8ea55424d40cae01:t65Q12zgRG1PqBG7c8AuBZEKX5s=">>},{<<"x-amz-date">>,<<"Tue, 03 Dec 2013 23:59:29 +0000">>}]},{p_headers,[]},{cookies,undefined},{meta,[]},{body_state,waiting},{multipart,undefined},{buffer,<<>>},{resp_compress,false},{resp_state,waiting},{resp_headers,[]},{resp_body,<<>>},{onresponse,undefined}],[{leo_gateway_s3_api,handle_2,6,[{file,"src/leo_gateway_s3_api.erl"},{line,472}]},{cowboy_handler,handler_handle,4,[{file,"src/cowboy_handler.erl"},{line,119}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,523}]}]]

When using DragonDisk the download proceeds to around 100MB and then crashes and restarts. S3Hub shows the same behaviour, s3cmd does not start with a complaint about content-lenght.

mocchira commented 10 years ago

@Licenser Thanks for your report. Since I've fixed this issue, please check it if you have any time to spare?

Licenser commented 10 years ago

Tested this again with 0.16.5 still the same results.

mocchira commented 10 years ago

Let me clarify that Tested this again with 0.16.5 means using 0.16.5 tagged branch? if so, since we still don't tag (not merged to master branch), you need to deploy leo_gateway's latest develop branch which is including this commit https://github.com/leo-project/leo_gateway/commit/76bd089d721510e464e6af0e54517ba3e353801f to fix this issue.

yosukehara commented 10 years ago

This issue was included in LeoFS v0.16.8.