search

ex-aws / ex_aws

A flexible, easy to use set of clients AWS APIs for Elixir
https://hex.pm/packages/ex_aws
MIT License
1.29k stars 529 forks source link

use with minio(upload file) #253

Closed mvalitov closed 8 years ago

mvalitov commented 8 years ago

I'm trying to use ExAws library along with minio. I try to upload the file:

filepath
|> S3.Upload.stream_file
|> S3.upload(bucket_name, path)
|> ExAws.request!

and I get an error:

[error] #PID<0.1041.0> running Myapp.Endpoint terminated
Server: localhost:4000 (http)
Request: POST /resources/upload
** (exit) an exception was raised:
    ** (MatchError) no match of right hand side value: %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<InitiateMultipartUploadResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Bucket>user-1</Bucket><Key>docs/tsetup.0.9.51.tar.xz</Key><UploadId>35e9c8b0-ac34-4411-b39b-790a0104df93</UploadId></InitiateMultipartUploadResult>", headers: [{"Accept-Ranges", "bytes"}, {"Server", "Minio/DEVELOPMENT.GOGET (linux; amd64)"}, {"Vary", "Origin"}, {"X-Amz-Request-Id", "K4O7RNNSRLAMSL26"}, {"Date", "Tue, 11 Oct 2016 20:53:58 GMT"}, {"Content-Type", "text/xml; charset=utf-8"}, {"Transfer-Encoding", "chunked"}], status_code: 200}
        (ex_aws) lib/ex_aws/s3/upload.ex:39: ExAws.S3.Upload.initialize!/2
        (ex_aws) lib/ex_aws/s3/upload.ex:80: ExAws.Operation.ExAws.S3.Upload.perform/2
        (ex_aws) lib/ex_aws.ex:41: ExAws.request!/2
        (Myapp) web/models/resource.ex:68: Myapp.Resource.put_file_to_cloud/1
        (Myapp) web/controllers/resource_controller.ex:59: Myapp.ResourceController.upload/2
        (Myapp) web/controllers/resource_controller.ex:1: Myapp.ResourceController.action/2
        (Myapp) web/controllers/resource_controller.ex:1: Myapp.ResourceController.phoenix_controller_pipeline/2
        (Myapp) lib/Myapp/endpoint.ex:1: Myapp.Endpoint.instrument/4
        (Myapp) lib/phoenix/router.ex:261: Myapp.Router.dispatch/2
        (Myapp) web/router.ex:1: Myapp.Router.do_call/2
        (Myapp) lib/Myapp/endpoint.ex:1: Myapp.Endpoint.phoenix_pipeline/1
        (Myapp) lib/plug/debugger.ex:123: Myapp.Endpoint."call (overridable 3)"/2
        (Myapp) lib/Myapp/endpoint.ex:1: Myapp.Endpoint.call/2
        (plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

I see by the response that the file upload begins. This mini returns the answer is not as expected exaws? or the problem is in something else?

Upd: after adding sweet_xml now get another error:

[error] GenServer #PID<0.593.0> terminating
** (MatchError) no match of right hand side value: nil
    (ex_aws) lib/ex_aws/s3/upload.ex:69: ExAws.S3.Upload.upload_chunk!/3
    lib/flow/materialize.ex:493: anonymous fn/4 in Experimental.Flow.Materialize.mapper/2
    lib/flow/materialize.ex:462: Experimental.Flow.Materialize."-mapper_ops/1-lists^foldl/2-1-"/3
    lib/flow/materialize.ex:462: anonymous fn/5 in Experimental.Flow.Materialize.mapper_ops/1
    lib/flow/map_reducer.ex:57: Experimental.Flow.MapReducer.handle_events/3
    lib/gen_stage.ex:2190: Experimental.GenStage.consumer_dispatch/7
    lib/gen_stage.ex:2313: Experimental.GenStage.take_pc_events/3
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_consumer", {#PID<0.592.0>, #Reference<0.0.2.533>}, [{<<255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 1, 1, 0, 72, 0, 72, 0, 0, 255, 219, 0, 67, 0, 2, 1, 1, 2, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 5, 3, 3, 3, 3, 3, 6, ...>>, 1}]}
State: {%{#Reference<0.0.2.533> => nil}, %{consumers: [{#Reference<0.0.2.544>, #Reference<0.0.2.546>}], done: [#Reference<0.0.2.533>], done?: false, trigger: #Function<2.69990182/4 in Experimental.Flow.Window.Global.materialize/4>}, {0, 4}, [], #Function<29.77971684/4 in Experimental.Flow.Materialize.mapper_ops/1>}
[error] Ranch protocol #PID<0.587.0> (:cowboy_protocol) of listener Myapp.Endpoint.HTTP terminated
** (exit) an exception was raised:
    ** (MatchError) no match of right hand side value: nil
        (ex_aws) lib/ex_aws/s3/upload.ex:69: ExAws.S3.Upload.upload_chunk!/3
        lib/flow/materialize.ex:493: anonymous fn/4 in Experimental.Flow.Materialize.mapper/2
        lib/flow/materialize.ex:462: Experimental.Flow.Materialize."-mapper_ops/1-lists^foldl/2-1-"/3
        lib/flow/materialize.ex:462: anonymous fn/5 in Experimental.Flow.Materialize.mapper_ops/1
        lib/flow/map_reducer.ex:57: Experimental.Flow.MapReducer.handle_events/3
        lib/gen_stage.ex:2190: Experimental.GenStage.consumer_dispatch/7
        lib/gen_stage.ex:2313: Experimental.GenStage.take_pc_events/3
        (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

at the same time, I see that the download is in wireshark trace, but in the storage folder is empty

mvalitov commented 8 years ago

in stackoverflow on I was told that the response from the minio lacks etag parameter. I would like to clarify whether this is so?

benwilson512 commented 8 years ago

I'm afraid I have no idea. The Upload code is pretty simple, I recommend just adapting it for whatever minio wants. It is not the goal of this library to support services other than AWS.

mvalitov commented 8 years ago

sorry, I did not want to ask you because of this function change, I just wanted to clarify whether the upload function expects a reply option etag

benwilson512 commented 8 years ago

Ah my bad. Yes it expects an etag here: https://github.com/CargoSense/ex_aws/blob/master/lib/ex_aws/s3/upload.ex#L69

This is because complete_multipart_upload requires each part number have an associated etag.

mvalitov commented 8 years ago

thanks for the answer. I looked more at the response from Minio, and I see that there is etag:

Frame 1138: 344 bytes on wire (2752 bits), 344 bytes captured (2752 bits) on interface 0
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
Transmission Control Protocol, Src Port: 9000 (9000), Dst Port: 42324 (42324), Seq: 515, Ack: 1539815, Len: 278
Hypertext Transfer Protocol
    HTTP/1.1 200 OK\r\n
    Accept-Ranges: bytes\r\n
    Etag: "2501de281894f7a1f79e8e0d6b8e7d85"\r\n
    Server: Minio/DEVELOPMENT.GOGET (linux; amd64)\r\n
    Vary: Origin\r\n
    X-Amz-Request-Id: M81ULLKCXKH0KRXD\r\n
    Date: Fri, 14 Oct 2016 15:00:16 GMT\r\n
    Content-Length: 0\r\n
    Content-Type: text/plain; charset=utf-8\r\n
    \r\n
    [HTTP response 2/2]
    [Time since request: 0.008850143 seconds]
    [Prev request in frame: 1016]
    [Prev response in frame: 1028]
    [Request in frame: 1032]
benwilson512 commented 8 years ago

Can you add an |> IO.inspect after |> ExAws.request!(config) found here https://github.com/CargoSense/ex_aws/blob/master/lib/ex_aws/s3/upload.ex#L67 and paste the result?

mvalitov commented 8 years ago

I added a line to this file myapp/deps/ex_aws/lib/ex_aws/s3/upload.ex:

  def upload_chunk!({chunk, i}, op, config) do
    %{headers: headers} = ExAws.S3.upload_part(op.bucket, op.path, op.upload_id, i, chunk, op.opts)
    |> ExAws.request!(config)
    |> IO.inspect

    {_, etag} = List.keyfind(headers, "ETag", 0)
    {i, etag}
  end

but in the log does not appear any new lines:

[info] POST /resources/upload
[debug] Processing by Myapp.ResourceController.upload/2
  Parameters: %{"_csrf_token" => "GSITBRoBVw5ZDQsSJwhCDCpbCTIbNgAAmAaBtS0A1GnzUK8MOjPGnA==", "_utf8" => "✓", "resource" => %{"file" => %Plug.Upload{content_type: "audio/mp3", filename: "test-test.mp3", path: "/tmp/plug-1476/multipart-460254-944553-4"}, "parent_id" => "16"}}
  Pipelines: [:browser]
[debug] QUERY OK source="resources" db=1.3ms
SELECT r0."id", r0."name", r0."path", r0."type", r0."user_id", r0."parent_id", r0."inserted_at", r0."updated_at" FROM "resources" AS r0 WHERE (r0."id" = $1) [16]
[error] GenServer #PID<0.422.0> terminating
** (MatchError) no match of right hand side value: nil
    (ex_aws) lib/ex_aws/s3/upload.ex:69: ExAws.S3.Upload.upload_chunk!/3
    lib/flow/materialize.ex:493: anonymous fn/4 in Experimental.Flow.Materialize.mapper/2
    lib/flow/materialize.ex:462: Experimental.Flow.Materialize."-mapper_ops/1-lists^foldl/2-1-"/3
    lib/flow/materialize.ex:462: anonymous fn/5 in Experimental.Flow.Materialize.mapper_ops/1
    lib/flow/map_reducer.ex:57: Experimental.Flow.MapReducer.handle_events/3
    lib/gen_stage.ex:2190: Experimental.GenStage.consumer_dispatch/7
    lib/gen_stage.ex:2313: Experimental.GenStage.take_pc_events/3
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_consumer", {#PID<0.420.0>, #Reference<0.0.4.5401>}, [{<<138, 7, 167, 1, 242, 29, 102, 129, 226, 190, 48, 110, 144, 100, 96, 65, 157, 44, 136, 106, 88, 115, 35, 139, 233, 124, 30, 75, 145, 62, 58, 84, 68, 184, 202, 106, 47, 229, 57, 170, 158, 37, 91, 74, 170, 207, 112, ...>>, 3}]}
State: {%{#Reference<0.0.4.5401> => nil}, %{consumers: [{#Reference<0.0.4.5409>, #Reference<0.0.4.5412>}], done: [#Reference<0.0.4.5401>], done?: false, trigger: #Function<2.69990182/4 in Experimental.Flow.Window.Global.materialize/4>}, {1, 4}, [], #Function<29.77971684/4 in Experimental.Flow.Materialize.mapper_ops/1>}
[error] Ranch protocol #PID<0.414.0> (:cowboy_protocol) of listener Myapp.Endpoint.HTTP terminated
** (exit) an exception was raised:
    ** (MatchError) no match of right hand side value: nil
        (ex_aws) lib/ex_aws/s3/upload.ex:69: ExAws.S3.Upload.upload_chunk!/3
        lib/flow/materialize.ex:493: anonymous fn/4 in Experimental.Flow.Materialize.mapper/2
        lib/flow/materialize.ex:462: Experimental.Flow.Materialize."-mapper_ops/1-lists^foldl/2-1-"/3
        lib/flow/materialize.ex:462: anonymous fn/5 in Experimental.Flow.Materialize.mapper_ops/1
        lib/flow/map_reducer.ex:57: Experimental.Flow.MapReducer.handle_events/3
        lib/gen_stage.ex:2190: Experimental.GenStage.consumer_dispatch/7
        lib/gen_stage.ex:2313: Experimental.GenStage.take_pc_events/3
        (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

I edited the correct file?

benwilson512 commented 8 years ago

Did you mix deps.compile ex_aws after making the change?

mvalitov commented 8 years ago

hmm, that I did not.

%{body: "",
  headers: [{"Accept-Ranges", "bytes"},
   {"Etag", "\"2501de281894f7a1f79e8e0d6b8e7d85\""},
   {"Server", "Minio/DEVELOPMENT.GOGET (linux; amd64)"}, {"Vary", "Origin"},
   {"X-Amz-Request-Id", "EXM9T3WYIDR3YHN9"},
   {"Date", "Fri, 14 Oct 2016 15:57:02 GMT"}, {"Content-Length", "0"},
   {"Content-Type", "text/plain; charset=utf-8"}], status_code: 200}
[error] GenServer #PID<0.422.0> terminating
** (MatchError) no match of right hand side value: nil
    (ex_aws) lib/ex_aws/s3/upload.ex:70: ExAws.S3.Upload.upload_chunk!/3
    lib/flow/materialize.ex:493: anonymous fn/4 in Experimental.Flow.Materialize.mapper/2
    lib/flow/materialize.ex:462: Experimental.Flow.Materialize."-mapper_ops/1-lists^foldl/2-1-"/3
    lib/flow/materialize.ex:462: anonymous fn/5 in Experimental.Flow.Materialize.mapper_ops/1
    lib/flow/map_reducer.ex:57: Experimental.Flow.MapReducer.handle_events/3
    lib/gen_stage.ex:2190: Experimental.GenStage.consumer_dispatch/7
    lib/gen_stage.ex:2313: Experimental.GenStage.take_pc_events/3
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_consumer", {#PID<0.420.0>, #Reference<0.0.1.3778>}, [{<<138, 7, 167, 1, 242, 29, 102, 129, 226, 190, 48, 110, 144, 100, 96, 65, 157, 44, 136, 106, 88, 115, 35, 139, 233, 124, 30, 75, 145, 62, 58, 84, 68, 184, 202, 106, 47, 229, 57, 170, 158, 37, 91, 74, 170, 207, 112, ...>>, 3}]}
State: {%{#Reference<0.0.1.3778> => nil}, %{consumers: [{#Reference<0.0.1.3786>, #Reference<0.0.1.3789>}], done: [#Reference<0.0.1.3778>], done?: false, trigger: #Function<2.69990182/4 in Experimental.Flow.Window.Global.materialize/4>}, {1, 4}, [], #Function<29.77971684/4 in Experimental.Flow.Materialize.mapper_ops/1>}
[error] Ranch protocol #PID<0.414.0> (:cowboy_protocol) of listener Myapp.Endpoint.HTTP terminated
** (exit) an exception was raised:
    ** (MatchError) no match of right hand side value: nil
        (ex_aws) lib/ex_aws/s3/upload.ex:70: ExAws.S3.Upload.upload_chunk!/3
        lib/flow/materialize.ex:493: anonymous fn/4 in Experimental.Flow.Materialize.mapper/2
        lib/flow/materialize.ex:462: Experimental.Flow.Materialize."-mapper_ops/1-lists^foldl/2-1-"/3
        lib/flow/materialize.ex:462: anonymous fn/5 in Experimental.Flow.Materialize.mapper_ops/1
        lib/flow/map_reducer.ex:57: Experimental.Flow.MapReducer.handle_events/3
        lib/gen_stage.ex:2190: Experimental.GenStage.consumer_dispatch/7
        lib/gen_stage.ex:2313: Experimental.GenStage.take_pc_events/3
        (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
benwilson512 commented 8 years ago

ah it's a capitalization issue. S3 does "ETag" and minion does "Etag"`. I'll put in some code to make it case insensitive.

mvalitov commented 8 years ago

many thanks!

benwilson512 commented 8 years ago

Can you try on master?

mvalitov commented 8 years ago

Yes, I checked, uploading works perfectly!

benwilson512 commented 8 years ago

Awesome, glad we could sort it out :)

harshavardhana commented 8 years ago

ah it's a capitalization issue. S3 does "ETag" and minion does "Etag"`. I'll put in some code to make it case insensitive.

HTTP headers are case in-sensitive so "ETag" or "Etag" or "eTag" mean the same thing. Just adding my two cents.

Thanks for fixing @benwilson512

benwilson512 commented 8 years ago

I am well aware. That doesn't mean the code that checked for the header was however. Now it is.