ibm-research / swifthlm

SwiftHLM - a middleware for using OpenStack Swift with tape and other hight latency media storage backends
Apache License 2.0
14 stars 4 forks source link

swift GET&HEAD requests hang after adding hlm to the pipeline in proxy-server.conf #10

Closed tommyJin closed 7 years ago

tommyJin commented 7 years ago

Hi there,

As the title says, I encounter the issue that when I try to run curl -v -X GET -H 'X-Auth-Token: ${token}' http://127.0.0.1:8080/v1/AUTH_test/con1/obj1 or curl -v -X HEAD -H 'X-Auth-Token: ${token}' http://127.0.0.1:8080/v1/AUTH_test/con1/obj1

the terminal seems to hang forever and never process to the next step like returning the metadata or the content of the object.

Basic info: python: 2.7 swift: 2.5.0(Liberty) swifthlm: 0.2.2 master branch(before 2017/06/20 submit) system: CentOS7

Is it possible that the middleware call function suspends or something else pauses the request? Cause the request would response if I remove the 'hlm' from the pipeline in proxy-server.conf.

By using swift client, I find that it is the swift issue not the hlm issue. Sorry about it.

Best regards, Tommy

slavisasarafijanovic commented 7 years ago

Hi Tommy,

in case of GET, SwiftHLM middleware invokes a function to check the object status. Normally the GET request should not get paused or suspended, because SwiftHLM status check is a synchronous call. Upon the synchronous object status check, if the object is present on LLM (low latency media) normal processing of GET is continued, else the error code 412 (Precondition Failed) is returned to the user/application with the error message "Object needs to be RECALL-ed before it can be accessed". However we observed the status check hang in some cases due to Paramiko select.select() and channel.exit_status_ready() seemed to not behave as expected resulting in status call hanging. This was discovered and fixed by Harald in https://github.com/ibm-research/swifthlm/commit/f0cfa289e6047c09ea53155551eaa83cf6a06805 about two weeks ago. Given that you use SwiftHLM version from June, there is a good likelihood that you catched this bug, so I suggest you to try same curl commands using the most recent SwiftHLM code.

Regards, Slavisa

tommyJin commented 7 years ago

Hi @slavisasarafijanovic ,

It is my fault that I describe wrongly about the response of 'GET' and 'HEAD'. The 'GET' request returns an error, and we are working on checking status stuff. But the 'HEAD' request hangs until I quit by pressing 'CTRL + C'. In this case, I'd like to run curl -v -X HEAD -H 'X-Auth-Token: ${token}' http://127.0.0.1:8080/v1/AUTH_test/con1/obj1 to get the metadata of the object, but the hanging stops me from getting the metadata.

The output of the hang is:

[osddev@localhost ~]$ curl -v -X HEAD -H 'X-Auth-Token: AUTH_tk84c518327150471097ad06ab76c6e4c0' http://127.0.0.1:8080/v1/AUTH_test/con1/obj1

  • About to connect() to 127.0.0.1 port 8080 (#0)
  • Trying 127.0.0.1...
  • Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)

    HEAD /v1/AUTH_test/con1/obj1 HTTP/1.1 User-Agent: curl/7.29.0 Host: 127.0.0.1:8080 Accept: / X-Auth-Token: AUTH_tk84c518327150471097ad06ab76c6e4c0

    < HTTP/1.1 200 OK < Content-Length: 2100736 < Accept-Ranges: bytes < Last-Modified: Wed, 30 Aug 2017 06:12:46 GMT < Etag: 564c5061dfe524fc384dc22301b59e93 < X-Timestamp: 1504073565.39965 < Content-Type: application/octet-stream < X-Trans-Id: tx654d225d03564db8b9bf2-0059a6577b < Date: Wed, 30 Aug 2017 06:13:15 GMT <

(The cursor is always flashing here)

Currently I'd like to retrieve the metadata of an object for status check through HTTP. But right now, I could only use swift client to retrieve the metadata like swift -A 'http://127.0.0.1:8080/auth/v1.0' -U 'test:tester' -K 'testing' stat -v --os-storage-url http://127.0.0.1:8080/v1/AUTH_test/con2/obj0

Any hint would be helpful, thanks in advance!

Best regards, Tommy

tommyJin commented 7 years ago

Hi all,

Problem solved. I use program to retrieve the response, and everything works fine.

BTW, the curl also could get the metadata by curl -X HEAD -I -H 'X-Auth-Token: ${token}' http://127.0.0.1:8080/vv1/AUTH_test/con2/obj0

Thanks everyone.

Best wishes, Tommy