search

ClusterHQ / flocker

Container data volume manager for your Dockerized application
https://clusterhq.com
Apache License 2.0
3.39k stars 290 forks source link

Failed to bind the following fields: MaxResults = 100 #2802

Open gufranmmu opened 8 years ago

gufranmmu commented 8 years ago

I have changed the endpoint url from flocker code and using it with eucalyptus. When i execute flocker-dataset-agent I get the following error:

{"exception": "botocore.exceptions.ClientError", "aws_request_id": "Failed to bind the following fields:\nMaxResults = 100\n", "timestamp": 1464160012.080899, "aws_message": "Failed to bind the following fields:\nMaxResults = 100\n", "reason": "An error occurred (400 Bad Request) when calling the DescribeVolumes operation: Failed to bind the following fields:\nMaxResults = 100\n", "action_status": "failed", "aws_code": "400 Bad Request", "task_uuid": "994ea742-4560-4ef0-902a-658b4309a831", "action_type": "flocker:node:agents:blockdevice:aws", "task_level": [2, 2, 3, 3]}
{"exception": "botocore.exceptions.ClientError", "aws_request_id": "Failed to bind the following fields:\nMaxResults = 100\n", "timestamp": 1464160012.081313, "aws_message": "Failed to bind the following fields:\nMaxResults = 100\n", "reason": "An error occurred (400 Bad Request) when calling the DescribeVolumes operation: Failed to bind the following fields:\nMaxResults = 100\n", "action_status": "failed", "aws_code": "400 Bad Request", "task_uuid": "994ea742-4560-4ef0-902a-658b4309a831", "action_type": "flocker:agent:discovery", "task_level": [2, 2, 4]}
{"exception": "botocore.exceptions.ClientError", "aws_request_id": "Failed to bind the following fields:\nMaxResults = 100\n", "timestamp": 1464160012.081592, "traceback": "Traceback: <class 'botocore.exceptions.ClientError'>: An error occurred (400 Bad Request) when calling the DescribeVolumes operation: Failed to bind the following fields:\nMaxResults = 100\n\n/opt/flocker/lib/python2.7/site-packages/machinist/_logging.py:102:receive\n/opt/flocker/lib/python2.7/site-packages/machinist/_fsm.py:534:receive\n/opt/flocker/lib/python2.7/site-packages/machinist/_fsm.py:592:output\n/opt/flocker/lib/python2.7/site-packages/flocker/node/_loop.py:529:output_CONVERGE\n--- <exception caught here> ---\n/opt/flocker/lib/python2.7/site-packages/twisted/internet/defer.py:150:maybeDeferred\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1798:discover_state\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1345:_count_calls\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1744:_discover_raw_state\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:1221:list_volumes\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:561:_run_with_logging\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:1086:_list_ebs_volumes\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:1085:<genexpr>\n/opt/flocker/lib/python2.7/site-packages/boto3/resources/collection.py:166:pages\n/opt/flocker/lib/python2.7/site-packages/botocore/paginate.py:85:__iter__\n/opt/flocker/lib/python2.7/site-packages/botocore/paginate.py:157:_make_request\n/opt/flocker/lib/python2.7/site-packages/botocore/client.py:310:_api_call\n/opt/flocker/lib/python2.7/site-packages/botocore/client.py:407:_make_api_call\n", "aws_message": "Failed to bind the following fields:\nMaxResults = 100\n", "reason": "An error occurred (400 Bad Request) when calling the DescribeVolumes operation: Failed to bind the following fields:\nMaxResults = 100\n", "aws_code": "400 Bad Request", "task_uuid": "994ea742-4560-4ef0-902a-658b4309a831", "message_type": "eliot:traceback", "task_level": [2, 3]}
{"task_uuid": "994ea742-4560-4ef0-902a-658b4309a831", "log_level": "INFO", "timestamp": 1464160012.081742, "message": "Intentionally delaying the next iteration of the convergence loop to avoid RequestLimitExceeded.", "message_type": "flocker:node:_loop:delay", "current_wait": 6.4, "task_level": [2, 4]}
{"delay": 6.4, "task_uuid": "994ea742-4560-4ef0-902a-658b4309a831", "log_level": "INFO", "timestamp": 1464160012.081893, "message": "Delaying until next convergence loop.", "message_type": "flocker:node:_loop:CONVERGE:delay", "task_level": [2, 5]}
{"fsm_identifi
wallnerryan commented 8 years ago

@gufranmmu this is untested configuration. Feel free to debug and submit PR for the support of eucalyptus.

gufranmmu commented 8 years ago

Thanks! Where do you think the error is coming from? I am out of ideas at this point.

gufranmmu commented 8 years ago

{"exception": "botocore.exceptions.ClientError", "aws_request_id": "Failed to bind the following fields:\nMaxResults = 100\n", "timestamp": 1464540797.651654, "traceback": "Traceback: <class 'botocore.exceptions.ClientError'>: An error occurred (400 Bad Request) when calling the DescribeVolumes operation: Failed to bind the following fields:\nMaxResults = 100\n\n/opt/flocker/lib/python2.7/site-packages/machinist/_logging.py:102:receive\n/opt/flocker/lib/python2.7/site-packages/machinist/_fsm.py:534:receive\n/opt/flocker/lib/python2.7/site-packages/machinist/_fsm.py:592:output\n/opt/flocker/lib/python2.7/site-packages/flocker/node/_loop.py:529:output_CONVERGE\n--- ---\n/opt/flocker/lib/python2.7/site-packages/twisted/internet/defer.py:150:maybeDeferred\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1798:discover_state\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1345:_count_calls\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1744:_discover_raw_state\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:1223:list_volumes\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:563:_run_with_logging\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:1088:_list_ebs_volumes\n/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:1087:\n/opt/flocker/lib/python2.7/site-packages/boto3/resources/collection.py:166:pages\n/opt/flocker/lib/python2.7/site-packages/botocore/paginate.py:85:iter\n/opt/flocker/lib/python2.7/site-packages/botocore/paginate.py:157:_make_request\n/opt/flocker/lib/python2.7/site-packages/botocore/client.py:310:_api_call\n/opt/flocker/lib/python2.7/site-packages/botocore/client.py:407:_make_api_call\n", "aws_message": "Failed to bind the following fields:\nMaxResults = 100\n", "reason": "An error occurred (400 Bad Request) when calling the DescribeVolumes operation: Failed to bind the following fields:\nMaxResults = 100\n", "aws_code": "400 Bad Request", "task_uuid": "0250e330-a1d5-45e4-8a9c-539d5a380523", "message_type": "eliot:traceback", "task_level": [2, 3]}

wallnerryan commented 8 years ago

cc @wallrj

wallnerryan commented 8 years ago

@gufranmmu ... @wallrj said It looks like the Eucalyptus API might not support result paging but can't find the API reference. May want to look that the calls coming from flocker are supported by Eucalyptus.

wallrj commented 8 years ago

Here's the formatted traceback:

In [5]: print x['traceback']
Traceback: : An error occurred (400 Bad Request) when calling the DescribeVolumes operation: Failed to bind the following fields:
MaxResults = 100

/opt/flocker/lib/python2.7/site-packages/machinist/logging.py:102:receive
/opt/flocker/lib/python2.7/site-packages/machinist/fsm.py:534:receive
/opt/flocker/lib/python2.7/site-packages/machinist/_fsm.py:592:output
/opt/flocker/lib/python2.7/site-packages/flocker/node/_loop.py:529:output_CONVERGE
--- ---
/opt/flocker/lib/python2.7/site-packages/twisted/internet/defer.py:150:maybeDeferred
/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1798:discover_state
/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1345:_count_calls
/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/blockdevice.py:1744:_discover_raw_state
/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:1223:list_volumes
/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:563:_run_with_logging
/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:1088:_list_ebs_volumes
/opt/flocker/lib/python2.7/site-packages/flocker/node/agents/ebs.py:1087:
/opt/flocker/lib/python2.7/site-packages/boto3/resources/collection.py:166:pages
/opt/flocker/lib/python2.7/site-packages/botocore/paginate.py:85:__iter
/opt/flocker/lib/python2.7/site-packages/botocore/paginate.py:157:_make_request
/opt/flocker/lib/python2.7/site-packages/botocore/client.py:310:_api_call
/opt/flocker/lib/python2.7/site-packages/botocore/client.py:407:_make_api_call

It looks like Eucalyptus doesn't support result paging.

I couldn't find any reported issues though:

@gufranmmu Let us know what you find.

gufranmmu commented 8 years ago

Thanks for the reply. I removed page limit 

        return list(itertools.chain.from_iterable(list(
             volumes for volumes in
             self.connection.volumes.pages()
         )))

and when i execute flocker-dataset-agent, i see the existing volumes now:

{"count": 1, "function": "_discover_raw_state", "task_uuid": "b681faf8-1006-4180-8817-d709f8998e55", "timestamp": 1464790099.763917, "message_type": "flocker:node:agents:blockdevice:list_volumes", "task_level": [5, 2, 2, 2, 2, 2, 2]} {"count": 1, "task_uuid": "b681faf8-1006-4180-8817-d709f8998e55", "task_level": [5, 2, 2, 2, 2, 2, 4, 1], "action_type": "flocker:node:agents:blockdevice:aws", "timestamp": 1464790099.785461, "operation": ["_list_ebs_volumes", [], {}], "action_status": "started"} {"timestamp": 1464790099.786701, "task_uuid": "b681faf8-1006-4180-8817-d709f8998e55", "message_type": "flocker:node:agents:blockdevice:aws:boto_logs", "message": "Calling paginated ec2:describe_volumes with {}", "task_level": [5, 2, 2, 2, 2, 2, 4, 2]} {"timestamp": 1464790099.834783, "task_uuid": "b681faf8-1006-4180-8817-d709f8998e55", "volume_ids": ["vol-43ce574a", "vol-64990c28", "vol-bccd7701", "vol-9def03fc", "vol-b5b3fadc", "vol-755a4cb1", "vol-e250fdc9", "vol-76bf9b35", "vol-90a65b48", "vol-e3490ad6", "vol-fbc5ba53", "vol-4334994b", "vol-48579a6b", "vol-b171564c", "vol-44c10342", "vol-cce3f4ed"], "message_type": "flocker:node:agents:blockdevice:aws:boto_result:listed_volumes", "task_level": [5, 2, 2, 2, 2, 2, 5]} {"timestamp": 1464790099.83498, "task_uuid": "b681faf8-1006-4180-8817-d709f8998e55", "volume_ids": [], "message_type": "flocker:node:agents:blockdevice:aws:boto_result:listed_cluster_volumes", "task_level": [5, 2, 2, 2, 2, 2, 6]} {"timestamp": 1464790099.963368, "task_uuid": "b681faf8-1006-4180-8817-d709f8998e55", "message_type": "agent:blockdevice:raw_state", "raw_state": "RawState(compute_instance_id=u'i-eef18725', devices_with_filesystems=FilepathPSet([]), devices=UuidToFilepathPMap({}), system_mounts=FilepathToFilepathPMap({FilePath('/dev/vda1'): FilePath('/'), FilePath('/dev/vdb'): FilePath('/ephemeral')}), volumes=BlockdevicevolumePVector([]), _live_instances=UnicodePSet([u'i-c5e221f5', u'i-e5ecd9dd', u'i-4d306bbb', u'i-09eca163', u'i-df5bef42', u'i-d0bd1673', u'i-0c5c3001', u'i-116653d8', u'i-91f76591', u'i-843209f7', u'i-4e797979', u'i-6e8cb27c', u'i-d6b8b1b2', u'i-094c174b', u'i-caa8ae78', u'i-5334c1d3', u'i-fa4b4262', u'i-1990ebfc', u'i-89d66145', u'i-e9151828', u'i-800f3e05', u'i-cf446c27', u'i-a25b92db', u'i-7fb442fa', u'i-3ce4dce6', u'i-fff2c992', u'i-c83aad6b', u'i-68fd7f2f', u'i-e359db19', u'i-28174e17', u'i-e55d4e8b', u'i-33f52dcf', u'i-4074611b', u'i-651ca973', u'i-0fca3dfb', u'i-5224d160', u'i-a45ab42f', u'i-bd0fa716', u'i-5d39b6f2', u'i-7dbe357a', u'i-7931d5bd', u'i-3dc95ac8', u'i-92a19316', u'i-b7a79a26', u'i-d57eecde', u'i-391b9869', u'i-cc85c6c6', u'i-ae6dd0f6', u'i-2d52f8d3', u'i-0fe510b1', u'i-c846def5', u'i-9adb3696', u'i-5274ee4f', u'i-f5e634bb', u'i-4748e408', u'i-e53147d3', u'i-9c8b1929', u'i-f268bb91', u'i-d26612ab', u'i-51f2d9c1', u'i-66f641eb', u'i-81e03ebe', u'i-592206e5', u'i-3d636759', u'i-0c9ec32e', u'i-0841eba6', u'i-eef18725']))", "task_level": [5, 2, 2, 2, 2, 2, 8]}

However, when i see new error:

[root@mesos-slave-1 ~]# docker run -ti -v volumename:/data --volume-driver=flocker busybox sh
Unable to find image 'busybox:latest' locally
Trying to pull repository docker.io/library/busybox ... latest: Pulling from library/busybox

597f79aa54ea: Pull complete
0d380282e68b: Pull complete
Digest: sha256:4a731fb46adc5cefe3ae374a8b6020fc1b6ad667a279647766e9a3cd89f6fa92
Status: Downloaded newer image for docker.io/busybox:latest

Error response from daemon: Cannot start container d7c5a69f1dfbcb7574bad83e03e3925c70676f2483e87101527a1448a320c016: TimeoutError: User timeout caused connection failure.
You have new mail in /var/spool/mail/root
gufranmmu commented 8 years ago

I have been finally able to modify the code for eucalyptus support and it is working for me. What are the requirements for pull request?