search

sebdah / dynamic-dynamodb

Dynamic DynamoDB provides auto scaling for AWS DynamoDB
http://dynamic-dynamodb.readthedocs.org/
Apache License 2.0
618 stars 124 forks source link

Service crashed after running for several months #318

Closed ghost closed 7 years ago

ghost commented 7 years ago

Hello

I have had dynamic-dynamoDb running for several months but last night around 1 am the service stopped running with an error. This is the error which occurred, I had to manually restart the service in the morning. So far it has carried on running fine.

Thanks

Phil

2016-12-15 01:16:15,154 - dynamic-dynamodb - INFO - webservice_log_v2 - Consumed read units: 0.00% 2016-12-15 01:16:15,204 - dynamic-dynamodb - INFO - webservice_log_v2 - Consumed write units: 2.60% 2016-12-15 01:16:15,293 - dynamic-dynamodb - INFO - webservice_log_v2 - Consumed read units: 0.00% 2016-12-15 01:16:15,302 - dynamic-dynamodb - INFO - webservice_log_v2 - Read throttle count: 0 2016-12-15 01:16:15,354 - dynamic-dynamodb - INFO - webservice_log_v2 - Throttled read percent by provision: 0.00% 2016-12-15 01:16:15,372 - dynamic-dynamodb - INFO - webservice_log_v2 - Throttled read percent by consumption: 0.00% 2016-12-15 01:16:15,372 - dynamic-dynamodb - INFO - webservice_log_v2 - Scaling down reads is not done when usage is at 0% 2016-12-15 01:16:15,373 - dynamic-dynamodb - INFO - webservice_log_v2 - Consecutive read checks 0/1 2016-12-15 01:16:15,464 - dynamic-dynamodb - INFO - webservice_log_v2 - Consumed write units: 2.60% 2016-12-15 01:16:15,473 - dynamic-dynamodb - INFO - webservice_log_v2 - Write throttle count: 0 2016-12-15 01:16:15,523 - dynamic-dynamodb - INFO - webservice_log_v2 - Throttled write percent by provision: 0.00% 2016-12-15 01:16:15,540 - dynamic-dynamodb - INFO - webservice_log_v2 - Throttled write percent by consumption: 0.00% 2016-12-15 01:16:15,540 - dynamic-dynamodb - INFO - webservice_log_v2 - Consecutive write checks 0/1 2016-12-15 01:16:15,540 - dynamic-dynamodb - INFO - webservice_log_v2 - No need to change provisioning 2016-12-15 01:24:49,753 - dynamic-dynamodb - ERROR - [Errno -2] Name or service not known Traceback (most recent call last): File "/usr/local/lib/python2.6/site-packages/dynamic_dynamodb/init.py", line 50, in run execute() File "/usr/local/lib/python2.6/site-packages/dynamic_dynamodb/init.py", line 109, in execute for table_name, table_key in sorted(dynamodb.get_tables_and_gsis()): File "/usr/local/lib/python2.6/site-packages/dynamic_dynamodb/aws/dynamodb.py", line 31, in get_tables_and_gsis for table_instance in list_tables(): File "/usr/local/lib/python2.6/site-packages/dynamic_dynamodb/aws/dynamodb.py", line 222, in list_tables table_list = DYNAMODB_CONNECTION.list_tables() File "/usr/local/lib/python2.6/site-packages/boto/dynamodb2/layer1.py", line 1124, in list_tables body=json.dumps(params)) File "/usr/local/lib/python2.6/site-packages/boto/dynamodb2/layer1.py", line 2842, in make_request retry_handler=self._retry_handler) File "/usr/local/lib/python2.6/site-packages/boto/connection.py", line 1030, in _mexe raise ex gaierror: [Errno -2] Name or service not known

mkaurin commented 7 years ago

gai errors usually have to do with ipv6 configuration on your OS. IIRC, what usually happens is that your DNS server gives you the AAAA record (ipv6), and you only have ipv4 connectivity.

I would take a look at this http://askubuntu.com/questions/32298/prefer-a-ipv4-dns-lookups-before-aaaaipv6-lookups

And also man gai-conf

Very likely that this is an OS/gai issue

sebdah commented 7 years ago

Closing as too old.