search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.49k stars 2.82k forks source link

[Bug]: Search failed with error `code=503, message=failed to search: segment lacks[segment=445961929941983497]: channel not available[channel=by-dev-rootcoord-xxxv1` after querynode pod failure chaos test #28815

Closed zhuwenxing closed 2 months ago

zhuwenxing commented 9 months ago

Is there an existing issue for this?

Environment

- Milvus version:2.3-20231128-ccca932c
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka): pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2023-11-28T22:47:25.866Z] [2023-11-28 22:45:58 - DEBUG - ci_test]: (api_response) : <Collection>:

[2023-11-28T22:47:25.866Z] -------------

[2023-11-28T22:47:25.866Z] <name>: Checker__8eRtCX95

[2023-11-28T22:47:25.866Z] <description>: 

[2023-11-28T22:47:25.866Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}......  (api_request.py:37)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:45:58 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:45:59 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:45:59 - DEBUG - ci_test]: (api_request)  : [Collection.compact] args: [180], kwargs: {} (api_request.py:62)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:45:59 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:17 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:17 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:17 - INFO - ci_test]: assert create collection: 0.4452815055847168, init_entities: 84175 (test_all_collections_after_chaos.py:49)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:17 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [[[-3000, -2999, -2998, -2997, -2996, -2995, -2994, -2993, -2992, -2991, -2990, -2989, -2988, -2987, -2986, -2985, -2984, -2983, -2982, -2981, -2980, -2979, -2978, -2977, -2976, -2975, -2974, -2973, -2972, -2971, -2970, -2969, -2968, -2967, -2966, -2965, -2964, -2963, -2962, -2961, -2960, -2959, -29......, kwargs: {'timeout': 180} (api_request.py:62)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:17 - DEBUG - ci_test]: (api_response) : (insert count: 2000, delete count: 0, upsert count: 0, timestamp: 445962407847133188, success count: 2000, err count: 0)  (api_request.py:37)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:17 - INFO - ci_test]: assert insert: 0.2441849708557129 (test_all_collections_after_chaos.py:61)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:17 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:20 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:20 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:21 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:21 - INFO - ci_test]: assert flush: 3.0340681076049805, entities: 86175 (test_all_collections_after_chaos.py:71)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:21 - INFO - ci_test]: index info: [{'collection': 'Checker__8eRtCX95', 'field': 'float_vector', 'index_name': 'index__CX2USSXd', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}] (test_all_collections_after_chaos.py:87)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:21 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)

[2023-11-28T22:47:25.866Z] [2023-11-28 22:46:21 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-11-28T22:47:25.867Z] [2023-11-28 22:46:21 - DEBUG - ci_test]: (api_request)  : [Collection.search] args: [[[0.06384376956948457, 0.11954292737424162, 0.10490028098231843, 0.017139506883689218, 0.0786578225805758, 0.14829803376057202, 0.12920088662784723, 0.11029315268789419, 0.08731686761364627, 0.06311001002481748, 0.008008766185680518, 0.11851682896380743, 0.05814297879738317, 0.02873089591404046, 0......., kwargs: {} (api_request.py:62)

[2023-11-28T22:47:25.867Z] [2023-11-28 22:46:21 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=503, message=failed to search: segment lacks[segment=445961929941983497]: channel not available[channel=by-dev-rootcoord-dml_10_445961929942182564v1])>, <Time:{'RPC start': '2023-11-28 22:46:21.463205', 'RPC error': '2023-11-28 22:46:21.465949'}> (decorators.py:128)

[2023-11-28T22:47:25.867Z] [2023-11-28 22:46:21 - ERROR - ci_test]: Traceback (most recent call last):

[2023-11-28T22:47:25.867Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper

[2023-11-28T22:47:25.867Z]     res = func(*args, **_kwargs)

[2023-11-28T22:47:25.867Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request

[2023-11-28T22:47:25.867Z]     return func(*arg, **kwargs)

[2023-11-28T22:47:25.867Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 782, in search

[2023-11-28T22:47:25.867Z]     resp = conn.search(

[2023-11-28T22:47:25.867Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 129, in handler

[2023-11-28T22:47:25.867Z]     raise e from e

[2023-11-28T22:47:25.867Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 125, in handler

[2023-11-28T22:47:25.867Z]     return func(*args, **kwargs)

[2023-11-28T22:47:25.867Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 164, in handler

[2023-11-28T22:47:25.867Z]     return func(self, *args, **kwargs)

[2023-11-28T22:47:25.867Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 104, in handler

[2023-11-28T22:47:25.867Z]     raise e from e

[2023-11-28T22:47:25.867Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 68, in handler

[2023-11-28T22:47:25.867Z]     return func(*args, **kwargs)

[2023-11-28T22:47:25.867Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 781, in search

[2023-11-28T22:47:25.867Z]     return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)

[2023-11-28T22:47:25.867Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 742, in _execute_search

[2023-11-28T22:47:25.867Z]     raise e from e

[2023-11-28T22:47:25.867Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 734, in _execute_search

[2023-11-28T22:47:25.867Z]     raise MilvusException(response.status.code, response.status.reason)

[2023-11-28T22:47:25.867Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=503, message=failed to search: segment lacks[segment=445961929941983497]: channel not available[channel=by-dev-rootcoord-dml_10_445961929942182564v1])>

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-for-release-cron/detail/chaos-test-for-release-cron/8470/pipeline

log:

artifacts-querynode-pod-failure-8470-pytest-logs.tar.gz

artifacts-querynode-pod-failure-8470-server-logs.tar.gz

Anything else?

4am chaos-testing

[2023-11-28T22:45:40.231Z] + kubectl get pods -o wide

[2023-11-28T22:45:40.232Z] + grep querynode-pod-failure-8470

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-etcd-0                                 1/1     Running            0                31m     10.104.21.12    4am-node24   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-etcd-1                                 1/1     Running            0                31m     10.104.20.157   4am-node22   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-etcd-2                                 1/1     Running            0                31m     10.104.16.78    4am-node21   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-datacoord-6f9559cdfb-v8fnm      1/1     Running            0                31m     10.104.14.210   4am-node18   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-datanode-6f4d5d8b49-d7b24       1/1     Running            0                31m     10.104.14.214   4am-node18   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-datanode-6f4d5d8b49-k7d6m       1/1     Running            0                31m     10.104.24.164   4am-node29   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-indexcoord-588db44d9c-d6gcd     1/1     Running            0                31m     10.104.14.215   4am-node18   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-indexnode-58566f4488-g6f9s      1/1     Running            0                31m     10.104.14.216   4am-node18   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-indexnode-58566f4488-v4q7c      1/1     Running            0                31m     10.104.24.165   4am-node29   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-indexnode-58566f4488-vtvxn      1/1     Running            0                31m     10.104.1.61     4am-node10   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-proxy-7dcbcbd5f9-2qkg9          1/1     Running            0                31m     10.104.21.237   4am-node24   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-querycoord-c8bc69598-whfvs      1/1     Running            0                31m     10.104.14.209   4am-node18   <none>           <none>

[2023-11-28T22:45:40.487Z] querynode-pod-failure-8470-milvus-querynode-89dc6468d-c8h2c       1/1     Running            3 (10m ago)      31m     10.104.24.163   4am-node29   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-milvus-querynode-89dc6468d-sqr52       1/1     Running            3                31m     10.104.14.211   4am-node18   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-milvus-querynode-89dc6468d-xmxnt       1/1     Running            3 (10m ago)      31m     10.104.21.234   4am-node24   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-milvus-rootcoord-67c865f784-w4kw2      1/1     Running            0                31m     10.104.14.208   4am-node18   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-minio-0                                1/1     Running            0                31m     10.104.24.179   4am-node29   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-minio-1                                1/1     Running            0                31m     10.104.21.10    4am-node24   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-minio-2                                1/1     Running            0                31m     10.104.20.158   4am-node22   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-minio-3                                1/1     Running            0                31m     10.104.16.79    4am-node21   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-pulsar-bookie-0                        1/1     Running            0                31m     10.104.24.180   4am-node29   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-pulsar-bookie-1                        1/1     Running            0                31m     10.104.21.11    4am-node24   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-pulsar-bookie-init-2s5hv               0/1     Completed          0                31m     10.104.14.207   4am-node18   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-pulsar-broker-0                        1/1     Running            0                31m     10.104.21.238   4am-node24   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-pulsar-proxy-0                         1/1     Running            0                31m     10.104.14.212   4am-node18   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-pulsar-pulsar-init-77qp2               0/1     Completed          0                31m     10.104.14.206   4am-node18   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-pulsar-recovery-0                      1/1     Running            0                31m     10.104.21.235   4am-node24   <none>           <none>

[2023-11-28T22:45:40.488Z] querynode-pod-failure-8470-pulsar-zookeeper-0                     1/1     Running            0                31m     10.104.24.181   4am-node29   <none>           <none>
yanliang567 commented 9 months ago

/assign @congqixia /unassign

congqixia commented 9 months ago

Same like #28857, could you please verify? /assign @zhuwenxing

zhuwenxing commented 9 months ago

It still reproduced in 2.3-20231205-3d7bcb5e-amd64


[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:35 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:38 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:38 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:38 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:38 - INFO - ci_test]: assert flush: 3.036179542541504, entities: 5000 (test_all_collections_after_chaos.py:71)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:39 - INFO - ci_test]: index info: [{'collection': 'SearchChecker__WDMottvy', 'field': 'float_vector', 'index_name': 'index__vqpJ58XD', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}] (test_all_collections_after_chaos.py:87)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:39 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:41 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:41 - DEBUG - ci_test]: (api_request)  : [Collection.search] args: [[[0.12861101059653923, 0.0073168426706904524, 0.09216541040468074, 0.07176322089673397, 0.09538922667491329, 0.03271475734423939, 0.0783028088693867, 0.07102629789407297, 0.14599988454247495, 0.016754264535591545, 0.1441581835040213, 0.13765911133929298, 0.03877683399555063, 0.02677501214728548, 0......., kwargs: {} (api_request.py:62)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:41 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=503, message=failed to search: segment lacks[segment=446119342078766866]: channel not available[channel=by-dev-rootcoord-dml_5_446119342078766854v0])>, <Time:{'RPC start': '2023-12-05 21:38:41.827668', 'RPC error': '2023-12-05 21:38:41.833847'}> (decorators.py:128)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:41 - ERROR - ci_test]: Traceback (most recent call last):

[2023-12-05T21:39:43.253Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper

[2023-12-05T21:39:43.253Z]     res = func(*args, **_kwargs)

[2023-12-05T21:39:43.253Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request

[2023-12-05T21:39:43.253Z]     return func(*arg, **kwargs)

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 782, in search

[2023-12-05T21:39:43.253Z]     resp = conn.search(

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 129, in handler

[2023-12-05T21:39:43.253Z]     raise e from e

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 125, in handler

[2023-12-05T21:39:43.253Z]     return func(*args, **kwargs)

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 164, in handler

[2023-12-05T21:39:43.253Z]     return func(self, *args, **kwargs)

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 104, in handler

[2023-12-05T21:39:43.253Z]     raise e from e

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 68, in handler

[2023-12-05T21:39:43.253Z]     return func(*args, **kwargs)

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 748, in search

[2023-12-05T21:39:43.253Z]     return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 709, in _execute_search

[2023-12-05T21:39:43.253Z]     raise e from e

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 702, in _execute_search

[2023-12-05T21:39:43.253Z]     check_status(response.status)

[2023-12-05T21:39:43.253Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/utils.py", line 54, in check_status

[2023-12-05T21:39:43.253Z]     raise MilvusException(status.code, status.reason, status.error_code)

[2023-12-05T21:39:43.253Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=503, message=failed to search: segment lacks[segment=446119342078766866]: channel not available[channel=by-dev-rootcoord-dml_5_446119342078766854v0])>

[2023-12-05T21:39:43.253Z]  (api_request.py:45)

[2023-12-05T21:39:43.253Z] [2023-12-05 21:38:41 - ERROR - ci_test]: (api_response) : <MilvusException: (code=503, message=failed to search: segment lacks[segment=446119342078766866]: channel not available[channel=by-dev-rootcoord-dml_5_446119342078766854v0])> (api_request.py:46)

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-for-release-cron/detail/chaos-test-for-release-cron/8596/pipeline

log: artifacts-querynode-pod-kill-8596-server-logs.tar.gz artifacts-querynode-pod-kill-8596-pytest-logs.tar.gz

zhuwenxing commented 9 months ago

/unassign /assign @congqixia

congqixia commented 9 months ago

my mistake, fix pr still blocked by ut issue

28956, will assign back after merged

zhuwenxing commented 9 months ago

Not reproduced in 2.3-20231208-ecbb6160

congqixia commented 9 months ago

@zhuwenxing Got it. Thanks!

zhuwenxing commented 9 months ago

It reproduced again in 2.3-20231212-286dce0d-amd64 failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-for-release-cron/detail/chaos-test-for-release-cron/8792/pipeline log:

artifacts-querynode-pod-kill-8792-pytest-logs.tar.gz

artifacts-querynode-pod-kill-8792-server-logs.tar.gz

zhuwenxing commented 9 months ago

This issue is still reproduced in 2.3-20231215-97d71c25 failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-kafka-for-release-cron/detail/chaos-test-kafka-for-release-cron/9157/pipeline log: artifacts-querynode-pod-kill-9157-pytest-logs.tar.gz artifacts-querynode-pod-kill-9157-server-logs.tar.gz

Two test cases failed: one still is segment lacks, but another one is failed to search: node offline[node=18]

congqixia commented 9 months ago

distribution went wrong after querynode chaos there was lots of set aciton generated but no trace found in querynode could you please help investigate this /assign @yah01

ThreadDao commented 7 months ago

@yah01 reproduce: image: master-20240130-fda6f8d0 https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-kafka-cron/detail/chaos-test-kafka-cron/11091/pipeline/289

[2024-01-31T21:26:44.257Z] [2024-01-31 21:25:35 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=503, message=failed to search: segment lacks[segment=447410166423819541]; segment lacks[segment=447410166425624829]: channel not available[channel=by-dev-rootcoord-dml_11_447410166424219065v0])>, <Time:{'RPC start': '2024-01-31 21:25:35.575838', 'RPC error': '2024-01-31 21:25:35.579349'}> (decorators.py:134)
congqixia commented 2 months ago

@zhuwenxing Seems this issue it not up to date I'd close this issue, please file a new one if same problem reoccurred