search

noobaa / noobaa-operator

Operator for NooBaa - object data service for hybrid and multi cloud environments :cloud: :wrench:
https://www.noobaa.io
Apache License 2.0
103 stars 101 forks source link

Avoid upload errors under load #148

Open ron1 opened 4 years ago

ron1 commented 4 years ago

Environment: OCP 3.11, Rook Ceph 1.1.6 block provisioner, NooBaa 2.0.8-SNAPSHOT including PR #146

Pre-req: Configure two pv-pool BackingStores with one volume each wrapped in a Mirror BucketClass. Then define a StorageClass for this BucketClass and create an OBC. Use rclone to upload a large Maven repository to NooBaa.

Expected Result: Files are successfully uploaded.

Actual Result: After 60 seconds, 500 Internal Errors are reported. A snippet from the nuxeo-core-0 log is included below.

Nov-17 2:41:47.812 [Endpoint/7386]    [L0] core.endpoint.s3.s3_rest:: S3 REQUEST PUT /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=4776ad1d4fa5f11cb129a8b0176934770eea65a0af30cf89b535d027a0bf8c66 op put_object request_id k32e999f-7c2bg2-1658 { 'user-agent': 'rclone/v1.50.1', 'content-length': '2961', 'content-md5': '1HaxAQf4LAZ8yndVgGXLkQ==', 'content-type': 'application/octet-stream', 'x-amz-acl': 'private', 'x-amz-meta-mtime': '1561668647.101045613', 'accept-encoding': 'gzip', host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https', forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'x-forwarded-for': '10.113.245.89' }
Nov-17 2:41:47.812 [Endpoint/7386]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom  
Nov-17 2:41:47.813 [Endpoint/7386]    [L0] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar search X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime headers { 'User-Agent': 'rclone/v1.50.1', 'Content-Length': '5884', 'Content-MD5': 'O7Ryw7fVk2cQpT7e3ngo9g==', 'Content-Type': 'application/x-java-archive', 'X-Amz-Acl': 'private', 'X-Amz-Meta-Mtime': '1561671637', 'Accept-Encoding': 'gzip', Host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https', Forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'X-Forwarded-For': '10.113.245.89' } region us-east-1 canonical_str 
PUT
/cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime
content-md5:O7Ryw7fVk2cQpT7e3ngo9g==
content-type:application/x-java-archive
host:internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com
x-amz-acl:private
x-amz-meta-mtime:1561671637

content-md5;content-type;host;x-amz-acl;x-amz-meta-mtime
UNSIGNED-PAYLOAD
 string_to_sign 
AWS4-HMAC-SHA256
20191117T024147Z
20191117/us-east-1/s3/aws4_request
78a4167dc147fd643ac58997295aa4dc227a5fc575bb53219fd6a67375c38cb9

Nov-17 2:41:47.813 [Endpoint/7386]    [L0] core.endpoint.s3.s3_rest:: S3 REQUEST PUT /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=46eaffa34e98d033dd05b384a3c70dbaa050343a6ac7e423b2fd81ca584ff582 op put_object request_id k32e999h-7cshjn-dek { 'user-agent': 'rclone/v1.50.1', 'content-length': '5884', 'content-md5': 'O7Ryw7fVk2cQpT7e3ngo9g==', 'content-type': 'application/x-java-archive', 'x-amz-acl': 'private', 'x-amz-meta-mtime': '1561671637', 'accept-encoding': 'gzip', host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https', forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'x-forwarded-for': '10.113.245.89' }
Nov-17 2:41:47.814 [Endpoint/7386]    [L0] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom search X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime headers { 'User-Agent': 'rclone/v1.50.1', 'Content-Length': '11061', 'Content-MD5': 'dhU6DJJgmAaIeseMwJEGJg==', 'Content-Type': 'application/octet-stream', 'X-Amz-Acl': 'private', 'X-Amz-Meta-Mtime': '1561670048.487738559', 'Accept-Encoding': 'gzip', Host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https', Forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'X-Forwarded-For': '10.113.245.89' } region us-east-1 canonical_str 
PUT
/cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime
content-md5:dhU6DJJgmAaIeseMwJEGJg==
content-type:application/octet-stream
host:internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com
x-amz-acl:private
x-amz-meta-mtime:1561670048.487738559

content-md5;content-type;host;x-amz-acl;x-amz-meta-mtime
UNSIGNED-PAYLOAD
 string_to_sign 
AWS4-HMAC-SHA256
20191117T024147Z
20191117/us-east-1/s3/aws4_request
37a318ea388cf02305649db1a31bf4641e6969d04fbe02a2141320b2dc3f761a

Nov-17 2:41:47.814 [Endpoint/7386]    [L0] core.endpoint.s3.s3_rest:: S3 REQUEST PUT /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=9bee70a959048ec5b1e10bd16d7a891fb988159d633b056a05ed412d0664c45a op put_object request_id k32e999h-7d9059-18lz { 'user-agent': 'rclone/v1.50.1', 'content-length': '11061', 'content-md5': 'dhU6DJJgmAaIeseMwJEGJg==', 'content-type': 'application/octet-stream', 'x-amz-acl': 'private', 'x-amz-meta-mtime': '1561670048.487738559', 'accept-encoding': 'gzip', host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https', forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'x-forwarded-for': '10.113.245.89' }
Nov-17 2:41:47.814 [Endpoint/7386]    [L0] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar search X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime headers { 'User-Agent': 'rclone/v1.50.1', 'Content-Length': '7839', 'Content-MD5': '6VYLTYkYu5v8rwo9L7ISnw==', 'Content-Type': 'application/x-java-archive', 'X-Amz-Acl': 'private', 'X-Amz-Meta-Mtime': '1562330375', 'Accept-Encoding': 'gzip', Host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https', Forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'X-Forwarded-For': '10.113.245.89' } region us-east-1 canonical_str 
PUT
/cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime
content-md5:6VYLTYkYu5v8rwo9L7ISnw==
content-type:application/x-java-archive
host:internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com
x-amz-acl:private
x-amz-meta-mtime:1562330375

content-md5;content-type;host;x-amz-acl;x-amz-meta-mtime
UNSIGNED-PAYLOAD
 string_to_sign 
AWS4-HMAC-SHA256
20191117T024147Z
20191117/us-east-1/s3/aws4_request
4eac0807679db85b7c60a8540b7f9cf3c93677b35c261a04c76704d404d8fe12

Nov-17 2:41:47.815 [Endpoint/7386]    [L0] core.endpoint.s3.s3_rest:: S3 REQUEST PUT /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=bb4c60cb6f378cb8d75d609d6e315c745422945119edaa18689bc992ace9bfdb op put_object request_id k32e999i-7dochq-t18 { 'user-agent': 'rclone/v1.50.1', 'content-length': '7839', 'content-md5': '6VYLTYkYu5v8rwo9L7ISnw==', 'content-type': 'application/x-java-archive', 'x-amz-acl': 'private', 'x-amz-meta-mtime': '1562330375', 'accept-encoding': 'gzip', host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https', forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'x-forwarded-for': '10.113.245.89' }
Nov-17 2:41:47.815 [Endpoint/7386]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar  
Nov-17 2:41:47.815 [Endpoint/7386]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom  
Nov-17 2:41:47.815 [Endpoint/7386]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar  
Nov-17 2:41:47.815 [Endpoint/7386]    [L0] core.sdk.object_io:: upload_object: start upload { bucket: 'cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb', key: 'org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom', content_type: 'application/octet-stream', size: 2961, md5_b64: '1HaxAQf4LAZ8yndVgGXLkQ==', sha256_b64: undefined, xattr: { mtime: '1561668647.101045613' }, tagging: undefined, encryption: undefined }
Nov-17 2:41:47.815 [Endpoint/7386]    [L0] core.sdk.object_io:: upload_object: start upload { bucket: 'cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb', key: 'org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar', content_type: 'application/x-java-archive', size: 5884, md5_b64: 'O7Ryw7fVk2cQpT7e3ngo9g==', sha256_b64: undefined, xattr: { mtime: '1561671637' }, tagging: undefined, encryption: undefined }
Nov-17 2:41:47.815 [Endpoint/7386]    [L0] core.sdk.object_io:: upload_object: start upload { bucket: 'cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb', key: 'org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom', content_type: 'application/octet-stream', size: 11061, md5_b64: 'dhU6DJJgmAaIeseMwJEGJg==', sha256_b64: undefined, xattr: { mtime: '1561670048.487738559' }, tagging: undefined, encryption: undefined }
Nov-17 2:41:47.816 [Endpoint/7386]    [L0] core.sdk.object_io:: upload_object: start upload { bucket: 'cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb', key: 'org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar', content_type: 'application/x-java-archive', size: 7839, md5_b64: '6VYLTYkYu5v8rwo9L7ISnw==', sha256_b64: undefined, xattr: { mtime: '1562330375' }, tagging: undefined, encryption: undefined }
Nov-17 2:41:47.816 [Endpoint/7386]    [L0] core.server.object_services.object_server:: create_object_upload: { bucket: SENSITIVE-f9daa3039fc23465a3fc578d1ba291f6, key: 'org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom', content_type: 'application/octet-stream', size: 2961, md5_b64: '1HaxAQf4LAZ8yndVgGXLkQ==', xattr: { mtime: '1561668647.101045613' } }
Nov-17 2:41:47.816 [Endpoint/7386]    [L0] core.server.object_services.object_server:: create_object_upload: { bucket: SENSITIVE-f9daa3039fc23465a3fc578d1ba291f6, key: 'org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar', content_type: 'application/x-java-archive', size: 5884, md5_b64: 'O7Ryw7fVk2cQpT7e3ngo9g==', xattr: { mtime: '1561671637' } }
Nov-17 2:41:47.817 [Endpoint/7386]    [L0] core.server.object_services.object_server:: create_object_upload: { bucket: SENSITIVE-f9daa3039fc23465a3fc578d1ba291f6, key: 'org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom', content_type: 'application/octet-stream', size: 11061, md5_b64: 'dhU6DJJgmAaIeseMwJEGJg==', xattr: { mtime: '1561670048.487738559' } }
Nov-17 2:41:47.817 [Endpoint/7386]    [L0] core.server.object_services.object_server:: create_object_upload: { bucket: SENSITIVE-f9daa3039fc23465a3fc578d1ba291f6, key: 'org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar', content_type: 'application/x-java-archive', size: 7839, md5_b64: '6VYLTYkYu5v8rwo9L7ISnw==', xattr: { mtime: '1562330375' } }
Nov-17 2:41:47.819 [Endpoint/7386] [ERROR] CONSOLE:: RPC._on_request: ERROR srv object_api.create_object_upload reqid 110636@fcall://fcall(3eyc2rfw0) connid fcall://fcall(3eyc2rfw0) MongoError: no connection available for operation and number of stored operation > 0
    at Function.create (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/error.js:43:12)
    at Store.add (/root/node_modules/noobaa-core/node_modules/mongodb/lib/topologies/topology_base.js:38:18)
    at disconnectHandler (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/topologies/server.js:258:30)
    at Server.insert (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/topologies/server.js:653:7)
    at Server.insert (/root/node_modules/noobaa-core/node_modules/mongodb/lib/topologies/topology_base.js:321:25)
    at insertDocuments (/root/node_modules/noobaa-core/node_modules/mongodb/lib/operations/collection_ops.js:853:19)
    at insertOne (/root/node_modules/noobaa-core/node_modules/mongodb/lib/operations/collection_ops.js:883:3)
    at /root/node_modules/noobaa-core/node_modules/mongodb/lib/utils.js:433:24
    at Promise._execute (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/debuggability.js:313:9)
    at Promise._resolveFromExecutor (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/promise.js:488:18)
    at new Promise (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/promise.js:79:10)
    at executeOperation (/root/node_modules/noobaa-core/node_modules/mongodb/lib/utils.js:428:10)
    at Collection.insertOne (/root/node_modules/noobaa-core/node_modules/mongodb/lib/collection.js:463:10)
    at MDStore.insert_object (/root/node_modules/noobaa-core/src/server/object_services/md_store.js:124:36)
    at Object.create_object_upload (/root/node_modules/noobaa-core/src/server/object_services/object_server.js:84:30)
    at process._tickCallback (internal/process/next_tick.js:68:7)
Nov-17 2:41:47.819 [Endpoint/7386] [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv object_api.create_object_upload params { bucket: SENSITIVE-f9daa3039fc23465a3fc578d1ba291f6, key: 'org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom', content_type: 'application/octet-stream', size: 2961, md5_b64: '1HaxAQf4LAZ8yndVgGXLkQ==', sha256_b64: undefined, xattr: { mtime: '1561668647.101045613' }, tagging: undefined, encryption: undefined } reqid 110636@fcall://fcall(3eyc2rfw0) took [3.0+0.8=3.8] Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
Nov-17 2:41:47.819 [Endpoint/7386] [ERROR] CONSOLE:: RPC._on_request: ERROR srv object_api.create_object_upload reqid 110637@fcall://fcall(3eyc2rfw0) connid fcall://fcall(3eyc2rfw0) MongoError: no connection available for operation and number of stored operation > 0
    at Function.create (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/error.js:43:12)
    at Store.add (/root/node_modules/noobaa-core/node_modules/mongodb/lib/topologies/topology_base.js:38:18)
    at disconnectHandler (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/topologies/server.js:258:30)
    at Server.insert (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/topologies/server.js:653:7)
    at Server.insert (/root/node_modules/noobaa-core/node_modules/mongodb/lib/topologies/topology_base.js:321:25)
    at insertDocuments (/root/node_modules/noobaa-core/node_modules/mongodb/lib/operations/collection_ops.js:853:19)
    at insertOne (/root/node_modules/noobaa-core/node_modules/mongodb/lib/operations/collection_ops.js:883:3)
    at /root/node_modules/noobaa-core/node_modules/mongodb/lib/utils.js:433:24
    at Promise._execute (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/debuggability.js:313:9)
    at Promise._resolveFromExecutor (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/promise.js:488:18)
    at new Promise (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/promise.js:79:10)
    at executeOperation (/root/node_modules/noobaa-core/node_modules/mongodb/lib/utils.js:428:10)
    at Collection.insertOne (/root/node_modules/noobaa-core/node_modules/mongodb/lib/collection.js:463:10)
    at MDStore.insert_object (/root/node_modules/noobaa-core/src/server/object_services/md_store.js:124:36)
    at Object.create_object_upload (/root/node_modules/noobaa-core/src/server/object_services/object_server.js:84:30)
Nov-17 2:41:47.819 [Endpoint/7386] [ERROR] CONSOLE:: RPC._on_request: ERROR srv object_api.create_object_upload reqid 110638@fcall://fcall(3eyc2rfw0) connid fcall://fcall(3eyc2rfw0) MongoError: no connection available for operation and number of stored operation > 0
    at Function.create (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/error.js:43:12)
    at Store.add (/root/node_modules/noobaa-core/node_modules/mongodb/lib/topologies/topology_base.js:38:18)
    at disconnectHandler (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/topologies/server.js:258:30)
    at Server.insert (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/topologies/server.js:653:7)
    at Server.insert (/root/node_modules/noobaa-core/node_modules/mongodb/lib/topologies/topology_base.js:321:25)
    at insertDocuments (/root/node_modules/noobaa-core/node_modules/mongodb/lib/operations/collection_ops.js:853:19)
    at insertOne (/root/node_modules/noobaa-core/node_modules/mongodb/lib/operations/collection_ops.js:883:3)
    at /root/node_modules/noobaa-core/node_modules/mongodb/lib/utils.js:433:24
    at Promise._execute (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/debuggability.js:313:9)
    at Promise._resolveFromExecutor (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/promise.js:488:18)
    at new Promise (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/promise.js:79:10)
    at executeOperation (/root/node_modules/noobaa-core/node_modules/mongodb/lib/utils.js:428:10)
    at Collection.insertOne (/root/node_modules/noobaa-core/node_modules/mongodb/lib/collection.js:463:10)
    at MDStore.insert_object (/root/node_modules/noobaa-core/src/server/object_services/md_store.js:124:36)
    at Object.create_object_upload (/root/node_modules/noobaa-core/src/server/object_services/object_server.js:84:30)
Nov-17 2:41:47.819 [Endpoint/7386] [ERROR] CONSOLE:: RPC._on_request: ERROR srv object_api.create_object_upload reqid 110639@fcall://fcall(3eyc2rfw0) connid fcall://fcall(3eyc2rfw0) MongoError: no connection available for operation and number of stored operation > 0
    at Function.create (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/error.js:43:12)
    at Store.add (/root/node_modules/noobaa-core/node_modules/mongodb/lib/topologies/topology_base.js:38:18)
    at disconnectHandler (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/topologies/server.js:258:30)
    at Server.insert (/root/node_modules/noobaa-core/node_modules/mongodb-core/lib/topologies/server.js:653:7)
    at Server.insert (/root/node_modules/noobaa-core/node_modules/mongodb/lib/topologies/topology_base.js:321:25)
    at insertDocuments (/root/node_modules/noobaa-core/node_modules/mongodb/lib/operations/collection_ops.js:853:19)
    at insertOne (/root/node_modules/noobaa-core/node_modules/mongodb/lib/operations/collection_ops.js:883:3)
    at /root/node_modules/noobaa-core/node_modules/mongodb/lib/utils.js:433:24
    at Promise._execute (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/debuggability.js:313:9)
    at Promise._resolveFromExecutor (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/promise.js:488:18)
    at new Promise (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/promise.js:79:10)
    at executeOperation (/root/node_modules/noobaa-core/node_modules/mongodb/lib/utils.js:428:10)
    at Collection.insertOne (/root/node_modules/noobaa-core/node_modules/mongodb/lib/collection.js:463:10)
    at MDStore.insert_object (/root/node_modules/noobaa-core/src/server/object_services/md_store.js:124:36)
    at Object.create_object_upload (/root/node_modules/noobaa-core/src/server/object_services/object_server.js:84:30)
Nov-17 2:41:47.820 [Endpoint/7386]  [WARN] core.sdk.object_io:: upload_object: failed upload { bucket: 'cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb', key: 'org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom', md_conditions: undefined } { Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23) rpc_code: 'INTERNAL', rpc_data: { retryable: true } }
Nov-17 2:41:47.820 [Endpoint/7386] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20191117T024147Z&amp;X-Amz-Expires=900&amp;X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&amp;X-Amz-Signature=4776ad1d4fa5f11cb129a8b0176934770eea65a0af30cf89b535d027a0bf8c66</Resource><RequestId>k32e999f-7c2bg2-1658</RequestId></Error> PUT /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.pom?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=4776ad1d4fa5f11cb129a8b0176934770eea65a0af30cf89b535d027a0bf8c66 {"user-agent":"rclone/v1.50.1","content-length":"2961","content-md5":"1HaxAQf4LAZ8yndVgGXLkQ==","content-type":"application/octet-stream","x-amz-acl":"private","x-amz-meta-mtime":"1561668647.101045613","accept-encoding":"gzip","host":"internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com","x-forwarded-host":"internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=","x-forwarded-for":"10.113.245.89"} Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
Nov-17 2:41:47.820 [Endpoint/7386] [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv object_api.create_object_upload params { bucket: SENSITIVE-f9daa3039fc23465a3fc578d1ba291f6, key: 'org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar', content_type: 'application/x-java-archive', size: 5884, md5_b64: 'O7Ryw7fVk2cQpT7e3ngo9g==', sha256_b64: undefined, xattr: { mtime: '1561671637' }, tagging: undefined, encryption: undefined } reqid 110637@fcall://fcall(3eyc2rfw0) took [3.3+1.5=4.8] Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
Nov-17 2:41:47.821 [Endpoint/7386] [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv object_api.create_object_upload params { bucket: SENSITIVE-f9daa3039fc23465a3fc578d1ba291f6, key: 'org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom', content_type: 'application/octet-stream', size: 11061, md5_b64: 'dhU6DJJgmAaIeseMwJEGJg==', sha256_b64: undefined, xattr: { mtime: '1561670048.487738559' }, tagging: undefined, encryption: undefined } reqid 110638@fcall://fcall(3eyc2rfw0) took [3.4+1.3=4.7] Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
Nov-17 2:41:47.821 [Endpoint/7386] [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv object_api.create_object_upload params { bucket: SENSITIVE-f9daa3039fc23465a3fc578d1ba291f6, key: 'org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar', content_type: 'application/x-java-archive', size: 7839, md5_b64: '6VYLTYkYu5v8rwo9L7ISnw==', sha256_b64: undefined, xattr: { mtime: '1562330375' }, tagging: undefined, encryption: undefined } reqid 110639@fcall://fcall(3eyc2rfw0) took [3.5+1.2=4.7] Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
Nov-17 2:41:47.821 [Endpoint/7386]  [WARN] core.sdk.object_io:: upload_object: failed upload { bucket: 'cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb', key: 'org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar', md_conditions: undefined } { Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23) rpc_code: 'INTERNAL', rpc_data: { retryable: true } }
Nov-17 2:41:47.821 [Endpoint/7386]  [WARN] core.sdk.object_io:: upload_object: failed upload { bucket: 'cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb', key: 'org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom', md_conditions: undefined } { Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23) rpc_code: 'INTERNAL', rpc_data: { retryable: true } }
Nov-17 2:41:47.821 [Endpoint/7386]  [WARN] core.sdk.object_io:: upload_object: failed upload { bucket: 'cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb', key: 'org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar', md_conditions: undefined } { Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23) rpc_code: 'INTERNAL', rpc_data: { retryable: true } }
Nov-17 2:41:47.821 [Endpoint/7386] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20191117T024147Z&amp;X-Amz-Expires=900&amp;X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&amp;X-Amz-Signature=46eaffa34e98d033dd05b384a3c70dbaa050343a6ac7e423b2fd81ca584ff582</Resource><RequestId>k32e999h-7cshjn-dek</RequestId></Error> PUT /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/nuxeo-segmentio-connector/10.10/nuxeo-segmentio-connector-10.10-test-sources.jar?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=46eaffa34e98d033dd05b384a3c70dbaa050343a6ac7e423b2fd81ca584ff582 {"user-agent":"rclone/v1.50.1","content-length":"5884","content-md5":"O7Ryw7fVk2cQpT7e3ngo9g==","content-type":"application/x-java-archive","x-amz-acl":"private","x-amz-meta-mtime":"1561671637","accept-encoding":"gzip","host":"internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com","x-forwarded-host":"internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=","x-forwarded-for":"10.113.245.89"} Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
Nov-17 2:41:47.821 [Endpoint/7386] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20191117T024147Z&amp;X-Amz-Expires=900&amp;X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&amp;X-Amz-Signature=9bee70a959048ec5b1e10bd16d7a891fb988159d633b056a05ed412d0664c45a</Resource><RequestId>k32e999h-7d9059-18lz</RequestId></Error> PUT /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7.pom?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=9bee70a959048ec5b1e10bd16d7a891fb988159d633b056a05ed412d0664c45a {"user-agent":"rclone/v1.50.1","content-length":"11061","content-md5":"dhU6DJJgmAaIeseMwJEGJg==","content-type":"application/octet-stream","x-amz-acl":"private","x-amz-meta-mtime":"1561670048.487738559","accept-encoding":"gzip","host":"internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com","x-forwarded-host":"internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=","x-forwarded-for":"10.113.245.89"} Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
Nov-17 2:41:47.822 [Endpoint/7386] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20191117T024147Z&amp;X-Amz-Expires=900&amp;X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&amp;X-Amz-Signature=bb4c60cb6f378cb8d75d609d6e315c745422945119edaa18689bc992ace9bfdb</Resource><RequestId>k32e999i-7dochq-t18</RequestId></Error> PUT /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/ecm/nuxeo-diff-test/10.10-HF09/nuxeo-diff-test-10.10-HF09-sources.jar?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=bb4c60cb6f378cb8d75d609d6e315c745422945119edaa18689bc992ace9bfdb {"user-agent":"rclone/v1.50.1","content-length":"7839","content-md5":"6VYLTYkYu5v8rwo9L7ISnw==","content-type":"application/x-java-archive","x-amz-acl":"private","x-amz-meta-mtime":"1562330375","accept-encoding":"gzip","host":"internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com","x-forwarded-host":"internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=","x-forwarded-for":"10.113.245.89"} Error: no connection available for operation and number of stored operation > 0
    at RpcRequest._set_response (/root/node_modules/noobaa-core/src/rpc/rpc_request.js:163:26)
    at RPC._on_response (/root/node_modules/noobaa-core/src/rpc/rpc.js:412:32)
    at RPC._on_message (/root/node_modules/noobaa-core/src/rpc/rpc.js:748:22)
    at RpcFcallConnection.conn.on.msg (/root/node_modules/noobaa-core/src/rpc/rpc.js:583:40)
    at RpcFcallConnection.emit (events.js:198:13)
    at RpcFcallConnection.EventEmitter.emit (domain.js:448:20)
    at Immediate.setImmediate [as _onImmediate] (/root/node_modules/noobaa-core/src/rpc/rpc_fcall.js:15:53)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
Nov-17 2:41:47.823 [Endpoint/7386]    [L0] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/nuxeo-platform-lang-ext-incomplete/10.10-HF09/nuxeo-platform-lang-ext-incomplete-10.10-HF09.jar search X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime headers { 'User-Agent': 'rclone/v1.50.1', 'Content-Length': '3603', 'Content-MD5': 'BvL8rPnKb9tmzvUZDhUuzQ==', 'Content-Type': 'application/x-java-archive', 'X-Amz-Acl': 'private', 'X-Amz-Meta-Mtime': '1562330168', 'Accept-Encoding': 'gzip', Host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https', Forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'X-Forwarded-For': '10.113.245.89' } region us-east-1 canonical_str 
PUT
/cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/nuxeo-platform-lang-ext-incomplete/10.10-HF09/nuxeo-platform-lang-ext-incomplete-10.10-HF09.jar
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime
content-md5:BvL8rPnKb9tmzvUZDhUuzQ==
content-type:application/x-java-archive
host:internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com
x-amz-acl:private
x-amz-meta-mtime:1562330168

content-md5;content-type;host;x-amz-acl;x-amz-meta-mtime
UNSIGNED-PAYLOAD
 string_to_sign 
AWS4-HMAC-SHA256
20191117T024147Z
20191117/us-east-1/s3/aws4_request
471a45b7a50f5c9cc5ab96215a5b6422d1018220bab01eaa279a006b417954dc

Nov-17 2:41:47.824 [Endpoint/7386]    [L0] core.endpoint.s3.s3_rest:: S3 REQUEST PUT /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/nuxeo-platform-lang-ext-incomplete/10.10-HF09/nuxeo-platform-lang-ext-incomplete-10.10-HF09.jar?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=daaca9355da9a19901363d29fd4a40328636fdbc8f87f7fbfd846911311c8510 op put_object request_id k32e999r-7ixuvf-iks { 'user-agent': 'rclone/v1.50.1', 'content-length': '3603', 'content-md5': 'BvL8rPnKb9tmzvUZDhUuzQ==', 'content-type': 'application/x-java-archive', 'x-amz-acl': 'private', 'x-amz-meta-mtime': '1562330168', 'accept-encoding': 'gzip', host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https', forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'x-forwarded-for': '10.113.245.89' }
Nov-17 2:41:47.824 [Endpoint/7386]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb org/nuxeo/nuxeo-platform-lang-ext-incomplete/10.10-HF09/nuxeo-platform-lang-ext-incomplete-10.10-HF09.jar  
Nov-17 2:41:47.824 [Endpoint/7386]    [L0] core.sdk.object_io:: upload_object: start upload { bucket: 'cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb', key: 'org/nuxeo/nuxeo-platform-lang-ext-incomplete/10.10-HF09/nuxeo-platform-lang-ext-incomplete-10.10-HF09.jar', content_type: 'application/x-java-archive', size: 3603, md5_b64: 'BvL8rPnKb9tmzvUZDhUuzQ==', sha256_b64: undefined, xattr: { mtime: '1562330168' }, tagging: undefined, encryption: undefined }
Nov-17 2:41:47.825 [Endpoint/7386]    [L0] core.util.signature_utils:: _string_to_sign_v4 method PUT pathname /cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/ecm/core/nuxeo-core-convert-api/10.10/_remote.repositories search X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime headers { 'User-Agent': 'rclone/v1.50.1', 'Content-Length': '253', 'Content-MD5': 'e0qUsVGxBAD+29ZaEbO+tQ==', 'Content-Type': 'application/octet-stream', 'X-Amz-Acl': 'private', 'X-Amz-Meta-Mtime': '1561669286.086617293', 'Accept-Encoding': 'gzip', Host: 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Host': 'internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https', Forwarded: 'for=10.113.245.89;host=internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com;proto=https;proto-version=', 'X-Forwarded-For': '10.113.245.89' } region us-east-1 canonical_str 
PUT
/cks-dev1-694b62ed-9bb5-44a7-9168-7e57613b7ffb/org/nuxeo/ecm/core/nuxeo-core-convert-api/10.10/_remote.repositories
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=yAMzs0pm5zn2bFXuMOe3%2F20191117%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191117T024147Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime
content-md5:e0qUsVGxBAD+29ZaEbO+tQ==
content-type:application/octet-stream
host:internal-s3-noobaa-1937458161.us-east-1.elb.amazonaws.com
x-amz-acl:private
x-amz-meta-mtime:1561669286.086617293

content-md5;content-type;host;x-amz-acl;x-amz-meta-mtime
UNSIGNED-PAYLOAD
 string_to_sign 
AWS4-HMAC-SHA256
20191117T024147Z
20191117/us-east-1/s3/aws4_request
d65937fb93c7154cf1882364d6bd533b12cdd32bb09c675aed94c2d2dba9c246
guymguym commented 4 years ago

Perhaps related to the case in #147 - because if you scale the noobaa-core sts replicas the result is undefined. Is it the same case?

ron1 commented 4 years ago

@guymguym No, I encountered this with the out-of-the-box single instance of noobaa-core.

guymguym commented 4 years ago

Hey @ron1

OK, so the logs you attached show that we got disconnected from the DB. We've seen this symptom occurring mostly when low on cpu/memory resources, so lets start by checking that first.

Can you get the cpu/memory resources allocated to the noobaa-core-0 pod - both to the core and db containers? you can get those with:

oc get pod noobaa-core-0 -o json | jq '.spec.containers[] | .name,.resources'

Then, in order to change the allocated resources you can update the noobaa spec and set it as shown below and the operator will reconcile to the statefulset and restart the pod. The numbers below are currently used in our performance testing, and you can use lower numbers depending on your use case.

apiVersion: noobaa.io/v1alpha1
kind: NooBaa
metadata:
  name: noobaa
  namespace: openshift-storage
spec:
  coreResources:
    limits:
      cpu: "16"
      memory: 32Gi
    requests:
      cpu: "8"
      memory: 16Gi
  dbResources:
    limits:
      cpu: "8"
      memory: 32Gi
    requests:
      cpu: "4"
      memory: 16Gi