search

noobaa / noobaa-operator

Operator for NooBaa - object data service for hybrid and multi cloud environments :cloud: :wrench:
https://www.noobaa.io
Apache License 2.0
103 stars 101 forks source link

Intermittent [RpcError: self signed certificate] #1209

Open pgfan1024 opened 1 year ago

pgfan1024 commented 1 year ago

Environment info

Actual behavior

  1. I am using noobba storage for pgbackrest backups. The first backup and expiration fails at expiration of backups as below job/pod logs: time="2023-08-24T13:16:13Z" level=info msg="stderr=[ERROR: [101]: repo2: [500] Internal Server Error\nERROR: [104]: expire command encountered 1 error(s), check the log file for details\n]"

However with the immediate retry the backup_expiration job works fine.

In the noobaa endpoint logs, there are frequent:

Aug-24 13:16:13.941 [Endpoint/13] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?>InternalErrorWe encountered an internal error. Please try again./pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb/noobaa-23082023/repo2/backup/db/backup.infollp6tahq-3p0ihd-u89 PUT /pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb/noobaa-23082023/repo2/backup/db/backup.info {"user-agent":"pgBackRest/2.41","authorization":"AWS4-HMAC-SHA256 Credential=b1Cxm7BFeY6BN8jPl9Ur/20230824/ap-southeast2/s3/aws4_request,SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date,Signature=b7158a9f5e204eb44492792fcffa27a83be51a3d62a543545c68a07695a9df69","content-length":"4629","content-md5":"8aEfPagLTjKtDCXQdVnmug==","x-amz-content-sha256":"dab0b2d568d72acd899693a48d08287ca2669731c7d7dd43a057e605a5b07723","x-amz-date":"20230824T131513Z","host":"s3-openshift-storage.apps.ocp4-redacted.ocpdemos.com","x-forwarded-host":"s3-openshift-storage.apps.ocp4-redacted.ocpdemos.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.128.2.58;host=s3-openshift-storage.apps.ocp4-redacted.ocpdemos.com;proto=https","x-forwarded-for":"10.128.2.58"} [RpcError: self signed certificate] { rpc_code: 'INTERNAL', rpc_data: { retryable: true } }

Expected behavior

Backup and expiration should work in the first attempt.

Steps to reproduce

  1. Install Crunchy Postgres Operator: https://github.com/CrunchyData/postgres-operator
  2. noobaa backup (repo 2) specific values: backups: pgbackrest: image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.41-4 configuration:
    • secret: name: pgo-noobaa-creds global: process-max: "2" start-fast: "y" log-level-console: "info" repo2-path: /noobaa-23082023/repo2/ repo2-s3-uri-style: path repo2-storage-verify-tls: "n" repo2-retention-full: "4" repo2-retention-diff: "3" repos:

More information - Screenshots / Logs / Other output

noobaa-endpoint log at the time of error. (Please also see error in noobaa core at the end)

Aug-24 13:15:52.990 [Endpoint/13] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?>InternalErrorWe encountered an internal error. Please try again./pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb/noobaa-23082023/repo2/backup/db/backup.infollp6subx-4lx9h0-1d1o PUT /pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb/noobaa-23082023/repo2/backup/db/backup.info {"user-agent":"pgBackRest/2.41","authorization":"AWS4-HMAC-SHA256 Credential=b1Cxm7BFeY6BN8jPl9Ur/20230824/ap-southeast2/s3/aws4_request,SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date,Signature=b7158a9f5e204eb44492792fcffa27a83be51a3d62a543545c68a07695a9df69","content-length":"4629","content-md5":"8aEfPagLTjKtDCXQdVnmug==","x-amz-content-sha256":"dab0b2d568d72acd899693a48d08287ca2669731c7d7dd43a057e605a5b07723","x-amz-date":"20230824T131513Z","host":"s3-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com","x-forwarded-host":"s3-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.128.2.58;host=s3-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com;proto=https","x-forwarded-for":"10.128.2.58"} [RpcError: self signed certificate] { rpc_code: 'INTERNAL', rpc_data: { retryable: true } } Aug-24 13:16:13.802 [Endpoint/13] [L0] core.sdk.object_sdk:: _load_bucket_namespace { _id: '64e5f4ab45f814002dacef00', name: SENSITIVE-613260574318c85f, active_triggers: [ length: 0 ], system_owner: SENSITIVE-9cf0fd89409efef8, bucket_owner: SENSITIVE-d4bc82999e444a8c, bucket_info: { name: SENSITIVE-613260574318c85f, owner_account: { email: SENSITIVE-d4bc82999e444a8c, id: '644b5b4c964845002a8c5e2d' }, tiering: { name: SENSITIVE-1406974c42d8da14, tiers: [ { order: 0, tier: SENSITIVE-5be85d811b4b2f08, spillover: false, disabled: false, mode: 'NOT_ENOUGH_HEALTHY_RESOURCES' }, length: 1 ], storage: { total: { n: 6422397, peta: 1 }, used: 6422397, used_other: 0, free: 0, reserved: 0, unavailable_free: { n: 0, peta: 1 }, unavailable_used: 6422397 }, data: { free: 0 }, mode: 'NOT_ENOUGH_HEALTHY_RESOURCES' }, tag: '', num_objects: { value: 3869, last_update: 1692882863172 }, writable: false, storage: { values: { used: 5730159, used_other: 0, total: 5730159, free: 0 }, last_update: 1692882865258 }, data: { size: 19537072, size_reduced: 5730159, free: 0, available_size_for_upload: 0, available_quantity_for_upload: { n: 1125899906842623, peta: 7 }, last_update: 1692882865258 }, usage_by_pool: { pools: [ { pool_name: 'noobaa-default-backing-store', storage: { blocks_size: 5730159 } }, length: 1 ], last_update: 1692882863172 }, stats_by_type: [ length: 0 ], mode: 'NOT_ENOUGH_HEALTHY_RESOURCES', host_tolerance: 0, node_tolerance: 0, bucket_type: 'REGULAR', versioning: 'DISABLED', bucket_claim: { namespace: 'openshift-storage', bucket_class: 'noobaa-default-bucket-class' }, policy_modes: { resiliency_status: 'OPTIMAL', quota_status: 'QUOTA_NOT_SET' }, triggers: [ length: 0 ] } } Aug-24 13:16:13.803 [Endpoint/13] [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb noobaa-23082023/repo2/backup/db/backup.info Aug-24 13:16:13.803 [Endpoint/13] [L0] core.sdk.object_io:: upload_object: start upload { bucket: 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb', key: 'noobaa-23082023/repo2/backup/db/backup.info', content_type: 'application/octet-stream', content_encoding: undefined, size: 4629, md5_b64: '8aEfPagLTjKtDCXQdVnmug==', sha256_b64: '2rCy1WjXKs2JlpOkjQgofKJmlzHH191DoFfmBaWwdyM=', xattr: {}, tagging: undefined, encryption: undefined, lock_settings: undefined } Aug-24 13:16:13.817 [Endpoint/13] [L0] core.sdk.object_io:: UPLOAD: { obj_id: '64e7581d82c2a2000d1fa86d', bucket: 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb', key: 'noobaa-23082023/repo2/backup/db/backup.info' } streaming to pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb noobaa-23082023/repo2/backup/db/backup.info Aug-24 13:16:13.819 [Endpoint/13] [L0] core.sdk.object_io:: UPLOAD: part 0 ChunkAPI { chunk_info: { data: undefined, size: 4629, pos: 0, chunk_coder_config: { replicas: 1, data_frags: 1, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_frags: 0, compress_type: 'snappy', frag_digest_type: 'sha1' }, frag_size: 1203, compress_size: 1203, digest_b64: 'ifDQYH8EDuKIFvZICiUQIbqlDhbpnzQscdaGER8jGAoCSE+LA0IHdKDK8HCiNMsd', cipher_key_b64: 'mz5m/62fdJN/DohJlIEk0xxmrwaED0ZfPIzYTPo3Ifg=', frags: [ { data_index: 0, data: <Buffer 36 f1 bc 7f bc 32 24 02 a8 98 6d 3f 0c 36 92 1e fc 65 ec b6 f6 a9 a0 a5 f1 ea 38 8d 4d c3 95 50 dd 99 ef 4a fc 17 e1 85 0f f4 e4 98 5c 09 da 1a 43 e8 ... 1153 more bytes>, digest_b64: 'vvVF6przkwKONssXdagRsj/KDTE=', blocks: [] } ], tier_id: '64e5f4a345f814002dace812', bucket_id: '64e5f4ab45f814002dacef00', parts: [ { obj_id: '64e7581d82c2a2000d1fa86d', chunk_id: undefined, multipart_id: undefined, start: 0, end: 4629, seq: 0, uncommitted: true } ], master_key_id: '64e5f4ab45f814002dacef01' }, system_store: undefined, had_errors: false } Aug-24 13:16:13.819 [Endpoint/13] [L0] core.server.object_services.map_server:: GetMapping.find_dups: found keys 1 Aug-24 13:16:13.822 [Endpoint/13] [L0] core.server.object_services.map_server:: GetMapping.find_dups: dup_chunks [] Aug-24 13:16:13.827 [Endpoint/13] [ERROR] core.server.node_services.node_allocator:: allocate_node: no nodes for allocation in pool set [ <ref 1> { _id: 644b5b71964845002a8c5e33, name: 'noobaa-default-backing-store', system: <ref 2> { _id: 644b5b4b964845002a8c5e1e, name: 'noobaa', owner: { _id: 644b5b4b964845002a8c5e1d, name: SENSITIVE-b8e720282050fed7, email: SENSITIVE-9cf0fd89409efef8, password: SENSITIVE-5cdba8b7e533005e, has_login: true, access_keys: [ { access_key: SENSITIVE-c602684641bf489b, secret_key: SENSITIVE-af8018a9cc93ec65 } ], master_key_id: { _id: 644b5b4c964845002a8c5e2b, disabled: false, cipher_iv: <Buffer 8e 68 6a 83 8c 5d a9 ca 73 19 ee 18 16 32 20 82>, cipher_key: <Buffer 5d 35 ab b8 0d 60 37 ea 34 95 dc 3d de 45 2c 68 7e 04 d2 3f e8 b4 ae 13 1c 84 4d 11 0c f5 8b 2a>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1d account', master_key_id: { _id: 644b5b4b964845002a8c5e1f, disabled: false, cipher_iv: <Buffer c0 34 a4 ed 73 93 d1 d2 88 63 f4 21 33 6c 20 63>, cipher_key: <Buffer 83 89 80 3d 8b ef 70 a2 9a 47 df 69 1a 27 81 82 1b d4 06 f1 03 da 0b 9c b9 22 25 de 5e a3 0b 5f>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1e system', master_key_id: 00000000aaaabbbbccccdddd } }, allowed_buckets: { full_permission: true }, default_resource: [Circular 1], allow_bucket_creation: true, roles_by_system: { '644b5b4b964845002a8c5e1e': [ 'admin' ] } }, state: { mode: 'READY', last_update: 1682660174075 }, resources: { agent_installer: 'noobaa-setup.exe', s3rest_installer: 'noobaa-s3rest.exe', linux_agent_installer: 'noobaa-setup' }, n2n_config: { tcp_tls: true, udp_dtls: true, udp_port: true, tcp_active: true, tcp_permanent_passive: { max: 60600, min: 60101 } }, debug_level: 0, freemium_cap: { cap_terabytes: 20, phone_home_notified: false, phone_home_upgraded: false }, master_key_id: { _id: 644b5b4b964845002a8c5e1f, disabled: false, cipher_iv: <Buffer c0 34 a4 ed 73 93 d1 d2 88 63 f4 21 33 6c 20 63>, cipher_key: <Buffer 83 89 80 3d 8b ef 70 a2 9a 47 df 69 1a 27 81 82 1b d4 06 f1 03 da 0b 9c b9 22 25 de 5e a3 0b 5f>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1e system', master_key_id: 00000000aaaabbbbccccdddd }, mongo_upgrade: { blocks_to_buckets: true }, system_address: [ { api: 'mgmt', kind: 'EXTERNAL', port: 443, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: '34.116.101.179' }, { api: 'mgmt', kind: 'EXTERNAL', port: 80, secure: false, weight: 0, service: 'noobaa-mgmt', hostname: '34.116.101.179' }, { api: 'bg', kind: 'EXTERNAL', port: 8445, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: '34.116.101.179' }, { api: 'hosted_agents', kind: 'EXTERNAL', port: 8446, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: '34.116.101.179' }, { api: 'mgmt', kind: 'EXTERNAL', port: 443, secure: true, weight: 100, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com' }, { api: 's3', kind: 'EXTERNAL', port: 443, secure: true, weight: 0, service: 's3', hostname: '35.189.42.129' }, { api: 'metrics', kind: 'EXTERNAL', port: 7004, secure: false, weight: 0, service: 's3', hostname: '35.189.42.129' }, { api: 's3', kind: 'EXTERNAL', port: 80, secure: false, weight: 0, service: 's3', hostname: '35.189.42.129' }, { api: 'md', kind: 'EXTERNAL', port: 8444, secure: true, weight: 0, service: 's3', hostname: '35.189.42.129' }, { api: 's3', kind: 'EXTERNAL', port: 443, secure: true, weight: 100, service: 's3', hostname: 's3-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com' }, { api: 'sts', kind: 'EXTERNAL', port: 443, secure: true, weight: 0, service: 'sts', hostname: '34.116.113.62' }, { api: 'sts', kind: 'EXTERNAL', port: 443, secure: true, weight: 100, service: 'sts', hostname: 'sts-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com' }, { api: 'postgres', kind: 'INTERNAL', port: 5432, secure: false, weight: 0, service: 'noobaa-db-pg', hostname: 'noobaa-db-pg.openshift-storage.svc.cluster.local' }, { api: 'mgmt', kind: 'INTERNAL', port: 443, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt.openshift-storage.svc.cluster.local' }, { api: 'mgmt', kind: 'INTERNAL', port: 80, secure: false, weight: 0, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt.openshift-storage.svc.cluster.local' }, { api: 'bg', kind: 'INTERNAL', port: 8445, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt.openshift-storage.svc.cluster.local' }, { api: 'hosted_agents', kind: 'INTERNAL', port: 8446, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt.openshift-storage.svc.cluster.local' }, { api: 's3', kind: 'INTERNAL', port: 443, secure: true, weight: 0, service: 's3', hostname: 's3.openshift-storage.svc.cluster.local' }, { api: 'metrics', kind: 'INTERNAL', port: 7004, secure: false, weight: 0, service: 's3', hostname: 's3.openshift-storage.svc.cluster.local' }, { api: 's3', kind: 'INTERNAL', port: 80, secure: false, weight: 0, service: 's3', hostname: 's3.openshift-storage.svc.cluster.local' }, { api: 'md', kind: 'INTERNAL', port: 8444, secure: true, weight: 0, service: 's3', hostname: 's3.openshift-storage.svc.cluster.local' }, { api: 'sts', kind: 'INTERNAL', port: 443, secure: true, weight: 0, service: 'sts', hostname: 'sts.openshift-storage.svc.cluster.local' } ], current_version: '5.11.0-1a18c33', upgrade_history: { successful_upgrades: [] }, last_stats_report: 1692882887253, global_last_update: 1692882865258, roles_by_account: { '64e5f4ab45f814002dacef03': [ 'admin' ], '644b5b4b964845002a8c5e1d': [ 'admin' ], '644b5b4c964845002a8c5e2d': [ 'operator' ] }, buckets_by_name: { 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb': { _id: 64e5f4ab45f814002dacef00, tag: '', name: SENSITIVE-613260574318c85f, system: [Circular 2], tiering: { _id: 64e5f4a945f814002daceeff, name: SENSITIVE-1406974c42d8da14, tiers: [ { tier: { _id: 64e5f4a345f814002dace812, name: SENSITIVE-5be85d811b4b2f08, system: [Circular 2], mirrors: [ { _id: 64e5f4a345f814002dace811, spread_pools: [Array] } ], chunk_config: { _id: 644b5b4b964845002a8c5e21, system: [Circular 2], chunk_coder_config: { replicas: 1, data_frags: 1, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_frags: 0, compress_type: 'snappy', frag_digest_type: 'sha1' } }, data_placement: 'SPREAD' }, order: 0, disabled: false, spillover: false } ], system: [Circular 2], chunk_split_config: { avg_chunk: 4194304, delta_chunk: 1048576 } }, versioning: 'DISABLED', bucket_claim: { namespace: 'openshift-storage', bucket_class: 'noobaa-default-bucket-class' }, master_key_id: { _id: 64e5f4ab45f814002dacef01, disabled: false, cipher_iv: <Buffer 3c 5c 65 4d d9 54 47 8a 6d d8 3e 2f ab 9a 38 95>, cipher_key: <Buffer 73 3d c8 6f bb fe 28 5f 3f 4f 32 07 d3 19 d3 8f d3 67 b0 4b 9e e6 35 ac 6b 70 b3 f2 b3 13 a2 50>, cipher_type: 'aes-256-gcm', description: 'master key of 64e5f4ab45f814002dacef00 bucket', master_key_id: { _id: 644b5b4b964845002a8c5e1f, disabled: false, cipher_iv: <Buffer c0 34 a4 ed 73 93 d1 d2 88 63 f4 21 33 6c 20 63>, cipher_key: <Buffer 83 89 80 3d 8b ef 70 a2 9a 47 df 69 1a 27 81 82 1b d4 06 f1 03 da 0b 9c b9 22 25 de 5e a3 0b 5f>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1e system', master_key_id: 00000000aaaabbbbccccdddd } }, owner_account: { _id: 644b5b4c964845002a8c5e2d, name: SENSITIVE-b8e720282050fed7, email: SENSITIVE-d4bc82999e444a8c, has_login: false, access_keys: [ { access_key: SENSITIVE-c3e825fc7571f7eb, secret_key: SENSITIVE-65fd55b7a588a802 } ], master_key_id: { _id: 644b5b4c964845002a8c5e2f, disabled: false, cipher_iv: <Buffer 50 6c d3 8d 03 f2 12 02 e7 77 c4 20 51 3e ad 6c>, cipher_key: <Buffer d5 f4 32 f0 ce 76 44 ea 94 b5 0d d7 6b 6c 6c a4 24 74 52 c5 65 0c 12 2c 90 de 7c ce 32 6e 32 8b>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4c964845002a8c5e2d account', master_key_id: { _id: 644b5b4b964845002a8c5e1f, disabled: false, cipher_iv: <Buffer c0 34 a4 ed 73 93 d1 d2 88 63 f4 21 33 6c 20 63>, cipher_key: <Buffer 83 89 80 3d 8b ef 70 a2 9a 47 df 69 1a 27 81 82 1b d4 06 f1 03 da 0b 9c b9 22 25 de 5e a3 0b 5f>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1e system', master_key_id: 00000000aaaabbbbccccdddd } }, allowed_buckets: { full_permission: true }, default_resource: { _id: 644b5b4b964845002a8c5e20, name: 'system-internal-storage-pool-644b5b4b964845002a8c5e1e', system: [Circular 2], resource_type: 'INTERNAL', storage_stats: { blocks_size: 0, last_update: 1689572300566 }, pool_node_type: 'BLOCK_STORE_MONGO', mongo_pool_info: {} }, allow_bucket_creation: true, sync_credentials_cache: [ { name: 'noobaa-default-backing-store', endpoint: 'https://www.googleapis.com', access_key: SENSITIVE-b88954a8f561039e, secret_key: SENSITIVE-4dbb91be0bb1314d, endpoint_type: 'GOOGLE' } ], roles_by_system: { '644b5b4b964845002a8c5e1e': [ 'operator' ] } }, storage_stats: { pools: { '644b5b71964845002a8c5e33': { blocks_size: 5730159 } }, blocks_size: 5730159, last_update: 1692882863172, objects_hist: [ { count: 0, label: '0 - 1 B', aggregated_sum: 0 }, { count: 0, label: '1 B - 2 B', aggregated_sum: 0 }, { count: 0, label: '2 B - 4 B', aggregated_sum: 0 }, { count: 0, label: '4 B - 8 B', aggregated_sum: 0 }, { count: 0, label: '8 B - 16 B', aggregated_sum: 0 }, { count: 636, label: '16 B - 32 B', aggregated_sum: 12813 }, { count: 50, label: '32 B - 64 B', aggregated_sum: 2920 }, { count: 1272, label: '64 B - 128 B', aggregated_sum: 101737 }, { count: 806, label: '128 B - 256 B', aggregated_sum: 147973 }, { count: 133, label: '256 B - 512 B', aggregated_sum: 50269 }, { count: 117, label: '512 B - 1.0 KB', aggregated_sum: 87378 }, { count: 218, label: '1.0 KB - 2.0 KB', aggregated_sum: 325233 }, { count: 188, label: '2.0 KB - 4.0 KB', aggregated_sum: 604991 }, { count: 122, label: '4.0 KB - 8.0 KB', aggregated_sum: 694253 }, { count: 111, label: '8.0 KB - 16 KB', aggregated_sum: 1493077 }, { count: 69, label: '16 KB - 32 KB', aggregated_sum: 1455367 }, { count: 95, label: '32 KB - 64 KB', aggregated_sum: 4146538 }, { count: 26, label: '64 KB - 128 KB', aggregated_sum: 2331275 }, { count: 13, label: '128 KB - 256 KB', aggregated_sum: 2789824 }, { count: 12, label: '256 KB - 512 KB', aggregated_sum: 3304104 }, { count: 0, label: '512 KB - 1.0 MB', aggregated_sum: 0 }, { count: 1, label: '1.0 MB - 2.0 MB', aggregated_sum: 1989320 } ], objects_size: 19537072, objects_count: 3869, chunks_capacity: 5730159, stats_by_content_type: [ { size: 1335185, count: 23, content_type: 'application/octet-stream' }, { size: 16881275, count: 3840, content_type: 'application/gzip' }, { size: 1320612, count: 6, content_type: 'text/cache-manifest' } ] }, lambda_triggers: [] } }, tiering_policies_by_name: { 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb.64e5f4a3': { _id: 64e5f4a945f814002daceeff, name: SENSITIVE-1406974c42d8da14, tiers: [ { tier: { _id: 64e5f4a345f814002dace812, name: SENSITIVE-5be85d811b4b2f08, system: [Circular 2], mirrors: [ { _id: 64e5f4a345f814002dace811, spread_pools: [ [Circular 1] ] } ], chunk_config: { _id: 644b5b4b964845002a8c5e21, system: [Circular 2], chunk_coder_config: { replicas: 1, data_frags: 1, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_frags: 0, compress_type: 'snappy', frag_digest_type: 'sha1' } }, data_placement: 'SPREAD' }, order: 0, disabled: false, spillover: false } ], system: [Circular 2], chunk_split_config: { avg_chunk: 4194304, delta_chunk: 1048576 } } }, tiers_by_name: { 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb.64e5f4a3.0': { _id: 64e5f4a345f814002dace812, name: SENSITIVE-5be85d811b4b2f08, system: [Circular 2], mirrors: [ { _id: 64e5f4a345f814002dace811, spread_pools: [ [Circular 1] ] } ], chunk_config: { _id: 644b5b4b964845002a8c5e21, system: [Circular 2], chunk_coder_config: { replicas: 1, data_frags: 1, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_frags: 0, compress_type: 'snappy', frag_digest_type: 'sha1' } }, data_placement: 'SPREAD' } }, pools_by_name: { 'noobaa-default-backing-store': [Circular 1], 'system-internal-storage-pool-644b5b4b964845002a8c5e1e': { _id: 644b5b4b964845002a8c5e20, name: 'system-internal-storage-pool-644b5b4b964845002a8c5e1e', system: [Circular 2], resource_type: 'INTERNAL', storage_stats: { blocks_size: 0, last_update: 1689572300566 }, pool_node_type: 'BLOCK_STORE_MONGO', mongo_pool_info: {} } }, chunk_configs_by_id: { '644b5b4b964845002a8c5e21': { _id: 644b5b4b964845002a8c5e21, system: [Circular 2], chunk_coder_config: { replicas: 1, data_frags: 1, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_frags: 0, compress_type: 'snappy', frag_digest_type: 'sha1' } }, '644b5b4b964845002a8c5e22': { _id: 644b5b4b964845002a8c5e22, system: [Circular 2], chunk_coder_config: { replicas: 1, data_frags: 4, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_type: 'cm256', parity_frags: 2, compress_type: 'snappy', frag_digest_type: 'sha1' } } } }, resource_type: 'CLOUD', storage_stats: { blocks_size: 5730159, last_update: 1692882863172 }, pool_node_type: 'BLOCK_STORE_GOOGLE', cloud_pool_info: { endpoint: 'https://www.googleapis.com', agent_info: { cloud_path: 'noobaa_blocks/644b5b71964845002a8c5e33', node_token: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzeXN0ZW1faWQiOiI2NDRiNWI0Yjk2NDg0NTAwMmE4YzVlMWUiLCJyb2xlIjoiYWdlbnQiLCJleHRyYSI6eyJub2RlX2lkIjoiNjQ0YjViNzI5NjQ4NDUwMDJhOGM1ZTM0In0sImF1dGhvcml6ZWRfYnkiOiJub29iYWEiLCJpYXQiOjE2ODI2NjAyMTB9.JgsnLoEe71PywSC9HCbZWtm5cStEvjeAWV3z5PU3KUY', create_node_token: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2NvdW50X2lkIjoiNjQ0YjViNGI5NjQ4NDUwMDJhOGM1ZTFkIiwic3lzdGVtX2lkIjoiNjQ0YjViNGI5NjQ4NDUwMDJhOGM1ZTFlIiwicm9sZSI6ImNyZWF0ZV9ub2RlIiwiYXV0aG9yaXplZF9ieSI6Im5vb2JhYSIsImlhdCI6MTY4MjY2MDIwOX0.KHgYoqtclNdZGvJ3770UeZHeX7JdGtb65auQiBGz-qs' }, access_keys: { access_key: SENSITIVE-b88954a8f561039e, account_id: <ref 3> { _id: 644b5b4c964845002a8c5e2d, name: SENSITIVE-b8e720282050fed7, email: SENSITIVE-d4bc82999e444a8c, has_login: false, access_keys: [ { access_key: SENSITIVE-c3e825fc7571f7eb, secret_key: SENSITIVE-65fd55b7a588a802 } ], master_key_id: { _id: 644b5b4c964845002a8c5e2f, disabled: false, cipher_iv: <Buffer 50 6c d3 8d 03 f2 12 02 e7 77 c4 20 51 3e ad 6c>, cipher_key: <Buffer d5 f4 32 f0 ce 76 44 ea 94 b5 0d d7 6b 6c 6c a4 24 74 52 c5 65 0c 12 2c 90 de 7c ce 32 6e 32 8b>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4c964845002a8c5e2d account', master_key_id: { _id: 644b5b4b964845002a8c5e1f, disabled: false, cipher_iv: <Buffer c0 34 a4 ed 73 93 d1 d2 88 63 f4 21 33 6c 20 63>, cipher_key: <Buffer 83 89 80 3d 8b ef 70 a2 9a 47 df 69 1a 27 81 82 1b d4 06 f1 03 da 0b 9c b9 22 25 de 5e a3 0b 5f>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1e system', master_key_id: 00000000aaaabbbbccccdddd } }, allowed_buckets: { full_permission: true }, default_resource: <ref 4> { _id: 644b5b4b964845002a8c5e20, name: 'system-internal-storage-pool-644b5b4b964845002a8c5e1e', system: <ref 2> { _id: 644b5b4b964845002a8c5e1e, name: 'noobaa', owner: { _id: 644b5b4b964845002a8c5e1d, name: SENSITIVE-b8e720282050fed7, email: SENSITIVE-9cf0fd89409efef8, password: SENSITIVE-5cdba8b7e533005e, has_login: true, access_keys: [ { access_key: SENSITIVE-c602684641bf489b, secret_key: SENSITIVE-af8018a9cc93ec65 } ], master_key_id: { _id: 644b5b4c964845002a8c5e2b, disabled: false, cipher_iv: <Buffer 8e 68 6a 83 8c 5d a9 ca 73 19 ee 18 16 32 20 82>, cipher_key: <Buffer 5d 35 ab b8 0d 60 37 ea 34 95 dc 3d de 45 2c 68 7e 04 d2 3f e8 b4 ae 13 1c 84 4d 11 0c f5 8b 2a>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1d account', master_key_id: { _id: 644b5b4b964845002a8c5e1f, disabled: false, cipher_iv: <Buffer c0 34 a4 ed 73 93 d1 d2 88 63 f4 21 33 6c 20 63>, cipher_key: <Buffer 83 89 80 3d 8b ef 70 a2 9a 47 df 69 1a 27 81 82 1b d4 06 f1 03 da 0b 9c b9 22 25 de 5e a3 0b 5f>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1e system', master_key_id: 00000000aaaabbbbccccdddd } }, allowed_buckets: { full_permission: true }, default_resource: [Circular 1], allow_bucket_creation: true, roles_by_system: { '644b5b4b964845002a8c5e1e': [ 'admin' ] } }, state: { mode: 'READY', last_update: 1682660174075 }, resources: { agent_installer: 'noobaa-setup.exe', s3rest_installer: 'noobaa-s3rest.exe', linux_agent_installer: 'noobaa-setup' }, n2n_config: { tcp_tls: true, udp_dtls: true, udp_port: true, tcp_active: true, tcp_permanent_passive: { max: 60600, min: 60101 } }, debug_level: 0, freemium_cap: { cap_terabytes: 20, phone_home_notified: false, phone_home_upgraded: false }, master_key_id: { _id: 644b5b4b964845002a8c5e1f, disabled: false, cipher_iv: <Buffer c0 34 a4 ed 73 93 d1 d2 88 63 f4 21 33 6c 20 63>, cipher_key: <Buffer 83 89 80 3d 8b ef 70 a2 9a 47 df 69 1a 27 81 82 1b d4 06 f1 03 da 0b 9c b9 22 25 de 5e a3 0b 5f>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1e system', master_key_id: 00000000aaaabbbbccccdddd }, mongo_upgrade: { blocks_to_buckets: true }, system_address: [ { api: 'mgmt', kind: 'EXTERNAL', port: 443, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: '34.116.101.179' }, { api: 'mgmt', kind: 'EXTERNAL', port: 80, secure: false, weight: 0, service: 'noobaa-mgmt', hostname: '34.116.101.179' }, { api: 'bg', kind: 'EXTERNAL', port: 8445, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: '34.116.101.179' }, { api: 'hosted_agents', kind: 'EXTERNAL', port: 8446, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: '34.116.101.179' }, { api: 'mgmt', kind: 'EXTERNAL', port: 443, secure: true, weight: 100, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com' }, { api: 's3', kind: 'EXTERNAL', port: 443, secure: true, weight: 0, service: 's3', hostname: '35.189.42.129' }, { api: 'metrics', kind: 'EXTERNAL', port: 7004, secure: false, weight: 0, service: 's3', hostname: '35.189.42.129' }, { api: 's3', kind: 'EXTERNAL', port: 80, secure: false, weight: 0, service: 's3', hostname: '35.189.42.129' }, { api: 'md', kind: 'EXTERNAL', port: 8444, secure: true, weight: 0, service: 's3', hostname: '35.189.42.129' }, { api: 's3', kind: 'EXTERNAL', port: 443, secure: true, weight: 100, service: 's3', hostname: 's3-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com' }, { api: 'sts', kind: 'EXTERNAL', port: 443, secure: true, weight: 0, service: 'sts', hostname: '34.116.113.62' }, { api: 'sts', kind: 'EXTERNAL', port: 443, secure: true, weight: 100, service: 'sts', hostname: 'sts-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com' }, { api: 'postgres', kind: 'INTERNAL', port: 5432, secure: false, weight: 0, service: 'noobaa-db-pg', hostname: 'noobaa-db-pg.openshift-storage.svc.cluster.local' }, { api: 'mgmt', kind: 'INTERNAL', port: 443, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt.openshift-storage.svc.cluster.local' }, { api: 'mgmt', kind: 'INTERNAL', port: 80, secure: false, weight: 0, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt.openshift-storage.svc.cluster.local' }, { api: 'bg', kind: 'INTERNAL', port: 8445, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt.openshift-storage.svc.cluster.local' }, { api: 'hosted_agents', kind: 'INTERNAL', port: 8446, secure: true, weight: 0, service: 'noobaa-mgmt', hostname: 'noobaa-mgmt.openshift-storage.svc.cluster.local' }, { api: 's3', kind: 'INTERNAL', port: 443, secure: true, weight: 0, service: 's3', hostname: 's3.openshift-storage.svc.cluster.local' }, { api: 'metrics', kind: 'INTERNAL', port: 7004, secure: false, weight: 0, service: 's3', hostname: 's3.openshift-storage.svc.cluster.local' }, { api: 's3', kind: 'INTERNAL', port: 80, secure: false, weight: 0, service: 's3', hostname: 's3.openshift-storage.svc.cluster.local' }, { api: 'md', kind: 'INTERNAL', port: 8444, secure: true, weight: 0, service: 's3', hostname: 's3.openshift-storage.svc.cluster.local' }, { api: 'sts', kind: 'INTERNAL', port: 443, secure: true, weight: 0, service: 'sts', hostname: 'sts.openshift-storage.svc.cluster.local' } ], current_version: '5.11.0-1a18c33', upgrade_history: { successful_upgrades: [] }, last_stats_report: 1692882887253, global_last_update: 1692882865258, roles_by_account: { '64e5f4ab45f814002dacef03': [ 'admin' ], '644b5b4b964845002a8c5e1d': [ 'admin' ], '644b5b4c964845002a8c5e2d': [ 'operator' ] }, buckets_by_name: { 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb': { _id: 64e5f4ab45f814002dacef00, tag: '', name: SENSITIVE-613260574318c85f, system: [Circular 2], tiering: { _id: 64e5f4a945f814002daceeff, name: SENSITIVE-1406974c42d8da14, tiers: [ [Object] ], system: [Circular 2], chunk_split_config: { avg_chunk: 4194304, delta_chunk: 1048576 } }, versioning: 'DISABLED', bucket_claim: { namespace: 'openshift-storage', bucket_class: 'noobaa-default-bucket-class' }, master_key_id: { _id: 64e5f4ab45f814002dacef01, disabled: false, cipher_iv: <Buffer 3c 5c 65 4d d9 54 47 8a 6d d8 3e 2f ab 9a 38 95>, cipher_key: <Buffer 73 3d c8 6f bb fe 28 5f 3f 4f 32 07 d3 19 d3 8f d3 67 b0 4b 9e e6 35 ac 6b 70 b3 f2 b3 13 a2 50>, cipher_type: 'aes-256-gcm', description: 'master key of 64e5f4ab45f814002dacef00 bucket', master_key_id: { _id: 644b5b4b964845002a8c5e1f, disabled: false, cipher_iv: <Buffer c0 34 a4 ed 73 93 d1 d2 88 63 f4 21 33 6c 20 63>, cipher_key: <Buffer 83 89 80 3d 8b ef 70 a2 9a 47 df 69 1a 27 81 82 1b d4 06 f1 03 da 0b 9c b9 22 25 de 5e a3 0b 5f>, cipher_type: 'aes-256-gcm', description: 'master key of 644b5b4b964845002a8c5e1e system', master_key_id: 00000000aaaabbbbccccdddd } }, owner_account: [Circular 3], storage_stats: { pools: { '644b5b71964845002a8c5e33': [Object] }, blocks_size: 5730159, last_update: 1692882863172, objects_hist: [ [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object] ], objects_size: 19537072, objects_count: 3869, chunks_capacity: 5730159, stats_by_content_type: [ [Object], [Object], [Object] ] }, lambda_triggers: [] } }, tiering_policies_by_name: { 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb.64e5f4a3': { _id: 64e5f4a945f814002daceeff, name: SENSITIVE-1406974c42d8da14, tiers: [ { tier: [Object], order: 0, disabled: false, spillover: false } ], system: [Circular 2], chunk_split_config: { avg_chunk: 4194304, delta_chunk: 1048576 } } }, tiers_by_name: { 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb.64e5f4a3.0': { _id: 64e5f4a345f814002dace812, name: SENSITIVE-5be85d811b4b2f08, system: [Circular 2], mirrors: [ { _id: 64e5f4a345f814002dace811, spread_pools: [Array] } ], chunk_config: { _id: 644b5b4b964845002a8c5e21, system: [Circular 2], chunk_coder_config: { replicas: 1, data_frags: 1, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_frags: 0, compress_type: 'snappy', frag_digest_type: 'sha1' } }, data_placement: 'SPREAD' } }, pools_by_name: { 'noobaa-default-backing-store': [Circular 1], 'system-internal-storage-pool-644b5b4b964845002a8c5e1e': [Circular 4] }, chunk_configs_by_id: { '644b5b4b964845002a8c5e21': { _id: 644b5b4b964845002a8c5e21, system: [Circular 2], chunk_coder_config: { replicas: 1, data_frags: 1, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_frags: 0, compress_type: 'snappy', frag_digest_type: 'sha1' } }, '644b5b4b964845002a8c5e22': { _id: 644b5b4b964845002a8c5e22, system: [Circular *2], chunk_coder_config: { replicas: 1, data_frags: 4, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_type: 'cm256', parity_frags: 2, compress_type: 'snappy', frag_digest_type: 'sha1' } } } }, resource_type: 'INTERNAL', storage_stats: { blocks_size: 0, last_update: 1689572300566 }, pool_node_type: 'BLOCK_STORE_MONGO', mongo_pool_info: {} }, allow_bucket_creation: true, sync_credentials_cache: [ { name: 'noobaa-default-backing-store', endpoint: 'https://www.googleapis.com', access_key: SENSITIVE-b88954a8f561039e, secret_key: SENSITIVE-4dbb91be0bb1314d, endpoint_type: 'GOOGLE' } ], roles_by_system: { '644b5b4b964845002a8c5e1e': [ 'operator' ] } }, secret_key: SENSITIVE-4dbb91be0bb1314d }, backingstore: { name: 'noobaa-default-backing-store', namespace: 'openshift-storage' }, endpoint_type: 'GOOGLE', target_bucket: 'noobaabucket767bb' } } ] [] [] Aug-24 13:16:13.827 [Endpoint/13] [WARN] core.server.object_services.map_server:: GetMapping allocate_blocks: no nodes for allocation avoid_nodes allocation_pools [[object Object]] Aug-24 13:16:13.933 [Endpoint/13] [L0] core.rpc.rpc_base_conn:: RPC CONNECTION CLOSED. got event from connection: wss://noobaa-mgmt.openshift-storage.svc:8445(wliakpkgzz) Error: self signed certificate at TLSSocket.onConnectSecure (_tls_wrap.js:1515:34) at TLSSocket.emit (events.js:400:28) at TLSSocket.emit (domain.js:475:12) at TLSSocket._finishInit (_tls_wrap.js:937:8) at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:709:12) Aug-24 13:16:13.933 [Endpoint/13] [L0] core.rpc.rpc_base_conn:: RPC CONNECTION CLOSED. got event from connection: wss://noobaa-mgmt.openshift-storage.svc:8445(wliakpkgzz) WS CLOSED Aug-24 13:16:13.933 [Endpoint/13] [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv scrubber_api.make_room_in_tier reqid 1@wss://noobaa-mgmt.openshift-storage.svc:8445(wliakpkgzz) connid wss://noobaa-mgmt.openshift-storage.svc:8445(wliakpkgzz) params { tier: 64e5f4a345f814002dace812, bucket: 64e5f4ab45f814002dacef00 } Error: self signed certificate at TLSSocket.onConnectSecure (_tls_wrap.js:1515:34) at TLSSocket.emit (events.js:400:28) at TLSSocket.emit (domain.js:475:12) at TLSSocket._finishInit (_tls_wrap.js:937:8) at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:709:12) Aug-24 13:16:13.934 [Endpoint/13] [ERROR] core.server.object_services.map_server:: GetMapping: ERROR Error: self signed certificate at TLSSocket.onConnectSecure (_tls_wrap.js:1515:34) at TLSSocket.emit (events.js:400:28) at TLSSocket.emit (domain.js:475:12) at TLSSocket._finishInit (_tls_wrap.js:937:8) at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:709:12) Aug-24 13:16:13.934 [Endpoint/13] [ERROR] CONSOLE:: RPC._on_request: ERROR srv object_api.get_mapping reqid 605842@fcall://fcall(7rancrom) connid fcall://fcall(7rancrom) Error: self signed certificate at TLSSocket.onConnectSecure (_tls_wrap.js:1515:34) at TLSSocket.emit (events.js:400:28) at TLSSocket.emit (domain.js:475:12) at TLSSocket._finishInit (_tls_wrap.js:937:8) at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:709:12) Aug-24 13:16:13.935 [Endpoint/13] [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv object_api.get_mapping reqid 605842@fcall://fcall(7rancrom) connid fcall://fcall(7rancrom) params { chunks: [ { _id: undefined, master_key_id: '64e5f4ab45f814002dacef01', bucket_id: '64e5f4ab45f814002dacef00', tier_id: '64e5f4a345f814002dace812', chunk_coder_config: { replicas: 1, data_frags: 1, cipher_type: 'aes-256-gcm', digest_type: 'sha384', parity_frags: 0, compress_type: 'snappy', frag_digest_type: 'sha1' }, size: 4629, frag_size: 1203, compress_size: 1203, digest_b64: 'ifDQYH8EDuKIFvZICiUQIbqlDhbpnzQscdaGER8jGAoCSE+LA0IHdKDK8HCiNMsd', cipher_key_b64: 'mz5m/62fdJN/DohJlIEk0xxmrwaED0ZfPIzYTPo3Ifg=', cipher_iv_b64: undefined, cipher_auth_tag_b64: undefined, dup_chunk: undefined, is_accessible: undefined, is_building_blocks: undefined, is_building_frags: undefined, frags: [ { _id: undefined, data_index: 0, parity_index: undefined, lrc_index: undefined, digest_b64: 'vvVF6przkwKONssXdagRsj/KDTE=', blocks: [ length: 0 ], allocations: undefined }, length: 1 ], parts: [ { obj_id: '64e7581d82c2a2000d1fa86d', chunk_id: undefined, multipart_id: undefined, start: 0, end: 4629, seq: 0, uncommitted: true }, length: 1 ] },

], location_info: undefined, move_to_tier: undefined, check_dups: true } took [114.7+0.6=115.3] [RpcError: self signed certificate] { rpc_code: 'INTERNAL', rpc_data: { retryable: true } } Aug-24 13:16:13.935 [Endpoint/13] [ERROR] core.sdk.object_io:: UPLOAD: _upload_chunks [RpcError: self signed certificate] { rpc_code: 'INTERNAL', rpc_data: { retryable: true } } Aug-24 13:16:13.935 [Endpoint/13] [ERROR] core.sdk.object_io:: object_io._upload_stream_internal: error occured on stream Uploader: [RpcError: self signed certificate] { rpc_code: 'INTERNAL', rpc_data: { retryable: true } } Aug-24 13:16:13.935 [Endpoint/13] [ERROR] core.sdk.object_io:: _upload_stream error [RpcError: self signed certificate] { rpc_code: 'INTERNAL', rpc_data: { retryable: true } } Aug-24 13:16:13.935 [Endpoint/13] [WARN] core.sdk.object_io:: upload_object: failed upload { bucket: 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb', key: 'noobaa-23082023/repo2/backup/db/backup.info', md_conditions: undefined, obj_id: '64e7581d82c2a2000d1fa86d', size: 4629, num_parts: 1 } [RpcError: self signed certificate] { rpc_code: 'INTERNAL', rpc_data: { retryable: true } } Aug-24 13:16:13.941 [Endpoint/13] [L0] core.sdk.object_io:: upload_object: aborted object upload { bucket: 'pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb', key: 'noobaa-23082023/repo2/backup/db/backup.info', md_conditions: undefined, obj_id: '64e7581d82c2a2000d1fa86d', size: 4629, num_parts: 1 } Aug-24 13:16:13.941 [Endpoint/13] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?>InternalErrorWe encountered an internal error. Please try again./pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb/noobaa-23082023/repo2/backup/db/backup.infollp6tahq-3p0ihd-u89 PUT /pgbackrest-repo2-8f176dbb-50c3-4992-a5e8-24a69851f3fb/noobaa-23082023/repo2/backup/db/backup.info {"user-agent":"pgBackRest/2.41","authorization":"AWS4-HMAC-SHA256 Credential=b1Cxm7BFeY6BN8jPl9Ur/20230824/ap-southeast2/s3/aws4_request,SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date,Signature=b7158a9f5e204eb44492792fcffa27a83be51a3d62a543545c68a07695a9df69","content-length":"4629","content-md5":"8aEfPagLTjKtDCXQdVnmug==","x-amz-content-sha256":"dab0b2d568d72acd899693a48d08287ca2669731c7d7dd43a057e605a5b07723","x-amz-date":"20230824T131513Z","host":"s3-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com","x-forwarded-host":"s3-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.128.2.58;host=s3-openshift-storage.apps.ocp4-redacted.ocpdemos.redacted.com;proto=https","x-forwarded-for":"10.128.2.58"} [RpcError: self signed certificate] { rpc_code: 'INTERNAL', rpc_data: { retryable: true } } Aug-24 13:16:29.896 [Endpoint/13] [L0] core.sdk.object_io:: READ reader pos 5378 Aug-24 13:16:30.262 [Endpoint/13] [ERROR] CONSOLE:: RPC._on_request: ERROR srv object_api.read_object_md reqid 605851@fcall://fcall(7rancrom) connid fcall://fcall(7rancrom) Error: No such object: obj_id undefined bucket SENSITIVE-613260574318c85f key noobaa-23082023/repo2/backup/db/20230823-135735F_20230823-140006D/backup.manifest at check_object_mode (/root/node_modules/noobaa-core/src/server/object_services/object_server.js:1508:15) at find_object_md (/root/node_modules/noobaa-core/src/server/object_services/object_server.js:1429:5) at runMicrotasks () at processTicksAndRejections (internal/process/task_queues.js:95:5) at async Object.read_object_md (/root/node_modules/noobaa-core/src/server/object_services/object_server.js:779:17) at async /root/node_modules/noobaa-core/src/rpc/rpc.js:340:32 at async RPC._on_request (/root/node_modules/noobaa-core/src/rpc/rpc.js:348:25) Aug-24 13:16:30.262 [Endpoint/13] [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv object_api.read_object_md reqid 605851@fcall://fcall(7rancrom) connid fcall://fcall(7rancrom) params { bucket: SENSITIVE-613260574318c85f, key: 'noobaa-23082023/repo2/backup/db/20230823-135735F_20230823-140006D/backup.manifest', version_id: undefined, md_conditions: undefined, encryption: undefined } took [1.6+0.2=1.9] [RpcError: No such object: obj_id undefined bucket SENSITIVE-613260574318c85f key noobaa-23082023/repo2/backup/db/20230823-135735F_20230823-140006D/backup.manifest] { rpc_code: 'NO_SUCH_OBJECT' }

========================================================================

Also frequent below errors in noobaa core logs:

Aug-24 13:15:37.738 [HostedAgents/37] [ERROR] core.agent.block_store_services.block_store_google:: failed to delete block noobaa_blocks/644b5b71964845002a8c5e33/blocks_tree/4ac.blocks/64e610e982c2a2000d1fa4ac ApiError: No such object: noobaabucket767bb/noobaa_blocks/644b5b71964845002a8c5e33/blocks_tree/4ac.blocks/64e610e982c2a2000d1fa4ac at new ApiError (/root/node_modules/noobaa-core/node_modules/@google-cloud/common/build/src/util.js:73:15) at Util.parseHttpRespBody (/root/node_modules/noobaa-core/node_modules/@google-cloud/common/build/src/util.js:208:38) at Util.handleResp (/root/node_modules/noobaa-core/node_modules/@google-cloud/common/build/src/util.js:149:117) at /root/node_modules/noobaa-core/node_modules/@google-cloud/common/build/src/util.js:491:22 at onResponse (/root/node_modules/noobaa-core/node_modules/retry-request/index.js:228:7) at /root/node_modules/noobaa-core/node_modules/teeny-request/build/src/index.js:226:13 at runMicrotasks () at processTicksAndRejections (internal/process/task_queues.js:95:5) { code: 404, errors: [ { message: 'No such object: noobaabucket767bb/noobaa_blocks/644b5b71964845002a8c5e33/blocks_tree/4ac.blocks/64e610e982c2a2000d1fa4ac', domain: 'global', reason: 'notFound' } ], response: PassThrough { _readableState: ReadableState { objectMode: false, highWaterMark: 16384, buffer: BufferList { head: null, tail: null, length: 0 }, length: 0, pipes: [], flowing: true, ended: true, endEmitted: true, reading: false, sync: false, needReadable: false, emittedReadable: false, readableListening: false, resumeScheduled: false, errorEmitted: false, emitClose: true, autoDestroy: true, destroyed: true, errored: null, closed: true, closeEmitted: true, defaultEncoding: 'utf8', awaitDrainWriters: null, multiAwaitDrain: false, readingMore: false, dataEmitted: true, decoder: null, encoding: null, [Symbol(kPaused)]: false }, _events: [Object: null prototype] { prefinish: [Function: prefinish], error: [ [Function (anonymous)], [Function (anonymous)] ], data: [Function (anonymous)], end: [Function (anonymous)] }, _eventsCount:

pgfan1024 commented 1 year ago

I am wondering if this is the correct place to report bugs/issues related to noobaa. Please suggest.