search

noobaa / noobaa-core

High-performance S3 application gateway to any backend - file / s3-compatible / multi-clouds / caching / replication ...
https://www.noobaa.io
Apache License 2.0
268 stars 78 forks source link

last buil 2.0.0-0cebe13 is broken - 'chunk_coder_config' of undefined #3864

Closed YuliaKovalenko closed 6 years ago

YuliaKovalenko commented 6 years ago

Environment info

Actual behavior

  1. RROR srv system_api.read_system Error: Cannot read property 'chunk_coder_config' of undefined

Expected behavior

  1. response is correct

Steps to reproduce

  1. Update system with last successful build v 2.0.0-0cebe13

Screenshots or Logs or other output that would be helpful

debug_module.js:442 Nov-12 14:25:12.221 [ERROR] core.rpc.rpc:: RPC._request: response ERROR srv system_api.read_system params undefined reqid 18@ws://51.141.160.102:8080(2c2c5hk.0002) took [130.9+281.0=411.9] Error: Cannot read property 'chunk_coder_config' of undefined at new RpcError (http://51.141.160.102:8080/fe/api.js:72006:132) at RpcRequest._set_response (http://51.141.160.102:8080/fe/api.js:72180:30) at RPC._on_response (http://51.141.160.102:8080/fe/api.js:71221:26) at RPC._on_message (http://51.141.160.102:8080/fe/api.js:71550:18) at RpcWsConnection.<anonymous> (http://51.141.160.102:8080/fe/api.js:71357:23) at RpcWsConnection.EventEmitter.emit (http://51.141.160.102:8080/fe/api.js:21092:17) at WebSocket.ws.onmessage (http://51.141.160.102:8080/fe/api.js:72517:28) InternalDebugLogger.log_internal @ debug_module.js:442

guymguym commented 6 years ago

@YuliaKovalenko /CC @liranmauda Can you please copy the complete stack trace? It will allow me to find the problem. Thanks

YuliaKovalenko commented 6 years ago

@guymguym , added stack trace, example of system http://51.141.160.102:8080

guymguym commented 6 years ago

@YuliaKovalenko, In this case, the relevant stack trace is on the server. Can you please fetch the log from the server? I can't login to the server because read_system fails so I can't find the server secret...

YuliaKovalenko commented 6 years ago

@guymguym , oops, sorry. server 51.141.160.102 secret c3ca2470

[WARN] core.util.mongo_client:: INVALID_SCHEMA_DB tieringpolicies ERRORS [ { keyword: 'required', dataPath: '', schemaPath: '#/required', params: { missingProperty: 'chunk_split_config' }, message: 'should have required property \'chunk_split_config\'', schema: { _id: { objectid: true }, name: { type: 'string' }, system: { objectid: true }, deleted: { date: true }, chunk_split_config: { '$ref': 'common_api#/definitions/chunk_split_config' }, tiers: { type: 'array', items: { type: 'object', required: [ 'order', 'tier', [length]: 2 ], properties: { order: { type: 'integer' }, tier: { objectid: true }, spillover: { type: 'boolean' }, disabled: { type: 'boolean' } } } } }, parentSchema: { id: 'tiering_policy_schema', type: 'object', required: [ '_id', 'name', 'system', 'chunk_split_config', 'tiers', [length]: 5 ], properties: { _id: { objectid: true }, name: { type: 'string' }, system: { objectid: true }, deleted: { date: true }, chunk_split_config: { '$ref': 'common_api#/definitions/chunk_split_config' }, tiers: { type: 'array', items: { type: 'object', required: [ 'order', 'tier', [length]: 2 ], properties: { order: { type: 'integer' }, tier: { objectid: true }, spillover: { type: 'boolean' }, disabled: { type: 'boolean' } } } } } }, data: { _id: 5a02114d73929b0432f4558b, system: 5a02114d73929b0432f45588, name: 'first.bucket#j9q1l29t', tiers: [ { tier: 5a02114d73929b0432f4558a, order: 0, spillover: false, disabled: false }, { tier: 5a02115073929b0432f45593, order: 1, spillover: true, disabled: false }, [length]: 2 ] } }, [length]: 1 ] DOC { _id: 5a02114d73929b0432f4558b, system: 5a02114d73929b0432f45588, name: 'first.bucket#j9q1l29t', tiers: [ { tier: 5a02114d73929b0432f4558a, order: 0, spillover: false, disabled: false }, { tier: 5a02115073929b0432f45593, order: 1, spillover: true, disabled: false }, [length]: 2 ] } and

Nov 13 10:14:20 noobaa WebServer[19475]: [WARN] core.util.mongo_client:: INVALID_SCHEMA_DB nodes ERRORS [ { keyword: 'objectid', dataPath: '.agent_config', schemaPath: '#/properties/agent_config/objectid', params: { keyword: 'objectid' }, message: 'should pass "objectid" keyword validation', schema: true, parentSchema: { objectid: true }, data: null }, [length]: 1 ] DOC { _id: 5a02115673929b0432f45597, peer_id: 5a02115673929b0432f45598, system: 5a02114d73929b0432f45588, pool: 5a02115073929b0432f45592, agent_config: null, heartbeat: 1510517658348, name: 'noobaa-internal-agent-5a02115073929b0432f45592', is_mongo_node: true, drives: [ { mount: '/', drive_id: '/dev/sda2', storage: { total: 31671447552, free: 27711877120, used: 73728 } }, [length]: 1 ], latency_to_server: [ 189, 175, 184, 189, 194, 195, 180, 209, 184, 180, 197, 176, 195, 211, 193, 195, 193, 201, 185, 215, [length]: 20 ], latency_of_disk_read: [ 2.8812819980084896, 2.026357002556324, 4.286922000348568, 3.460698001086712, 3.3973970003426075, 4.1337179988622665, 4.764934998005629, 9.968283001333475, 1.5729439966380596, 4.754034999758005, 2.352767001837492, 2.1246610023081303, 1.9957569986581802, 2.037158001214266, 1.7550499998033047, 1.7000479996204376, 1.7081490010023117, 1.6728479973971844, 2.00755700096488, 1.951954998075962, [length]: 20 ], latency_of_disk_write: [ 13.725090000778437, 2.834681000560522, 3.7403069995343685, 3.472198002040386, 4.632431000471115, 2.988383997231722, 2.559773001819849, 2.9064819999039173, 3.5717019997537136, 10.953910999000072, 3.5956020019948483, 3.3911969996988773, 3.5798010006546974, 3.7359060011804104, 4.07761599868536, 8.924953002482653, 2.8582810014486313, 3.1022880002856255, 3.560301002115011, 17.621399998664856, [length]: 20 ], storage: { total: 5368709120, free: 5368635392, used: 73728 }, version: '2.0.0-a2926aa', ip: '10.1.15.6', base_address: 'wss://127.0.0.1:8443', rpc_address: 'n2n://5a02115673929b0432f45598', enabled: true, geolocation: 'Sao Paulo', os_info: { hostname: 'noobaa', ostype: 'Centos 7.4.1708 (core)', platform: 'linux', arch: 'x64', release: '3.10.0-693.5.2.el7.x86_64', uptime: 1509879403284, loadavg: [ 0.025390625, 0.08984375, 0.1337890625, [length]: 3 ], totalmem: 4135690240, freemem: 991862784, cpus: [ { model: 'Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz', speed: 2397, times: { user: 112436100, nice: 7900, sys: 39963400, idle: 6105292000, irq: 0 } }, { model: 'Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz', speed: 2397, times: { user: 112361500, nice: 6900, sys: 39864900, idle: 6111883600, irq: 0 } }, [length]: 2 ], networkInterfaces: { lo: [ { address: '127.0.0.1', netmask: '255.0.0.0', family: 'IPv4', mac: '00:00:00:00:00:00', internal: true }, { address: '::1', netmask: 'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff', family: 'IPv6', mac: '00:00:00:00:00:00', scopeid: 0, internal: true }, [length]: 2 ], eth0: [ { address: '10.1.15.6', netmask: '255.255.255.0', family: 'IPv4', mac: '00:0d:3a:fd:14:23', internal: false }, { address: 'fe80::20d:3aff:fefd:1423', netmask: 'ffff:ffff:ffff:ffff::', family: 'IPv6', mac: '00:0d:3a:fd:14:23', scopeid: 2, internal: false }, [length]: 2 ] } }, debug_level: 0, node_type: 'BLOCK_STORE_MONGO', host_name: 'noobaa', mem_usage: 86417408, cpu_usage: 0.004876137917397309, host_id: 'hosted_agents5a02115073929b0432f45592', host_sequence: 1, n2n_config: { offer_ipv4: true, offer_ipv6: false, accept_ipv4: true, accept_ipv6: true, offer_internal: false, tcp_active: true, tcp_permanent_passive: { port: 60100 }, tcp_transient_passive: false, tcp_simultaneous_open: false, tcp_tls: true, udp_port: true, udp_dtls: true, stun_servers: [ [length]: 0 ] } }

guymguym commented 6 years ago

@YuliaKovalenko /CC @liranmauda,

Regarding INVALID_SCHEMA_DB tieringpolicies - This was introduced in #3856, but I think that you tried to upgrade a system that was created before that PR. But since we do not want to perform upgrades within 2.0 we did not handle conversion of these structures.

Regarding INVALID_SCHEMA_DB nodes - this is a warning only, which was fixed in #3859.

Do I believe these two issues will not be reproduced on a clean system. Please start the system from clean DB and let me know if any of these is still happening.

Thanks

YuliaKovalenko commented 6 years ago

@guymguym it always reproduced when we update server from previous version 2.XX