UI reporting - performance analysis

[tpurschke]

• details of test run with large data set point into the direction of middleware server c# bottleneck issue: https://github.com/CactuseSecurity/FWO_private/blob/master/dev-infos/performance-testing.md
• make sure API calls are "sane"
• is relevant import id correctly taken into account for each management?
• make all parameters mandatory to prevent api call creep
• what causes time difference (5000 rule report generation) between direct api call via curl (<3s) and FWO UI (>10 min) (was: coding error)

time curl --request POST      --insecure     --url https://localhost:9443/api/v1/graphql     --header 'content-type: application/json'     --header    'Accept: application/json'     --header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1bmlxdWVfbmFtZSI6ImFkbWluIiwieC1oYXN1cmEtdXNlci1pZCI6IjIiLCJ4LWhhc3VyYS11dWlkIjoidWlkPWFkbWluLG91PXRlbmFudDAsb3U9b3BlcmF0b3Isb3U9dXNlcixkYz1md29yY2gsZGM9aW50ZXJuYWwiLCJ4LWhhc3VyYS10ZW5hbnQtaWQiOiIxIiwieC1oYXN1cmEtdmlzaWJsZS1tYW5hZ2VtZW50cyI6InsgMiw0LDUsMSB9IiwieC1oYXN1cmEtdmlzaWJsZS1kZXZpY2VzIjoieyAxLDIsNCw1IH0iLCJyb2xlIjoiYWRtaW4iLCJ4LWhhc3VyYS1hbGxvd2VkLXJvbGVzIjpbImFkbWluIl0sIngtaGFzdXJhLWRlZmF1bHQtcm9sZSI6ImFkbWluIiwibmJmIjoxNjM5MDczODM3LCJleHAiOjE2MzkwODgyOTcsImlhdCI6MTYzOTA3MzgzNywiaXNzIjoiRldPIE1pZGRsZXdhcmUgTW9kdWxlIiwiYXVkIjoiRldPIn0.JBw_3yPzLF2G9NwL5Xa4gBC9LyQg_H1B3PoSKZ_qHl53R25x-GPpFHaPGk3rVR4SaWLw52nCv7GBItLvnQZZKhPfrbtOv_Roy4yKTvrcNfqfs_1-Y7HSmWXTVZiHe8UJJthBRKr4oT7zwtslnjJo9E4jnPr7IKSA9ytZuc274csogJLGUxBhLVIja256gQTC65lDvQzCU3tQe0Qjvv1BFGTaT17NMrtAQTPsSMQidIrec9M6h_QUBbI_Oa8Q5ziA86BI_4vcrang8_WTc3VH4VVg9sRX5Kbv3proCuOni3Qi6pC5WP28MSaKmnDsg_VYg5BJ2yES6ZBbCVGsZIN_2w'     --data '{"variables":{"gwName0":"forti-05000"},"query":"fragment networkObjectOverview on object { obj_ip obj_ip_end obj_name obj_id type: stm_obj_typ { id: obj_typ_id name: obj_typ_name } obj_color_id}fragment networkServiceOverview on service { svc_id svc_name svc_uid svc_port svc_port_end service_type: stm_svc_typ { id: svc_typ_id name: svc_typ_name } svc_color_id protocol_name: stm_ip_proto { id: ip_proto_id name: ip_proto_name }}fragment userOverview on usr { user_id user_uid user_name stm_usr_typ { usr_typ_name }}fragment ruleOverview on rule { rule_id rule_uid rule_action section_header: rule_head_text rule_comment rule_track rule_disabled src_zone: zone { zone_name zone_id } rule_metadatum { rule_metadata_id rule_created rule_first_hit rule_last_hit rule_last_modified rule_last_certified rule_last_certifier_dn rule_to_be_removed rule_decert_date rule_recertification_comment } rule_src_neg rule_dst_neg rule_svc_neg rule_num_numeric rule_name access_rule nat_rule xlate_rule rule_froms(where: {object:{obj_create:{_lte:$relevantImportId}, obj_last_seen:{_gte:$relevantImportId}}}) { usr { ...userOverview } object { ...networkObjectOverview } } dst_zone: zoneByRuleToZone { zone_name zone_id } rule_tos(where: {object:{obj_create:{_lte:$relevantImportId}, obj_last_seen:{_gte:$relevantImportId}}}) { object { ...networkObjectOverview } } rule_services(where: {service:{svc_create:{_lte:$relevantImportId}, svc_last_seen:{_gte:$relevantImportId}}}) { service { ...networkServiceOverview } }} query rulesReport ( $limit: Int $offset: Int $mgmId: [Int!] $relevantImportId: bigint $gwName0: String! ) { management( where: { mgm_id: {_in: $mgmId }, hide_in_gui: {_eq: false } } order_by: { mgm_name: asc } ) { id: mgm_id name: mgm_name devices ( where: { hide_in_gui: {_eq: false } } order_by: { dev_name: asc } ) { id: dev_id name: dev_name rules( limit: $limit offset: $offset where: { access_rule: {_eq: true} _and: [{}, {_and: [{import_control: { control_id: {_lte: $relevantImportId } }, importControlByRuleLastSeen: { control_id: {_gte: $relevantImportId } }}, {device: {dev_name : {_ilike: $gwName0 } }}] }] } order_by: { rule_num_numeric: asc } ) { ...ruleOverview } } } }"}' | python3 -mjson.tool > config.json

real    0m2.375s
user    0m1.572s
sys 0m0.123s

tim@acantha:~$ wc -l config.json 
789440 config.json
tim@acantha:~$

so pure report generation (Overview) of ca. 5.000 rules via API only takes 2 seconds

• [x] maybe we can re-add timing data to the log in debug mode during report generation again (time the different generation steps take)
• [x] rule numbering does not work anymore or rulesets with header lines (broken by performance improvement --> @NilsPur )
• [x] is there a hard memory limit of 4 GB for the middleware server?
• [x] need to generate a new JWT after the initial long API call to be safe
• [x] need to improve perfomance for the API call!
• [ ] need to find out why there is a 5s delay between the API network packets
• [x] need to improve error catching in MW server so it won't just stop
• [ ] need to improve logging in UI server as it does not show any entry while logging in when MW server is dead (just the browser shows "> is an unexpected token")
• [ ] API might crash with big scheduled reports (30.000 rules)

[tpurschke]

mostly done