Collect and visualize portion distribution

[zverevgeny]

Tool(scritp) that:

• collects data from .sys/primary_index_portion_stats
• visualizes retangles [(minSnapshot, minTimestamp), (maxSnapshot, maxTimestamp)] + optional some additional info

[vlad-gogov]

Instructions

Example run:

python3 intersect_portions.py --parsing_file=example1.txt

Output in terminal:

Parsing rectangles: 100001
remaining rectangles: 90001
remaining rectangles: 80001
remaining rectangles: 70001
remaining rectangles: 60001
remaining rectangles: 50001
remaining rectangles: 40001
remaining rectangles: 30001
remaining rectangles: 20001
remaining rectangles: 10001
remaining rectangles: 1
parsing time: 14.31 seconds
count drawn rectangles: 100001

In the lower left corner, you will find a control panel that allows you to move around the graph and adjust the zoom.

Rectangles are drawn with black dotted lines. Rectangle with a zero height or width are represented as red solid lines in the drawing.

Additional args for run:

• --parsing_file= - path to file with portion's logs (after run, file is create with name *_save_parsing.txt for next read with arg --read_from_save)
• --read_from_save= - path to file for read portion's logs after first run with --parsing_file
• --pk_min= - left border for drawing (DataTime format Y-m-d H:M:S)
• --pk_max= - right border for drawing (DataTime format Y-m-d H:M:S)
• --snapshot_plan_step_min= - down border for drawing (ms)
• --snapshot_plan_step_max= - up border for drawing (ms)
• --max_intersect - finds area with maximum intersection and moves to it.
• --drawing_step= - step to select rectangles
• --limit_rows= - read first limit_rows rows from file

Run with max_intersect:

python3 intersect_portions.py --parsing_file=example1.txt --max_intersect

It is necessary to find the area with the maximum intersection along the X-axis.

Output in terminal:

Parsing rectangles: 100001
remaining rectangles: 90001
remaining rectangles: 80001
remaining rectangles: 70001
remaining rectangles: 60001
remaining rectangles: 50001
remaining rectangles: 40001
remaining rectangles: 30001
remaining rectangles: 20001
remaining rectangles: 10001
remaining rectangles: 1
parsing time: 14.02 seconds
max_intersect_interval: (2024-11-11 15:49:41+00:00, 2024-11-11 15:59:47+00:00), max count intersect: 2624
count drawn rectangles: 100001

Run with set border:

python3 intersect_portions.py --parsing_file=example1.txt --pk_min="2024-11-11 16:00:00" --pk_max="2024-11-11 18:00:00"

Output in terminal:

Parsing rectangles: 100001
remaining rectangles: 90001
remaining rectangles: 80001
remaining rectangles: 70001
remaining rectangles: 60001
remaining rectangles: 50001
remaining rectangles: 40001
remaining rectangles: 30001
remaining rectangles: 20001
remaining rectangles: 10001
remaining rectangles: 1
parsing time: 9.50 seconds
count drawn rectangles: 12373