feat: Maintain array of points to query for each node

[KShivendu]

This PR does 3 things

• Port consistency checking logic to python
• Maintain array of points to query for each node (and retry fewer points in next attempts)
• Increase number of attempts to 5

I ported consistency check logic to python because:

• The bash script was getting unnecessarily complicated (and lengthy)
• It's relatively complicated to create and handle JSON data in bash despite jq. Python has much better tools for handling dictionaries and lists.
• It's harder to debug bash scripts because of missing tracebacks. I tried using trap but that wasn't sufficient.

For example:

# python:
def calculate_inconsistent_points(source_points, target_points, point_ids):
    source_point_idx_to_vector = {
        point["id"]: point["vector"] for point in source_points
    }
    target_point_idx_to_vector = {
        point["id"]: point["vector"] for point in target_points
    }

    inconsistent_point_ids = []
    for point_id in point_ids:
        if source_point_idx_to_vector.get(point_id) != target_point_idx_to_vector.get(
            point_id
        ):
            # Mismatching or missing points
            inconsistent_point_ids.append(point_id)

    return inconsistent_point_ids

inconsistent_points = calculate_inconsistent_points(
  first_node_points, fetched_points, point_ids
)

VS

# bash:
function calculate_inconsistent_points() {
    source_points=$1
    target_points=$2
    point_ids_to_check=$3

    inconsistent_points=()

    source_point_id_to_vector=$(echo "$source_points" | jq -r 'map({id: .id, vector: .vector}) | from_entries')
    target_point_id_to_vector=$(echo "$target_points" | jq -r 'map({id: .id, vector: .vector}) | from_entries')

    for point_id in $(echo "$point_ids_to_check" | jq -r '.[]'); do
        source_vector=$(echo "$source_point_id_to_vector" | jq -r --arg point_id "$point_id" '.[$point_id]')
        target_vector=$(echo "$target_point_id_to_vector" | jq -r --arg point_id "$point_id" '.[$point_id]')

        if [ "$source_vector" != "$target_vector" ]; then
            inconsistent_points+=("$point_id")
        fi
    done

    echo "${inconsistent_points[@]}"
}

inconsistent_points=()
array_response=$(calculate_inconsistent_points "$first_node_points" "$fetched_points" "$point_ids")
read -ra inconsistent_points <<< "$array_response"

[KShivendu]

To test this PR. I killed node-2 and it was working as expected.

[KShivendu]

Merging for now. Feel free to drop feedback.