How to Quickly Insert Millions of FIB Entries

[SLKyrim]

The slow insertion process might be caused by the fact that each FIB entry outputs an entry ID, as shown below. Is it possible to suppress this output to speed up the insertion of FIB entries?

{"id":"Q75A67DO4ELIN7SU2JJH6EMRIAMVFV1T93V649E13UJH3438GBC9QAFNU865TV758UIO74LE69U5RFLUAOQGJH0NLMJGEVI51RE47CL6IVS6RO2Q"} {"id":"Q75A67DO4ELIN7SU3RO1G8D1SAUVRPQC8U95GG5I3AJG7438GBC9QAFNU865TV758UIO74LE69U5RFLUAST0HH8LLAMG4UAF139KJEDFIVS61OIQ"} {"id":"Q75A67DO4ELIN7SU3RO1G8D1SEMF36I646E3AADH02U0B438GBC9QAFNT99GL8VU2VQT535C65U5N1KUASVGRI8MI8"} {"id":"Q75A67DO4ELIN7SU3RO1G8D1SAUVRPQC8I55SGD3FB501S82IMSUCVUCPH807ANU33VDH2DA65UL90KSA8QGHS8"} {"id":"Q75A67DO4ELIN7SU3RO1G8D1SAUVRPQC8I55SGD3FB501S82IMSVU84RI17LPSD01JQDN3DD19D5J1KIAGQ0JH0LLIM00VQC1NA45C50JF60"}

[yoursunny]

The public interface of NDN-DPDK service is the GraphQL API, rather than ndndpdk-ctrl command. If you need to execute millions of management commands, you should use the GraphQL API (from your own JavaScript, etc, application) instead of relying on CLI tools.

[SLKyrim]

I am not familiar with GraphQL. Could you provide a tutorial? For example, I have a TXT file containing millions of names. How can I use the GraphQL API to read this file and quickly insert the entries?

[yoursunny]

You can search "how to call GraphQL API with [insert your preferred programming language]" for tutorials.

See ndndpdk-ctrl README for the proper GraphQL endpoint address and how to retrieve the GraphQL schema through introspection. The ndndpdk-ctrl command and ndn-dpdk/sample/benchmark application serve as examples.

[SLKyrim]

Thank you for the prompt. I have implemented a Python script, and it takes about 4 minutes to insert 100,000 FIB entries on my server.

import requests
import json
import sys
import time

GQL_ENDPOINT = "http://127.0.0.1:3030/"

# Get the interface ID from command line arguments
interface_id = sys.argv[1] if len(sys.argv) > 1 else None

if interface_id is None:
    print("Please provide the interface ID as a parameter.")
    sys.exit(1)

# Define the function to insert FIB entries
def insert_fib_entry(name, nexthops):
    mutation = """
    mutation {
        insertFibEntry(
            name: "%s",
            nexthops: [%s]
        ) {
            name
        }
    }
    """ % (name, ', '.join(f'"{nh}"' for nh in nexthops))

    response = requests.post(GQL_ENDPOINT, json={'query': mutation})
    return response.json()

# Function to print the progress bar
def print_progress_bar(iteration, total, length=20):
    percent = (iteration / total) * 100
    filled_length = int(length * iteration // total)
    bar = '#' * filled_length + '-' * (length - filled_length)
    # Use \r to print the progress bar on the same line
    sys.stdout.write(f'\r[{bar}] {percent:.2f}%')
    sys.stdout.flush()

# Main function
def main():
    # Read FIB names from txt file
    with open('names.txt', 'r') as f:
        names = [line.strip() for line in f]

    total_entries = len(names)

    for i, name in enumerate(names):
        fib_response = insert_fib_entry(name, [interface_id])
        print_progress_bar(i + 1, total_entries)

if __name__ == "__main__":
    main()