pawamoy / shell-history

Visualize your shell usage with Highcharts!
ISC License
112 stars 2 forks source link

Use dichotomy instead of "one by one" when db commit fails #15

Open pawamoy opened 6 years ago

pawamoy commented 6 years ago

Write a recursive function using dichotomy to insert objects in the database.

This will allow faster database updates from big history files.

Boost priority

Fund with Polar

pawamoy commented 6 years ago

We could also improve import exec time by looking at UUID and start fields:

for session in data.sessions:
  if session not in db:
    insert(session.commands)
  else:
    session_last_command = get_session_last_command(db, session)
    # with filter, might be slow, esp. with lambda
    insert(filter(session.commands, lambda c: c.start > session_last_command.start))
    # with sort, might be much faster
    session_commands = sorted(session.commands)
    pivot = None
    for i, command in enumerate(session_commands):  # FIXME: use dichotomy instead of naive iteration
      if command.start > session_last_command.start:
        pivot = i
        break
      if pivot is not None:
        insert(session_commands[pivot:])

Finally we could end up with a fast session_last_command import function, and a slower but "100% sure not to miss any command" dichotomy import function.