Open pawamoy opened 6 years ago
We could also improve import exec time by looking at UUID and start fields:
for session in data.sessions:
if session not in db:
insert(session.commands)
else:
session_last_command = get_session_last_command(db, session)
# with filter, might be slow, esp. with lambda
insert(filter(session.commands, lambda c: c.start > session_last_command.start))
# with sort, might be much faster
session_commands = sorted(session.commands)
pivot = None
for i, command in enumerate(session_commands): # FIXME: use dichotomy instead of naive iteration
if command.start > session_last_command.start:
pivot = i
break
if pivot is not None:
insert(session_commands[pivot:])
Finally we could end up with a fast session_last_command
import function, and a slower but "100% sure not to miss any command" dichotomy
import function.
Write a recursive function using dichotomy to insert objects in the database.
This will allow faster database updates from big history files.
Boost priority