Improve Performance of Orders Control Panel

[alecpm]

Based on issues with NCI order list performance with over 8000 orders, we need to make improvements to the control panel and CSV export performance. See jazkarta/nci-redesign#200

The control panel has date filtering, but it's all done after retrieving the full BTree of orders. The internal storage is somewhat optimized for a more limited retrieval of sorted items, but that organization is not really being used. The _fetch_orders method can probably be extended to support a batch start, batch size, start date, and end date, and optimized to only retrieve orders within those parameters.

The storage is organized with datetime objects as BTree keys, so the dates can be filtered and sorted without retrieving the order objects. There are two branches with order data in the BTree:

'orders' -> date -> order_obj userid -> 'orders' -> date -> order_obj

We will need to loop over all the userids in in the order storage, which will not be super performant on sites with a lot of users actively making purchases, but we can do something like:

b_end = b_start + b_size
keys = []
values = []
for entry in storage.keys():
    if entry in EXCLUDED_KEYS:
        continue
    if entry == 'orders':
       user = None
       container = storage[entry]
    else:
        user = entry
        container = storage[entry]['orders']
    for count, date in enumerate(reversed(container.keys(min=start_date, max=end_date)))):
        keys.append({'user': user, 'date': date})
        if count >= b_end:
            break
keys.sort(key=lambda k: k['date'], reverse=True)
keys = keys[b_start:b_end]
for key in keys:
    user = key['user']
    if user:
        container = storage[user]['orders']
    else:
        container = storage['orders']
    entry = container[key['date']]
    ... do stuff to process entry for display ...
    values.append(processed_entry)
return values

Rather than retrieve the full BTree and sort it, this pulls only those entries within the specified date range up to a specified page limit. I think this sort of thing should be much more RAM and CPU efficient.

[alecpm]

This proof of concept needs some additional work to support providing the total count of date filtered results for batch generation, but that should just mean getting all filtered and sorted keys and then getting/slicing them on-demand.