request: multithreading use case with variable downloads

[MPagel]

I don't currently have a specific use case, but one thing you might want to consider is adding an example/tutorial on dynamically changing counts of downloads that are being processed in a multithreaded fashion.

As an example, once upon a time I was using tqdm to attempt to handle and track multiple downloads of AWS SQS messages, but it was a bloody mess. The setup was a check of a message queue and retrieving a list of up to 10 message ids at once (without contents). Then I launched threads to download each of those messages individually, but I would continue to see if there was another block of 10 message ids to add to the queue. These ids would also be processed for additional downloads. I had a high fixed number of simultaneous download threads, but additionally didn't want to have any individual message files take more than 2 minutes to download, so would adjust my number of threads dynamically to reflect different processing speeds.

So, I wanted a table, similar to what you have for the download example, but dividing the display into 3 panels: active downloads, completed downloads and an overall status bar. The active downloads panel would display all downloading message IDs. The completed downloads would occupy a panel that covered the rest of the screen space devoted to the table. The overall status bar would that would show number of completed / active / total download count where even the total count may be increasing over time.

That was one previous use case that I had. I am not working on the project any more. So, feel free to ignore this request or simplify the type of display as you see fit.

[sjmikler]

Hey, thanks for the request!

So in your use-case, if represented in a table, each download would occupy a single row? If so, can you describe the differences between this and one of the examples I included in the README?

If I understood correctly, in your use-case, new rows might be added in the meantime. Is this the only difference?

[MPagel]

that'd be the primary difference. A potential complication of the use-case is to be able to show a considerable number of additional rows, such that keeping all on the screen at the same time would not be possible. So, it would essentially be the download example combined with the Brownian motion example.

I would also like to keep the dynamically updating portion to occupy only up to a certain number of lines, and allow other log lines (e.g. the text of the actual "warnings" that are being counted in the last column) to scroll away above (or below) this panel.

[sjmikler]

Hey, so I modified the example as described and saved it as download_v2.py. It is a subtle change, but now the number of rows is increasing, as if users were making download requests in real time.

Moving rows to "finished" would need to be done using .at indexing, like in the brown2d example:

def shift_rows_up(table, last_row_color):
    num_rows = table.num_rows()
    num_columns = table.num_columns()

    for row in range(num_rows - 1):
        for col in range(num_columns):
            table.at[row, col] = table.at[row + 1, col]
            table.at[row, col, "color"] = table.at[row + 1, col, "color"]
    for col in range(num_columns):
        table.at[-2, col, "color"] = last_row_color

See also: advanced usage.