search

nion-software / nionswift

Nion Swift is open source scientific image processing software integrating hardware control, data acquisition, visualization, processing, and analysis using Python. Nion Swift is easily extended using Python. It runs on Windows, Linux, and macOS.
http://nion.com/swift
GNU General Public License v3.0
44 stars 33 forks source link

Exporting data should occur in a thread #1031

Open cmeyer opened 4 months ago

cmeyer commented 4 months ago

Users often want to export to continue processing on another computer, but also continue doing acquisition while exporting. 

### Tasks
- [ ] Restructure ExportAction so that execute does the actual export using parameters (cf. OpenRunScriptsAction)
jamesrussell216 commented 3 months ago

@cmeyer just looking at this one - I think it might make sense to split this one in half Chris. One part for moving the exporting on to a thread (not sure about the architecture needs on that but that's a dev question/problem), the other part to hook up a progress bar for it. Is that OK with you?

cmeyer commented 3 months ago

@jamesrussell216 What about changing #749 to be the 'dialog + progress bar' and leaving this as the threading issue. They're probably going to need to be implemented at the same time, but #749 might be a good spot to fill out the details of the dialog. I'm thinking something like the 'downloads' progress list in a browser, except for exporting.

jamesrussell216 commented 3 months ago

@cmeyer I'd leave #749 be as that's more specific to an export bug. I'll make a new issue for the progress bar and link it in here. And then yes let's make this one specific to moving exports off the main thread. Yes they do make sense to do at the same time, but they are separate - arguably moving the exports on to a separate thread could be done (and released) without a progress bar is really needed, it would still be a step forward over the current situation I'd say. But equally yes it does really make sense to do both at the same time when we're "in the area" so to speak.

cmeyer commented 2 months ago

After expanding #1053 I decided that this one stands as the "export in a thread" issue. Adjusted title slightly and made #1053 dependent on this.

jamesrussell216 commented 2 months ago

@cmeyer you mentioned in chat you had some thoughts about the potential best route to implement this - can you drop a reference of some form in here for us plz?

cmeyer commented 2 months ago

I can write up a more thorough explanation, but quickly: the periodic function executes on the UI thread every 50ms and runs pending async functions. Those async functions usually suspend themselves quickly (~10ms), letting other UI-level functions run. One of the ways they can suspend themselves is by running another function in a thread via the run_in_executor function and then awaiting that function. We should run the export thread via this mechanism.

An example of this is here:

https://github.com/nion-software/nionswift/blob/824d9d084ee91a4842f23539ff4055d6cca02ccf/nion/swift/model/DisplayItem.py#L3301-L3324

You can trace through the calls to get_value_and_position_text_async to see how it gets called from periodic.

We'll also need to design a mechanism for handling shutdown, cancellation, and other similar situations. We may want to discuss implementation plans in more detail before starting. There are a lot of pitfalls to this project.

jamesrussell216 commented 2 months ago

@cmeyer worth getting it on the agenda for tomorrow's software call?

cmeyer commented 2 months ago

Currently the export code resides in the export_clicked method in ExportDialog.py. It should be moved to reside in the ExportAction.execute method. It is ok for the execute method to extract the necessary parameters from the action context and then call a method in DocumentModel or standalone function or just put the code in ExportAction.execute. This is a useful refactoring to prepare for implementing threaded export.