search

iiasa / ixmp

The ix modeling platform for integrated and cross-cutting scenario analysis
https://docs.messageix.org/ixmp
Apache License 2.0
38 stars 112 forks source link

Progress indicator/logger for exports #365

Open jkikstra opened 4 years ago

jkikstra commented 4 years ago

For larger databases, using export_timeseries_data() on a ixmp.Platform object can take multiple hours. With this in mind, it would be very beneficial to log some feedback or show a progress bar, such that users can at least tell what is going on and whether anything is going wrong or not while waiting.

khaeru commented 4 years ago

This would indeed be a good feature.

click.progressbar (already a dependency) and tqdm (would be a new dependency) are two options for providing progress bars.

With the current default/only JDBCBackend, however, this kind of file I/O operation is a single Java function call: https://github.com/iiasa/ixmp/blob/5ad2ad9206f72ad5084288bdb0db8965cae2e3ac/ixmp/backend/jdbc.py#L510-L528

To display a progress indicator, the Python side / CLI code would require some kind of state variable(s) to be synchronized & checked, while that Java function continued running, i.e. this would need to be multi-threaded, something we currently don't do anywhere. Not impossible, but would require more advanced code than we currently have.