Improve logging mechanism

[jpfeuffer]

• Make the "log" parameter in KNIME a boolean that creates a log file in the filestore. (Avoids flow variable magic before every node to create a unique filename)
• Investigate if it is possible to parse the output and update the progress bar?
• Avoid buffer overflow if outputs are too long

[jpfeuffer]

@timosachsenberg What do you think about removing the whole stream gobbling and just by standard write the whole output of a tool to a file and read it in again in the end? I was investigating possible causes for slower execution times and most of the times people report that gathering stdout makes it quite slow.

[timosachsenberg]

Sounds good as stdout streams are pretty slow on windows. Would still be nice to get some feedback on running tools - especially the progress.

[jpfeuffer]

I hope to include a right-click menu item that basically displays the current contents of this log file. Real progress is hard because a lot of tools consist of different steps for which they increment their progress percentage seperately

[jpfeuffer]

Even if each tool displays total progress in command line, it could be slow again to repeatedly parse the last lines of the written log file and update the Node's progress bar.

[timosachsenberg]

In the worst case we might need to write the progress to a dedicated file by extending the ProgressLogger logic

[jpfeuffer]

Ah in OpenMS? Hm true.

[jpfeuffer]

But then we need to make sure this file is always found by the KNIME code. If it is not an explicit output parameter.

[timosachsenberg]

yeah would probably mean adding another "-log" parameter to TOPPBase. And setting it from GKN

[timosachsenberg]

Let's ask the KNIME guys. They might have some experience with that

[jpfeuffer]

Also our implementation is very confusing. The generic nodes actually have a View class in their folder but this does not seem to be used because the Factory uses a ExtToolStdoutNodeView that is provided by KNIME. Also this class from KNIME actually provides mechanisms to iteratively update the View. I have no idea why they are not used. I am having a really hard time right now on the KNIME Server to see what is going wrong because the error messages etc. are not really stored somewhere. I know the must be during saving somewhere but I can not find them.

[jpfeuffer]

Ok. Locally I have in the workspace under knime-workspace/myworkflow/mynode(#1)/internal/execErrout and execOutput the saved output. This is not present on the KNIME Server.

[jpfeuffer]

I have a local branch where I now read the Logs on the fly. PR coming soon.

[jpfeuffer]

fixed in #249