Silent publishing

[tokejepsen]

This PR gives a built-in option for end-users to publish without any GUI.

[mottosso]

I think for this to work, there will have to be some indicator progress. Otherwise you'd have no way of knowing whether it froze or just takes a really long time.

It'd be simple enough to do, we can convert util.publish into an iterator, and have it emit result dictionaries per plug-in. Then your splash gui could simply print the currently processed plug-in and perhaps overall status (errored, or all-clear).

[tokejepsen]

With the publish iterator on its way, let's talk about how to show the progress.

Initially I was thinking of a simple label of the plugin plus success/failed icon:

Ideally a progress bar would be better, but getting a total count of the plugins won't be possible without collecting first, unless you would count collection as a 1/4 of the progress.

[mottosso]

Ideally a progress bar would be better, but getting a total count of the plugins won't be possible without collecting first, unless you would count collection as a 1/4 of the progress.

That's a good point.

Maybe it'd be ok to yield different results at different times? For example..

iterator = pyblish.util.publish_iter()
plugins, context = next(iterator)  # Before collection
plugins, context = next(iterator)  # After collection, but before main processing
for Plugin, context in iterator:
  ...

Not sure how convenient that is..

[tokejepsen]

Maybe it'd be ok to yield different results at different times? For example..

Not entirely sure what you mean here.

We could also have two progress bars; one for collection and one for the main processing? Maybe that is what you meant...

[mottosso]

Not entirely sure what you mean here.

I mean yielding at multiple times in the function. It doesn't just have to come from the loop.

def publish():
  context = api.Context()
  yield context
  for collector in [SomeCollector]:
    api.process(collector)
  yield context

  for Plugin, instance in api.discover():
    yield Plugin, instance

Now when you call next(iterator) you'll get the first yield. When you call it again, you'll get the second yield. The third time onwards, you'll get the results from the loop.

[tokejepsen]

Alright let's walk through an example as I see it working.

I have ten plugins; three collectors, three validator, three extractors and one integrator.

• Initially all plugins are counted as the total, but 0 is processed:
  • 0/10 = 0
• The collection is run, so 3 plugins are getting processed:
  • 1/10 = 0.1
  • 2/10 = 0.2
  • 3/10 = 0.3
• The collection is done and a validator and extractor gets disregarded because of missing instances:
  • 3/8 = 0.375
• The remaining plugins get processed:
  • 4/8 = 0.5
  • 5/8 = 0.625
  • 6/8 = 0.75
  • 7/8 = 0.875
  • 8/8 = 1

Depending on the amount of plugins disregarded, the jump between the collection increment and the rest of the increments could be quite large. Also need to note that we are not considering the amount of instances a plugin needs to be process, but I can't figure out how to incorporate that.

[tokejepsen]

Actually I might have a solution for considering the instances.

Until the collection has been processed the percentage calculation would be:

(plugins processed) / (total plugins to process)

Once we have processed the collection, we'll have some instances. Continue from the above example the collection produced 4 instances that will be processed by the remaining plugins with the following calculation:

((instances processed / amount of instances) + plugins processed)) / (total plugins to process)

• ((1/4) + 3)/8 = 0.40625
• ((2/4) + 3)/8 = 0.4375
• ((3/4) + 3)/8 = 0.46875
• ((4/4) + 3)/8 = 0.5
• Next plugin
• ((1/4) + 4)/8 = 0.53125
• ((2/4) + 4)/8 = 0.5625
• ((3/4) + 4)/8 = 0.59375
• ((4/4) + 4)/8 = 0.625
• etc...

Will all this fairly complicated code, we are probably best off having the percentage calculation yielded from the publish iterator so the returns of the yields would be (percentage, result, context)

[mottosso]

Will all this fairly complicated code, we are probably best off having the percentage calculation yielded from the publish iterator so the returns of the yields would be (percentage, result, context)

How about we add progress to the result dictionary? It's already got a duration entry, for how long it took a particular plug-in to run, so I think some more of that kind of metadata would fit.

[tokejepsen]

How about we add progress to the result dictionary?

Sure, that sounds good to me. Would it be alright to add that in the publish.util module only?

[mottosso]

I guess for starters, add it like you did, and see if that's enough to build a reasonable GUI. If we add to result we'll need to do that across the board, for GUIs as well.

[tokejepsen]

Added a progress bar to work with the new pyblish.util.publish_iter

[tokejepsen]

What do you think of this now? Works with https://github.com/pyblish/pyblish-base/pull/326

[tokejepsen]

Is this mergable now?

When it is I'll focus on some tests in pyblish-base to cement the essential parts of the functionality.

[mottosso]

We can't merge this, as the feature doesn't exist in base yet.

[tokejepsen]

We can't merge this, as the feature doesn't exist in base yet.

Sorry, didn't mean for it to be merged, just whether it was looking complete to you.

[mottosso]

Other than that nitpick, sure, looks good!