Fix webclient_figure scripts

[will-moore]

Fixes figure scripts for python 3.

To test:

• Use the figure scripts from webclient with custom UI:
  • Thumbnail Figure
  • Split-View Figure
  • Make Movie
    • And from scripts menu: Movie ROI Figure

This ports some code from image_util to fix it in the script itself.

paint_thumbnail_grid isn't used by any other code and could be deprecated/removed from image_utils.py

[joshmoore]

./omero/figure_scripts/Thumbnail_Figure.py:100:5: E722 do not use bare except'

[will-moore]

Travis now failing with more flake8s from python3 since omero-test-infra is now py3.

./omero/util_scripts/Dataset_To_Plate.py:40:5: F821 undefined name 'long'
./omero/figure_scripts/ROI_Split_Figure.py:56:5: F821 undefined name 'long'
./omero/import_scripts/Populate_Metadata.py:35:5: F821 undefined name 'long'

[will-moore]

Travis is failing with importing fake images in integration tests:

       out, err = popen.communicate()
        rc = popen.wait()
        if rc != 0:
            raise Exception("import failed: [%r] %s\n%s\n%s" % (
>               args, rc, out, err))
E           Exception: import failed: [['/opt/omero/server/venv3/bin/python', './bin/omero', '-s', 'localhost', '-k', 'cbf3400c-b7e4-40a9-b038-c1d11335ecbe', '-p', '4064', 'import', '--skip', 'all', '--output', 'legacy', '--', path('/opt/omero/server/omero/tmp/omero_omero-server/969/import_fake_file_08w9va1yv&&plates=1&plateAcqs=1&plateCols=1&plateRows=1&fields=2.fake')]] 102
E           b''
E           b"Cannot get JAR files from '/opt/omero/server/venv3/lib/lib/client' (No such file or directory)\n"

[joshmoore]

This is likely the same issue @sbesson saw with OMERODIR in https://github.com/ome/omero-test-infra/pull/44#issuecomment-553877422

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#71. See the console output for more details. Possible conflicts:

• PR #152 will-moore 'populate metadata links file ann'
  • omero/import_scripts/Populate_Metadata.py

--conflicts

[joshmoore]

travis now working again. Failed with:

E       Traceback (most recent call last):
E         File "./script", line 672, in <module>
E           run_script()
E         File "./script", line 555, in run_script
E           ckeys.sort()
E       AttributeError: 'dict_keys' object has no attribute 'sort'

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-python-superbuild-push#75. See the console output for more details. Possible conflicts:

• PR #152 will-moore 'populate metadata links file ann'
  • omero/import_scripts/Populate_Metadata.py

--conflicts Conflict resolved in build OMERO-python-superbuild-push#76. See the console output for more details.

[pwalczysko]

Make Movie ... from the figure scripts still errors on py3-ci

...
Traceback (most recent call last): File "./script", line 74, in from cStringIO import StringIO ModuleNotFoundError: No module named 'cStringIO' --------------------------------- } 

[will-moore]

Those last commits should fix Batch_ROI_Export script and Make_Movie script, except that mencoder is not installed on py3-ci or merge-ci.

[sbesson]

Given that at least two open scripts PRs are currently blocked on the fixes included in this PR, if no huge blocker is found during testing, I am in favor of getting this merged. Issues reported during the functional testing can be captured as issues and prioritized before general availability release if needed.

[pwalczysko]

Make Movie - I guess there is some progress here, as the error has changed (tested on py3-ci). Nevertheless, still unable to test the whole workflow.

sh: mencoder: command not found

[joshmoore]

mencoder installed via:

[jamoore@idr1-slot2 py3-ci]$ git diff server/Dockerfile
diff --git a/server/Dockerfile b/server/Dockerfile
index b579538..461728e 100644
--- a/server/Dockerfile
+++ b/server/Dockerfile
@@ -66,6 +66,11 @@ RUN /py3/bin/pip install scc
 ENV VIRTUAL_ENV=/py3
 ENV PATH="$VIRTUAL_ENV/bin:$PATH"

+# Install mencoder
+# See https://github.com/ome/prod-playbooks/blob/7d8e59ced0b47a224d6be02907d913c392ef9379/ome-dundeeomero.yml#L48
+RUN yum install -y http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm
+RUN yum install -y mencoder

[pwalczysko]

On py3-ci, figure_scripts > Movie ROI Figure crashes immediately when selected the menu item (the first dialog does not materialize)

Not sure this is the right place to report, but see below @joshmoore @will-moore - I have a memory of that particualr script already working on py3-ci ...

File "/home/omero/workspace/OMERO-web/OMERO.web/lib/python/omero_api_IScript_ice.py", line 842, in getParams
    return _M_omero.api.IScript._op_getParams.invoke(self, ((scriptID, ), _ctx))

omero.ValidationException: exception ::omero::ValidationException
{
    serverStackTrace = 
    serverExceptionClass = 
    message = Can't find params for 5504!
No stdout.
stderr is in file 7719:
---------------------------------
Traceback (most recent call last):
  File "/home/omero/workspace/OMERO-server/OMERO.server/lib/python/omero/util/image_utils.py", line 30, in <module>
    from PIL import Image, ImageDraw, ImageFont  # see ticket:2597
ModuleNotFoundError: No module named 'PIL'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./script", line 37, in <module>
    import omero.util.image_utils as image_utils
  File "/home/omero/workspace/OMERO-server/OMERO.server/lib/python/omero/util/image_utils.py", line 32, in <module>
    import Image
ModuleNotFoundError: No module named 'Image'

---------------------------------

[pwalczysko]

Make Movie crashes too, with the empty "Send Feedback" dialog.

Definitely, we are in much worse state on py3-ci now in comparison with yesterday

[pwalczysko]

Retest after job rerun:

Re https://github.com/ome/scripts/pull/159#issuecomment-559042079 - Movie ROI Figure now fully passes.

Re: https://github.com/ome/scripts/pull/159#issuecomment-559043402 - the Make Movie as well as the other two "figure" scripts work now without any errors.

[pwalczysko]

The only problem remaining here is that Make Movie (the one started from RHP) is producing black movies only (2 very different images tested). some output is produced such as

[mjpeg @ 0x7ff5fe9665a0]AVFrame.format is not set
[mjpeg @ 0x7ff5fe9665a0]AVFrame.width or height is not set
...

for every plane of the image in question.

[will-moore]

That last commit should fix the black Make_Movie frames.

[pwalczysko]

Yes, the Make Movie is fixed now in py3-ci .

user-2 on https://py3-ci.openmicroscopy.org/web/webclient/?show=well-2102 is working without a problem.

user-3 on https://py3-ci.openmicroscopy.org/web/webclient/?show=image-2970 produces a movie which has a strange blinking effects - they cannot be seen if the image is played inside OMERO in iviewer - I would suspect this is because the frames coming into the avi movie are not in correct order, which might be connected to the fact that this is a multi-z (and multi-t of course) image.

[joshmoore]

@pwalczysko : i.e. good to merge?

[pwalczysko]

Good to merge

[sbesson]

Thanks @pwalczysko. Merging to fix the Travis on #160 and #163