Fix page count

[will-moore]

The unicode figure name issue was eventually addressed by changes to ci (see below). The only remaining changes to this PR are a tiny bug fix.

To test unicode figure export:

• Create a figure named with unicode characters.
• Export as TIFF and/or PDF
• Add a second page (File -> Page setup) and add a panel to the new page.
• Export as TIFF (creates a zip containing a TIFF for each page).

File names should be unchanged.

Previous errors:

AttributeError: 'bytes' object has no attribute 'write'

show full error

``` Traceback (most recent call last): File "./script", line 118, in compress zip_file.write(fullpath, archive_name) File "/opt/rh/rh-python36/root/lib64/python3.6/zipfile.py", line 1622, in write with open(filename, "rb") as src, self.open(zinfo, 'w') as dest: File "/opt/rh/rh-python36/root/lib64/python3.6/zipfile.py", line 1355, in open return self._open_to_write(zinfo, force_zip64=force_zip64) File "/opt/rh/rh-python36/root/lib64/python3.6/zipfile.py", line 1468, in _open_to_write self.fp.write(zinfo.FileHeader(zip64)) File "/opt/rh/rh-python36/root/lib64/python3.6/zipfile.py", line 723, in write n = self.fp.write(data) AttributeError: 'bytes' object has no attribute 'write' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./script", line 2333, in run_script() File "./script", line 2319, in run_script file_annotation = export_figure(conn, script_params) File "./script", line 2275, in export_figure return fig_export.build_figure() File "./script", line 991, in build_figure return self.create_file_annotation(image_ids) File "./script", line 1001, in create_file_annotation compress(zip_name, self.zip_folder_name) File "./script", line 120, in compress zip_file.close() File "/opt/rh/rh-python36/root/lib64/python3.6/zipfile.py", line 1681, in close self._write_end_record() File "/opt/rh/rh-python36/root/lib64/python3.6/zipfile.py", line 1784, in _write_end_record self.fp.write(endrec) File "/opt/rh/rh-python36/root/lib64/python3.6/zipfile.py", line 723, in write n = self.fp.write(data) AttributeError: 'bytes' object has no attribute 'write' ```

Without the line:

target = unicodedata.normalize('NFKD', target).encode('ascii', 'ignore')

I now get:

UnicodeEncodeError: 'ascii' codec can't encode character '\xb5' in position 10: ordinal not in range(128)

show full error

``` Traceback (most recent call last): File "./script", line 2333, in run_script() File "./script", line 2319, in run_script file_annotation = export_figure(conn, script_params) File "./script", line 2275, in export_figure return fig_export.build_figure() File "./script", line 991, in build_figure return self.create_file_annotation(image_ids) File "./script", line 1001, in create_file_annotation compress(zip_name, self.zip_folder_name) File "./script", line 109, in compress print('compress: %s - %s' % (target, base)) UnicodeEncodeError: 'ascii' codec can't encode character '\xb5' in position 10: ordinal not in range(128) ```

[will-moore]

Travis integration test failing with:

>       from numpy import fromfunction, int16
E       ModuleNotFoundError: No module named 'numpy'

Need to require numpy in test dependencies

[joshmoore]

@will-moore : I pushed a commit which moves omero-test-infra further, but it's still failing locally with:

        robj = run_script(client, id, args, "New_Figure")
>       ann = robj.getValue()
E       AttributeError: 'NoneType' object has no attribute 'getValue'

test/integration/test_figure_scripts.py:79: AttributeError

[will-moore]

Running omero-test-infra locally failed with

flake8.main.application   MainProcess   4505 INFO     Found a total of 107 violations and reported 2
./build/lib/omero_figure/views.py:489:23: E999 SyntaxError: invalid syntax
./build/lib/omero_figure/scripts/omero/figure_scripts/Figure_To_Pdf.py:814:45: E999 SyntaxError: invalid syntax

[will-moore]

OK, so rm -rf ./build/ fixed the flake8 errors (old cached files there).

[will-moore]

There are just 2 tests failing on travis:
test/integration/test_figure_files.py .                                  [ 16%]
test/integration/test_figure_scripts.py F.F..                            [100%]

Both of these are exporting as PDF:

• TestFigureScripts.test_export_figure_as[PDF]
• TestFigureScripts.test_export_figure_as[PDF_IMAGES]

The script stdout has:

b'Markdown not installed. See https://pypi.python.org/pypi/Markdown\nReportlab not installed.\nTraceback (most recent call last):\n  File "./script", line 2327, in <module>\n    run_script()\n  File "./script", line 2313, in run_script\n    file_annotation = export_figure(conn, script_params)\n  File "./script", line 2269, in export_figure\n    return fig_export.build_figure()\n  File "./script", line 943, in build_figure\n    self.create_figure()\n  File "./script", line 1790, in create_figure\n    "Need to install https://bitbucket.org/rptlab/reportlab")\nImportError: Need to install https://bitbucket.org/rptlab/reportlab\n'

I see this:

+ yum -y install python-reportlab python-markdown                             app-srv

Does that mean that reportlab is being installed on the OMERO.server docker or on the OMERO.web docker? (needs to be on the server).

[joshmoore]

Green now. Should be included in the next build.

[pwalczysko]

Note - testing suspended, as Figure does not work on py3-ci

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-plugins-push#113. See the console output for more details. Possible conflicts:

• PR #343 will-moore 'Scalebar units'
  • .omeroci/app-srv

--conflicts Conflict resolved in build OMERO-plugins-push#127. See the console output for more details.

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-plugins-push#114. See the console output for more details. Possible conflicts:

• PR #343 will-moore 'Scalebar units'
  • .omeroci/app-srv

--conflicts Conflict resolved in build OMERO-plugins-push#128. See the console output for more details.

[jburel]

Tested with some unicode only i.e. მიკროსკოპის პონი I end up with _.pdf Removing the unicode completely is, I don't think, the way forward

Note: I have installed in virtualenv reportlab and markdown

[will-moore]

There is some difference with Unicode handling in python3 on idr1-slot2 vv locally. For example, if I add this into my script, it works fine locally:

fname = "µTest"
    with open(fname, 'w') as out:
        out.write(fname)
    print(fname, os.path.exists(fname))

but fails when run in a script on that server:

Traceback (most recent call last):
  File "./script", line 2333, in <module>
    run_script()
  File "./script", line 2319, in run_script
    file_annotation = export_figure(conn, script_params)
  File "./script", line 2275, in export_figure
    return fig_export.build_figure()
  File "./script", line 914, in build_figure
    with open(fname, 'w') as out:
UnicodeEncodeError: 'ascii' codec can't encode character '\xb5' in position 0: ordinal not in range(128)

Any ideas?

[will-moore]

Reading https://stackoverflow.com/questions/20022115/python3-and-encoding-different-on-linux-and-on-osx

I tried this locally and on py3 server:

>>> import sys
>>> sys.stdout.encoding
>>> os.environ.get('LANG')

# locally:
UTF-8
en_US.UTF-8

# py3 server
ANSI_X3.4-1968
None

[jburel]

that is probably the source of the problem

[will-moore]

I don't know where to fix that, but hopefully @joshmoore or @manics could help?

[jburel]

Something like update-locale LANG=C.UTF-8 should set what we want

[jburel]

in the container py3ci_omero_1 on idr1-slot3 I got

. omero-virtualenv/bin/activate
 import sys
>>> sys.stdout.encoding
'UTF-8'
import os
>>> os.environ.get('LANG')
'en_US.UTF-8'

[will-moore]

This is required to fix the issue on py3-ci-devspace.openmicroscopy.org https://github.com/ome/omero-py/pull/150 but may also need changes to https://github.com/ome/omero-server-docker/blob/master/Dockerfile

[will-moore]

With that last commit, and the script uploaded to py3-ci server, I can export a figure with unicode in the name (connecting from local omero-web since figure app is not yet working on py3-ci).

[pwalczysko]

Tested on py3-ci, after the hint in https://openmicroscopy.slack.com/archives/CLHEW13NX/p1578329831029100

All works as expected. Tried to export a figure with unicode chars in name. Also, a figure with 2 panels, both export as tiff.

Also, I tried to rename an image with a unicode char in it (micrometer) Then, I added this image to the figure as a panel. Then exported the figure as tiff. This had as a result that the files in the "tiff with images" had still the micrometer in the name of the file I renamed in OMERO. Maybe I do not fully understand the header instructions of this PR ? These suggest that the unicode chars will be reomoved from the names in the exported objects. I did not observe any such removal. Instead, all is there as expected (which I like).

[jburel]

@pwalczysko it was initially removed. That was one of the problems since we could end up with a file with no name and only the extension e.g. _.pdf

[pwalczysko]

good to merge then fmpov

[joshmoore]

WFM, but to clarify, the description and name no longer really match what's in the PR, correct?

[jburel]

correct

[joshmoore]

Might be worth updating to help create changelogs later. Otherwise, :+1: for merging.

[will-moore]

Updated description and name