search

ercius / openNCEM

A collection of packages and tools for electron microscopy data analysis supported by the National Center for Electron Microscopy facility of the Molecular Foundry
GNU General Public License v3.0
60 stars 28 forks source link

Could not build ncempy. Cannot load backend 'Qt5Agg' which requires the 'qt5' #22

Closed kevinsmia1939 closed 4 years ago

kevinsmia1939 commented 4 years ago

Hello,

I could not build ncempy on openSUSE Tumbleweed. I tried installing python3-qt5 and python3-matplotlib-qt5 but the error still occur.

The error are shown below. Thanks.

[   54s] ======================================================================
[   54s] ERROR: DoGenfire (unittest.loader._FailedTest)
[   54s] ----------------------------------------------------------------------
[   54s] ImportError: Failed to import test module: DoGenfire
[   54s] Traceback (most recent call last):
[   54s]   File "/usr/lib64/python3.8/unittest/loader.py", line 154, in loadTestsFromName
[   54s]     module = __import__(module_name)
[   54s]   File "/home/abuild/rpmbuild/BUILD/openNCEM-1.5.0.1583714936.999f7a3/ncempy/edstomo/DoGenfire.py", line 1, in <module>
[   54s]     import genfire as gf
[   54s]   File "/usr/lib/python3.8/site-packages/genfire/__init__.py", line 2, in <module>
[   54s]     from . import reconstruct
[   54s]   File "/usr/lib/python3.8/site-packages/genfire/reconstruct.py", line 16, in <module>
[   54s]     matplotlib.use("Qt5Agg")
[   54s]   File "/usr/lib64/python3.8/site-packages/matplotlib/cbook/deprecation.py", line 307, in wrapper
[   54s]     return func(*args, **kwargs)
[   54s]   File "/usr/lib64/python3.8/site-packages/matplotlib/__init__.py", line 1307, in use
[   54s]     switch_backend(name)
[   54s]   File "/usr/lib64/python3.8/site-packages/matplotlib/pyplot.py", line 233, in switch_backend
[   54s]     raise ImportError(
[   54s] ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt5' interactive framework, as 'headless' is currently running
[   54s] 
[   54s] 
[   54s] ======================================================================
[   54s] ERROR: postprocess (unittest.loader._FailedTest)
[   54s] ----------------------------------------------------------------------
[   54s] ImportError: Failed to import test module: postprocess
[   54s] Traceback (most recent call last):
[   54s]   File "/usr/lib64/python3.8/unittest/loader.py", line 154, in loadTestsFromName
[   54s]     module = __import__(module_name)
[   54s]   File "/home/abuild/rpmbuild/BUILD/openNCEM-1.5.0.1583714936.999f7a3/ncempy/edstomo/postprocess.py", line 4, in <module>
[   54s]     import genfire
[   54s]   File "/usr/lib/python3.8/site-packages/genfire/__init__.py", line 2, in <module>
[   54s]     from . import reconstruct
[   54s]   File "/usr/lib/python3.8/site-packages/genfire/reconstruct.py", line 16, in <module>
[   54s]     matplotlib.use("Qt5Agg")
[   54s]   File "/usr/lib64/python3.8/site-packages/matplotlib/cbook/deprecation.py", line 307, in wrapper
[   54s]     return func(*args, **kwargs)
[   54s]   File "/usr/lib64/python3.8/site-packages/matplotlib/__init__.py", line 1307, in use
[   54s]     switch_backend(name)
[   54s]   File "/usr/lib64/python3.8/site-packages/matplotlib/pyplot.py", line 233, in switch_backend
[   54s]     raise ImportError(
[   54s] ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt5' interactive framework, as 'headless' is currently running

The full error log is here: https://build.opensuse.org/package/live_build_log/home:andythe_great/python3-ncempy/home_andythe_great_openSUSE_TW/x86_64

ZGainsforth commented 4 years ago

Hi Kevin,

You should be able to pip install “ncempy[edstomo]” and get all the components. If you want to build it from scratch then it is looking like there is an issue in the genfire dependency. I can try making a virtualbox with opensuse and see if I can reproduce it.

Zack

On Apr 13, 2020, at 6:36 PM, Kevin Tee notifications@github.com wrote:

Hello,

I could not build ncempy on openSUSE Tumbleweed. I tried installing python3-qt5 and python3-matplotlib-qt5 but the error still occur.

The error are shown below. Thanks.

[ 54s] ====================================================================== [ 54s] ERROR: DoGenfire (unittest.loader._FailedTest) [ 54s] ---------------------------------------------------------------------- [ 54s] ImportError: Failed to import test module: DoGenfire [ 54s] Traceback (most recent call last): [ 54s] File "/usr/lib64/python3.8/unittest/loader.py", line 154, in loadTestsFromName [ 54s] module = import(module_name) [ 54s] File "/home/abuild/rpmbuild/BUILD/openNCEM-1.5.0.1583714936.999f7a3/ncempy/edstomo/DoGenfire.py", line 1, in [ 54s] import genfire as gf [ 54s] File "/usr/lib/python3.8/site-packages/genfire/init.py", line 2, in [ 54s] from . import reconstruct [ 54s] File "/usr/lib/python3.8/site-packages/genfire/reconstruct.py", line 16, in [ 54s] matplotlib.use("Qt5Agg") [ 54s] File "/usr/lib64/python3.8/site-packages/matplotlib/cbook/deprecation.py", line 307, in wrapper [ 54s] return func(*args, *kwargs) [ 54s] File "/usr/lib64/python3.8/site-packages/matplotlib/init.py", line 1307, in use [ 54s] switch_backend(name) [ 54s] File "/usr/lib64/python3.8/site-packages/matplotlib/pyplot.py", line 233, in switch_backend [ 54s] raise ImportError( [ 54s] ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt5' interactive framework, as 'headless' is currently running [ 54s] [ 54s] [ 54s] ====================================================================== [ 54s] ERROR: postprocess (unittest.loader._FailedTest) [ 54s] ---------------------------------------------------------------------- [ 54s] ImportError: Failed to import test module: postprocess [ 54s] Traceback (most recent call last): [ 54s] File "/usr/lib64/python3.8/unittest/loader.py", line 154, in loadTestsFromName [ 54s] module = import(module_name) [ 54s] File "/home/abuild/rpmbuild/BUILD/openNCEM-1.5.0.1583714936.999f7a3/ncempy/edstomo/postprocess.py", line 4, in [ 54s] import genfire [ 54s] File "/usr/lib/python3.8/site-packages/genfire/init.py", line 2, in [ 54s] from . import reconstruct [ 54s] File "/usr/lib/python3.8/site-packages/genfire/reconstruct.py", line 16, in [ 54s] matplotlib.use("Qt5Agg") [ 54s] File "/usr/lib64/python3.8/site-packages/matplotlib/cbook/deprecation.py", line 307, in wrapper [ 54s] return func(args, **kwargs) [ 54s] File "/usr/lib64/python3.8/site-packages/matplotlib/init.py", line 1307, in use [ 54s] switch_backend(name) [ 54s] File "/usr/lib64/python3.8/site-packages/matplotlib/pyplot.py", line 233, in switch_backend [ 54s] raise ImportError( [ 54s] ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt5' interactive framework, as 'headless' is currently running The full error log is here: https://build.opensuse.org/package/live_build_log/home:andythe_great/python3-ncempy/home_andythe_great_openSUSE_TW/x86_64 https://build.opensuse.org/package/live_build_log/home:andythe_great/python3-ncempy/home_andythe_great_openSUSE_TW/x86_64 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ercius/openNCEM/issues/22, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABUL526WZQCNKPKGG4TFUJ3RMO4YHANCNFSM4MHLRM4Q.

kevinsmia1939 commented 4 years ago

Ah, it seems I have some missing python module while trying to build genfire. Funnily enough I have the same error while trying to build genfire and this error. I will ask the genfire dev.

genfire and ncempy are not available in openSUSE repository, I'm trying to submit it to opensuse at the moment.

https://build.opensuse.org/package/show/home:andythe_great/python3-genfire

kevinsmia1939 commented 4 years ago

Some update. There seems to be some issue with building genfire and Qt5. The genfire is not maintain anymore. https://github.com/genfire-em/GENFIRE-Python/issues/2

ercius commented 4 years ago

@kevinsmia1939 Thanks for reporting this both here and in the genfire repo genfire-em/GENFIRE-python#2. I am not having a problem running on my local machine (windows 10) as

import matplotlib
matplotlib.use("Qt5Agg")

Are you trying to run this on a 'headless' machine (i.e. no graphical output) like a docker image? If so, then the GUI will not be able to be shown. You can see for example this answer on StackOverflow.

Maybe you can try commenting out this line in genfire and trying again? The GUI might not run, but then it looks like you are interested in ncempy rather than genfire?

ercius commented 4 years ago

@kevinsmia1939 Also, can you send your output to the following:

import matplotlib
matplotlib.use()

This will show all of the backends your installation can use.

kevinsmia1939 commented 4 years ago

@kevinsmia1939 Also, can you send your output to the following:

import matplotlib
matplotlib.use()

This will show all of the backends your installation can use.

Hello,

I am indeed building genfire on headless server. The headless server is openSUSE's build service, it is use to create package for openSUSE and SUSE Enterprise Linux.

I upload the .spec file in the link below, the spec file contain instructions to build the package from source. https://build.opensuse.org/package/show/home:andythe_great/python-genfire

I have remove the matplotlib.use("Qt5Agg") in VolumeSlicer.py launch.py recontruct.py

And recompressed the file back into GENFIRE-python.tar.gz

However, a new error were found: https://build.opensuse.org/package/live_build_log/home:andythe_great/python-genfire/openSUSE_Tumbleweed/x86_64

kevinsmia1939 commented 4 years ago 
import matplotlib

I will try to figure out how to do that.

ercius commented 4 years ago

That new error looks like a problem with a qt resource file (a graphic icon?). I just loaded the GUI on a pyqt5 install with no problems. So, its likely a problem with your headless OS. It also looks like this is happening because of a test being run. I have a three more suggestions:

  1. Comment out this line which will avoid loading the GUI. You cant use it anyway, so you may as well skip it.
  2. Disable tests when you run this package. Try the --no-test option or something similar.
  3. If you cant disable tests then make up a 'dummy' test in genfire's setup.py. See here. 
    class test(_test):
def run(self):
    print('skipping tests for genfire')
ercius commented 4 years ago

Actually, I just looked at the .spec file you linked to. On line 78-79 is written:

%check
%python_exec setup.py test

I know nothing about this build prcess, but I bet if you remove these lines tests will not be run. Then it will probably install.

kevinsmia1939 commented 4 years ago

Actually, I just looked at the .spec file you linked to. On line 78-79 is written:

%check
%python_exec setup.py test

I know nothing about this build prcess, but I bet if you remove these lines tests will not be run. Then it will probably install.

It will install, but I would like to submit this to official opensuse repository, and they require checking unless I have a very good reason to not do it.

I will ask other openSUSE folks about this, thanks for the suggestion.

ercius commented 4 years ago

Maybe the python packaging guidelines for openSUSE are relevant? It does not have to pass tests if none are provided which GENFIRE does not provide.

kevinsmia1939 commented 4 years ago

Maybe the python packaging guidelines for openSUSE are relevant? It does not have to pass tests if none are provided which GENFIRE does not provide.

You are right, there is no test on genfire, so I can skip it. Do you perhaps know what test_aligned.npy is? Looks like something from numpy.

I try to build ncempy here and got the error: error: [Errno 2] No such file or directory: 'test_aligned.npy'

https://build.opensuse.org/package/live_build_log/home:andythe_great/python-ncempy/openSUSE_Tumbleweed/x86_64

Thanks

ercius commented 4 years ago

That sounds like a data file for a test for one of the packages.Right now I dont see any reference to test_aligned.npy in your log file, in genfire or in ncempy when I opened it today: https://build.opensuse.org/package/live_build_log/home:andythe_great/python-ncempy/openSUSE_Tumbleweed/x86_64 Can you be more specific of the error? Is it still a problem?

Also, the build information shows that you are pulling ncempy/master from github. Ive been pushing a lot of new code there recently (e.g. stack_align.py) and it has not been fully tested and released yet. I should use a development branch, but Ive been lax in that. It could be that test_aligned.npy was part of the code and I removed that code recently since its not there now.

Looks like your current build is failing due to pyqt again.

kevinsmia1939 commented 4 years ago

That sounds like a data file for a test for one of the packages.Right now I dont see any reference to test_aligned.npy in your log file, in genfire or in ncempy when I opened it today: https://build.opensuse.org/package/live_build_log/home:andythe_great/python-ncempy/openSUSE_Tumbleweed/x86_64 Can you be more specific of the error? Is it still a problem?

Also, the build information shows that you are pulling ncempy/master from github. Ive been pushing a lot of new code there recently (e.g. stack_align.py) and it has not been fully tested and released yet. I should use a development branch, but Ive been lax in that. It could be that test_aligned.npy was part of the code and I removed that code recently since its not there now.

Looks like your current build is failing due to pyqt again.

Hello,

I try master branch I pull from github just to simply avoid h5py_cache dependency.

The "test_aligned.npy" issue I have is gone, maybe a temporary issue during the package building process.

The issue currently is during the test process, from the look, when testing ncempy, it also test genfire isn't it? But since the genfire itself don't have test available, can genfire test be skip or remove?

Thanks

https://build.opensuse.org/package/live_build_log/home:andythe_great/python-ncempy/openSUSE_Tumbleweed/x86_64

[  114s] ======================================================================
[  114s] ERROR: DoGenfire (unittest.loader._FailedTest)
[  114s] ----------------------------------------------------------------------
[  114s] ImportError: Failed to import test module: DoGenfire
[  114s] Traceback (most recent call last):
[  114s]   File "/usr/lib64/python3.8/unittest/loader.py", line 154, in loadTestsFromName
[  114s]     module = __import__(module_name)
[  114s]   File "/home/abuild/rpmbuild/BUILD/openNCEM-master/ncempy/edstomo/DoGenfire.py", line 1, in <module>
[  114s]     import genfire as gf
[  114s]   File "/usr/lib/python3.8/site-packages/genfire/__init__.py", line 2, in <module>
[  114s]     from . import reconstruct
[  114s]   File "/usr/lib/python3.8/site-packages/genfire/reconstruct.py", line 16, in <module>
[  114s]     matplotlib.use("Qt5Agg")
[  114s]   File "/usr/lib64/python3.8/site-packages/matplotlib/cbook/deprecation.py", line 296, in wrapper
[  114s]     return func(*args, **kwargs)
[  114s]   File "/usr/lib64/python3.8/site-packages/matplotlib/cbook/deprecation.py", line 358, in wrapper
[  114s]     return func(*args, **kwargs)
[  114s]   File "/usr/lib64/python3.8/site-packages/matplotlib/__init__.py", line 1281, in use
[  114s]     plt.switch_backend(name)
[  114s]   File "/usr/lib64/python3.8/site-packages/matplotlib/pyplot.py", line 234, in switch_backend
[  114s]     raise ImportError(
[  114s] ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt5' interactive framework, as 'headless' is currently running
ercius commented 4 years ago

@kevinsmia1939 That error looks like it is trying to test DoGenfire which is part of the optional [edstomo] package of ncempy. It import genfire. Im not sure why unittest is trying to load and test DoGenfire.py.

It looks like your more recent build log is missing python2 and/or the ncempy zip file. Not sure if you are trying things or running into a different problem now.

kevinsmia1939 commented 4 years ago

@kevinsmia1939 That error looks like it is trying to test DoGenfire which is part of the optional [edstomo] package of ncempy. It import genfire. Im not sure why unittest is trying to load and test DoGenfire.py.

It looks like your more recent build log is missing python2 and/or the ncempy zip file. Not sure if you are trying things or running into a different problem now.

Hello, I just try changing the openncem master branch, I seems to misspelled some stuff which cause error. The same error now in the log. Python 2 is depreciate on openSUSE Tumbleweed now, it should not be anything to do with it.

So to stop the error, I should stop unittest to test genfire some how? I will ask some other package if they can do magic. Thanks.

ercius commented 4 years ago

Yes. You should stop the unittests for DoGenfire. You can see that its running some tests in these lines:

  • /usr/bin/python3 setup.py test [ 94s] running test [ 94s] WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox. [ 94s] running egg_info

Im not sure how to stop this from happening. DoGenfire.py is part of the [edstomo] optional ncempy package. It is not necessary for the base package. I do not see anywhere in ncempy tests where this would be running.

ercius commented 4 years ago

It looks like line 95 and 96 in your spec file is running the tests:

%check %{_bindir}/python3 setup.py test

If you skip this then that should avoid the error you are seeing.

kevinsmia1939 commented 4 years ago

It looks like line 95 and 96 in your spec file is running the tests:

%check %{_bindir}/python3 setup.py test

If you skip this then that should avoid the error you are seeing.

Hello, For openSUSE, the best practice is to test the package if it provide one, there have to be a good reason to disable the test. But I will try to submit without the test to the python repo maintainer and see what they have to say.

ercius commented 4 years ago

We just merged a commit which avoids importing the GUI portion of genfire. Try compiling with the latest master. It should fix the issue you are running into.

kevinsmia1939 commented 4 years ago

We just merged a commit which avoids importing the GUI portion of genfire. Try compiling with the latest master. It should fix the issue you are running into.

Hello,

I download the latest build just now and build it, but error still persist, but it is a bit different.

Full log: https://build.opensuse.org/package/live_build_log/home:andythe_great/python-ncempy-git/openSUSE_Tumbleweed/x86_64


[  112s] ======================================================================
[  112s] ERROR: DoGenfire (unittest.loader._FailedTest)
[  112s] ----------------------------------------------------------------------
[  112s] ImportError: Failed to import test module: DoGenfire
[  112s] Traceback (most recent call last):
[  112s]   File "/usr/lib64/python3.8/unittest/loader.py", line 154, in loadTestsFromName
[  112s]     module = __import__(module_name)
[  112s]   File "/home/abuild/rpmbuild/BUILD/openNCEM-master/ncempy/edstomo/DoGenfire.py", line 2, in <module>
[  112s]     from genfire import reconstruct, fileio
[  112s]   File "/usr/lib/python3.8/site-packages/genfire/__init__.py", line 2, in <module>
[  112s]     from . import reconstruct
[  112s]   File "/usr/lib/python3.8/site-packages/genfire/reconstruct.py", line 16, in <module>
[  112s]     matplotlib.use("Qt5Agg")
[  112s]   File "/usr/lib64/python3.8/site-packages/matplotlib/cbook/deprecation.py", line 296, in wrapper
[  112s]     return func(*args, **kwargs)
[  112s]   File "/usr/lib64/python3.8/site-packages/matplotlib/cbook/deprecation.py", line 358, in wrapper
[  112s]     return func(*args, **kwargs)
[  112s]   File "/usr/lib64/python3.8/site-packages/matplotlib/__init__.py", line 1281, in use
[  112s]     plt.switch_backend(name)
[  112s]   File "/usr/lib64/python3.8/site-packages/matplotlib/pyplot.py", line 234, in switch_backend
[  112s]     raise ImportError(
[  112s] ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt5' interactive framework, as 'headless' is currently running
[  112s] 
[  112s] 
[  112s] ----------------------------------------------------------------------
ercius commented 4 years ago

So, genfire still imports QT5 in the reconstruct module. It has a feature that allows the user to plot as the reconstruction continues.

This means either we need to update genfire to fail gracefully when it cant import pyqt5 or you need to figure out how to get around the 'headless' issue. You could potentially look at this page about setting some environment variables.

Lastly, using python setup.py test is deprecated. So, that is not even the bet way to test the package. ncempy has its own tests which should be used to test it instead of this deprecated way. You should use nosetests to run the ncempy tests: python setup.py nosetests However, this also requires that you download the test data sets.

Try submitting without the tests and point the maintainers to this conversation. Lets see what they say.

bnavigator commented 4 years ago

You can use Xvfb if you have to test GUI elements in OBS.

I sent you an SR for the package using the pytest-xvfb plugin. Another option is to use xvfb-run with nosetests, but ncempy does not seem to rely on nose vs pytest. setup.py test is deprecated by setuptools.

With that out of the way you will have to fix the next problem:

kevinsmia1939 commented 4 years ago

Xvfb

Thanks for the SR, I see now that several GB of data is needed to test properly. I will try to submit this to d:l:python, but I am building a git version because the current version depend on h5py_cache which is removed, I don't think they want the git version.

bnavigator commented 4 years ago

Try submitting without the tests and point the maintainers to this conversation. Lets see what they say.

Maybe you can add a conditional build for the tests:

bnavigator commented 4 years ago

but I am building a git version because the current version depend on h5py_cache which is removed, I don't think they want the git version.

You could ask here for a new release without the h5py_cache dependency or add a patch to the 1.5.0 version which include the changes that get rid of it. A quick look says it's functionality has merged into h5py, so the changes would not be that big. Look for the PRs/corresponding commits in this repo and add them as patch to your package.

Edit: #21 seems to be the relevant PR. Just download https://github.com/ercius/openNCEM/pull/21.patch and try to add it as a patch to your 1.5.0 package. If you are lucky, no rebasing is required.

ercius commented 4 years ago

ncempy does not seem to rely on nose vs pytest

  • test_io looks for a windows filepath. Obviously not suitable for a Linux package in OBS
  • The README in the test dir says you need a big chunk of real data. I don't think you can upload that to OBS.

I will admit that the tests in ncempy are a complete mess. They have been written by a few different people over the years. And, as usual, there are more pressing things to do than to write/clean-up the tests directory. Im planning to clean this up when I get a chance and to add small test files directly to the repo. The large files are not really necessary.

the current version depend on h5py_cache...

You could ask here for a new release without the h5py_cache...

I was about to make a new release anyway. Its long overdue. Ill try to get that done very soon.

ercius commented 4 years ago

I overhauled the tests and added 32 MB of test data to the repo. They all passed on my local machine. If you run only the tests in the ncempy/test directory using pytest then you should be good to go.

Ill work on the release next. I wanted to add one more feature, but there is already a lot added since the 1.5.0 release.

kevinsmia1939 commented 4 years ago

I overhauled the tests and added 32 MB of test data to the repo. They all passed on my local machine. If you run only the tests in the ncempy/test directory using pytest then you should be good to go.

Ill work on the release next. I wanted to add one more feature, but there is already a lot added since the 1.5.0 release.

Hello, I just download the git version. I have to tell pytest to ignore ncempy/test/test_stack_align.py because it is looking for some Windows' directory which is not on linux.

[   44s] =================================== FAILURES ===================================
[   44s] _______________________________ test_stack_align _______________________________
[   44s] 
[   44s] name = b'C:/Users/linol/Data/Acquisition_18.emd', mode = 'a'
[   44s] userblock_size = None, fapl = <h5py.h5p.PropFAID object at 0x7f87c1443e00>
[   44s] fcpl = None, swmr = False
[   44s] 
[   44s]     def make_fid(name, mode, userblock_size, fapl, fcpl=None, swmr=False):
[   44s]         """ Get a new FileID by opening or creating a file.
[   44s]         Also validates mode argument."""
[   44s]     
[   44s]         if userblock_size is not None:
[   44s]             if mode in ('r', 'r+'):
[   44s]                 raise ValueError("User block may only be specified "
[   44s]                                  "when creating a file")
[   44s]             try:
[   44s]                 userblock_size = int(userblock_size)
[   44s]             except (TypeError, ValueError):
[   44s]                 raise ValueError("User block size must be an integer")
[   44s]             if fcpl is None:
[   44s]                 fcpl = h5p.create(h5p.FILE_CREATE)
[   44s]             fcpl.set_userblock(userblock_size)
[   44s]     
[   44s]         if mode == 'r':
[   44s]             flags = h5f.ACC_RDONLY
[   44s]             if swmr and swmr_support:
[   44s]                 flags |= h5f.ACC_SWMR_READ
[   44s]             fid = h5f.open(name, flags, fapl=fapl)
[   44s]         elif mode == 'r+':
[   44s]             fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
[   44s]         elif mode in ['w-', 'x']:
[   44s]             fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
[   44s]         elif mode == 'w':
[   44s]             fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
[   44s]         elif mode == 'a':
[   44s]             # Open in append mode (read/write).
[   44s]             # If that fails, create a new file only if it won't clobber an
[   44s]             # existing one (ACC_EXCL)
[   44s]             try:
[   44s] >               fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
[   44s] 
[   44s] /usr/lib64/python3.8/site-packages/h5py/_hl/files.py:185: 
[   44s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   44s] 
[   44s] >   ???
[   44s] 
[   44s] h5py/_objects.pyx:54: 
[   44s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   44s] 
[   44s] >   ???
[   44s] 
[   44s] h5py/_objects.pyx:55: 
[   44s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   44s] 
[   44s] >   ???
[   44s] E   OSError: Unable to open file (unable to open file: name = 'C:/Users/linol/Data/Acquisition_18.emd', errno = 2, error message = 'No such file or directory', flags = 1, o_flags = 2)
[   44s] 
[   44s] h5py/h5f.pyx:88: OSError
[   44s] 
[   44s] During handling of the above exception, another exception occurred:
[   44s] 
[   44s] emd_path = 'C:/Users/linol/Data/Acquisition_18.emd'
[   44s] 
[   44s]     def test_stack_align(emd_path):
[   44s]         up = 2
[   44s]         method0 = 'hybrid'
[   44s]     
[   44s] >       with nio.emd.fileEMD('C:/Users/linol/Data/Acquisition_18.emd') as f0:
[   44s] 
[   44s] ncempy/test/test_stack_align.py:19: 
[   44s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   44s] ncempy/io/emd.py:89: in __init__
[   44s]     self.file_hdl = h5py.File(filename, 'a')
[   44s] /usr/lib64/python3.8/site-packages/h5py/_hl/files.py:406: in __init__
[   44s]     fid = make_fid(name, mode, userblock_size,
[   44s] /usr/lib64/python3.8/site-packages/h5py/_hl/files.py:187: in make_fid
[   44s]     fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
[   44s] h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
[   44s]     ???
[   44s] h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
[   44s]     ???
[   44s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   44s] 
[   44s] >   ???
[   44s] E   OSError: Unable to create file (unable to open file: name = 'C:/Users/linol/Data/Acquisition_18.emd', errno = 2, error message = 'No such file or di
[   44s] 
[   44s] h5py/h5f.pyx:108: OSError
[   44s] ----------------------------- Captured stdout call -----------------------------
[   44s] Error opening file for read/write: "C:/Users/linol/Data/Acquisition_18.emd"
[   44s] =============================== warnings summary ===============================

Another one is ncempy/test/test_io_dm.py because segmentation fault.


[    6s] ============================= test session starts ==============================
[    6s] platform linux -- Python 3.8.3, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python3
[    7s] cachedir: .pytest_cache
[    7s] Matplotlib: 3.2.1
[    7s] Freetype: 2.10.2
[    7s] rootdir: /home/abuild/rpmbuild/BUILD/openNCEM-master
[    7s] plugins: mpl-0.11, xvfb-1.2.0
[    7s] collecting ... collected 30 items
[    7s] 
[   11s] ncempy/test/test_algo_distortion.py::Testringdiff::test_distortion PASSED [  3%]
[   19s] ncempy/test/test_algo_local_max.py::TestLocalmax::test_local_max PASSED  [  6%]
[   19s] ncempy/test/test_algo_multicorr.py::TestShiftedimages::test_multicorr PASSED [ 10%]
[   19s] ncempy/test/test_algo_multicorr.py::TestShiftedimages::test_different_shifts PASSED [ 13%]
[   38s] ncempy/test/test_algo_radial_profile.py::TestRingdiff::test_radialprofile PASSED [ 16%]
[   38s] ncempy/test/test_eval_ring_diff.py::TestRingdiff::test_eval_minimum PASSED [ 20%]
[   38s] ncempy/test/test_eval_ring_diff.py::TestRingdiff::test_eval_full PASSED  [ 23%]
[   44s] ncempy/test/test_eval_ring_diff.py::TestRingdiff::test_eval_single PASSED [ 26%]
[   44s] ncempy/test/test_io.py::test_emd_berkeley PASSED                         [ 30%]
[   44s] ncempy/test/test_io.py::test_dm3_2d PASSED                               [ 33%]
[   44s] ncempy/test/test_io.py::test_dm4_2d PASSED                               [ 36%]
[   44s] ncempy/test/test_io.py::test_dm3_3d PASSED                               [ 40%]
[   44s] ncempy/test/test_io.py::test_dm4_3d PASSED                               [ 43%]
[   44s] ncempy/test/test_io.py::test_dm3_spectrum_1d PASSED                      [ 46%]
[   44s] ncempy/test/test_io.py::test_mrc PASSED                                  [ 50%]
[   44s] ncempy/test/test_io_dm.py::Testdm3::test_read_dm3 PASSED                 [ 53%]
[   44s] ncempy/test/test_io_dm.py::Testdm3::test_read_dm3_1d FAILED              [ 56%]
[   45s] ncempy/test/test_io_dm.py::Testdm3::test_read_dm3_on_memory PASSED       [ 60%]
[   46s] ncempy/test/test_io_dm.py::Testdm3::test_dm4_memory_vs_file_performance PASSED [ 63%]
[   46s] ncempy/test/test_io_dm.py::Testdm3::test_extract_on_memory Fatal Python error: Segmentation fault
[   46s] 
[   46s] Current thread 0x00007fd0b6a9b740 (most recent call first):
[   46s]   File "/home/abuild/rpmbuild/BUILD/openNCEM-master/ncempy/test/test_io_dm.py", line 104 in test_extract_on_memory
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/python.py", line 1445 in runtest
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 210 in <lambda>
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 237 in from_call
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 209 in call_runtest_hook
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 185 in call_and_report
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 99 in runtestprotocol
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 247 in _main
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 197 in wrap_session
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
[   46s]   File "/usr/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
[   46s]   File "/usr/lib/python3.8/site-packages/_pytest/config/__init__.py", line 92 in main
[   46s]   File "/usr/bin/py.test-3.8", line 11 in <module>
[   46s] /var/tmp/rpm-tmp.hVE1fg: line 39: 21301 Segmentation fault      (core dumped) PYTHONPATH=$PYTHONPATH:/home/abuild/rpmbuild/BUILDROOT/python-ncempy-master-0.x86_64/usr/lib/python3.8/site-packages PYTHONDONTWRITEBYTECODE=1 py.test-3.8 --ignore=_build.python2 --ignore=_build.python3 --ignore=_build.pypy3 -v --ignore ncempy/test/test_stack_align.py

The full log for the segmentation fault is here: https://build.opensuse.org/package/live_build_log/home:andythe_great/python-ncempy-git/openSUSE_Tumbleweed/x86_64

ercius commented 4 years ago

I have to tell pytest to ignore ncempy/test/test_stack_align.py because it is looking for some Windows' directory which is not on linux.

I missed updating that test. Its fixed now.

Another one is ncempy/test/test_io_dm.py because segmentation fault.

I found this issue yesterday because...tests really work! I had the fix in another branch and I just merged it (and another fix) in. See https://github.com/ercius/openNCEM/commit/f6df731a415e9dad0a501f75936169c89be83047 and https://github.com/ercius/openNCEM/commit/f30f10e5f59d11013276f473b78788882fa83739.

Hopefully, that will resolve all issues and the package will finally build.

kevinsmia1939 commented 4 years ago

I have to tell pytest to ignore ncempy/test/test_stack_align.py because it is looking for some Windows' directory which is not on linux.

I missed updating that test. Its fixed now.

Another one is ncempy/test/test_io_dm.py because segmentation fault.

I found this issue yesterday because...tests really work! I had the fix in another branch and I just merged it (and another fix) in. See f6df731 and f30f10e.

Hopefully, that will resolve all issues and the package will finally build.

Just downloaded the latest source. The error is down to 1 failure now. It is looking for 08_Carbon.dm3 but I see only 08_carbon.dm3 maybe typo?

[  115s] =================================== FAILURES ===================================
[  115s] ___________________________ Testdm3.test_read_dm3_1d ___________________________
[  115s] 
[  115s] self = <test_io_dm.Testdm3 object at 0x7f4dc5ec7400>
[  115s] data_location = PosixPath('/home/abuild/rpmbuild/BUILD/openNCEM-master/ncempy/data')
[  115s] 
[  115s]     def test_read_dm3_1d(self, data_location):
[  115s]     
[  115s] >       metadata, img = self._read_dm3_data(data_location / Path('08_Carbon.dm3'))
[  115s] 
[  115s] ncempy/test/test_io_dm.py:63: 
[  115s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[  115s] ncempy/test/test_io_dm.py:39: in _read_dm3_data
[  115s]     with ncempy.io.dm.fileDM(file_path, on_memory=on_memory) as f:
[  115s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[  115s] 
[  115s] self = <ncempy.io.dm.fileDM object at 0x7f4dc5c5b1a0>
[  115s] filename = '/home/abuild/rpmbuild/BUILD/openNCEM-master/ncempy/data/08_Carbon.dm3'
[  115s] verbose = False, on_memory = False
[  115s] 
[  115s]     def __init__(self, filename, verbose=False, on_memory=True):
[  115s]     
[  115s]         self.filename = filename
[  115s]     
[  115s]         # necessary declarations, if something fails
[  115s]         self.fid = None
[  115s]     
[  115s]         self._on_memory = on_memory
[  115s]     
[  115s]         # check for string
[  115s]         # if not isinstance(filename, str):
[  115s]         #    raise TypeError('Filename is supposed to be a string')
[  115s]     
[  115s]         # check filename type
[  115s]         if isinstance(filename, str):
[  115s]             pass
[  115s]         elif isinstance(filename, Path):
[  115s]             filename = str(filename)
[  115s]         else:
[  115s]             raise TypeError('Filename is supposed to be a string or pathlib.Path')
[  115s]     
[  115s]         # Add a top level variable to indicate verbose output for debugging
[  115s]         self.v = verbose
[  115s]     
[  115s]         # try opening the file
[  115s]         try:
[  115s]             if not self._on_memory:
[  115s] >               self.fid = open(filename, 'rb')
[  115s] E               FileNotFoundError: [Errno 2] No such file or directory: '/home/abuild/rpmbuild/BUILD/openNCEM-master/ncempy/data/08_Carbon.dm3'
[  115s] 
[  115s] ncempy/io/dm.py:104: FileNotFoundError
[  115s] ----------------------------- Captured stdout call -----------------------------
[  115s] Error reading file: "/home/abuild/rpmbuild/BUILD/openNCEM-master/ncempy/data/08_Carbon.dm3"
[  115s] =============================== warnings summary ===============================
ercius commented 4 years ago

Yes. Its a typo which will not be found by testing in Windows. Thanks for finding that.

Updated. Try it again.

kevinsmia1939 commented 4 years ago

Yes. Its a typo which will not be found by testing in Windows. Thanks for finding that.

Updated. Try it again.

Build pass now, thank so much. Will be waiting for 1.6.0.

ercius commented 4 years ago

Im glad it finally built. It was a good push to get me to update old things and tag a new release.

I forgot to update the version in setup.py when I tagged the release. The version on pypi is the updated version. You can also see the release information here.