Qwt5 not compatible with numpy-1.9

[o-smirnov]

Looks like we're going to hit a wall soon. Numpy 1.9 has dropped support for numpy.oldnumeric, which PyQwt5 appears to rely on. This causes the meqbrowser to fail.

[twillis449]

Try compiling from source code. The build checks if numeric or numarray is availble. If not, it should be ignoring them. Of course, you may have to install a whole bunch of -dev files if you are using Ubuntu in order to get things to work. I've only gone as far as trying to build and noting that messages about numarray etc not being found and will be ignored do appear.

[aragorn2101]

I have numpy 1.9.1, and yes, the meqtrees-timba build says that it is ignoring numeric and numarray, but it found numpy, so it is going to use numpy. However, when I tried to run meqbrowser, the software says that it needs numpy or numarray in order to work. So, unfortunately, it is unusable. I am reverting to a 'legacy' numpy 1.8.2.

[o-smirnov]

Yeah the problem is not the timba build per se -- the problem is that the PyQwt5 library internally checks for numpy.oldnumeric and bombs out if it is not found. So basically PyQwt5 is not compatible with numpy 1.9, and since the former is no longer maintained, the chances of a fix are slim. This breaks anything dependent on PyQwt5, meqbrowser included. Maybe @gijzelaerr can make a super-special fixed package for us...

[gijzelaerr]

sorry but no, shipping a custom pyqwt5 package in our repo is going to become a nightmare.

If PyQwt5 is not maintained anymore, why not fork it, fix the problem and make a release? That is something I will package.

[twillis449]

If you build PyQwt from source, the PyQwt configure.py script can be fed the options --disable-numarray and --disable-numeric which should disable detection and use of numeric and numarray.

[o-smirnov]

Can fork, sure -- but the "fork" will essentially consist of a tiny change to the default build options (as Tony points out), so is it even a fork then?

[twillis449]

@aragorn2101 could you try building PyQwt from source with numpy 1.9 and see if the problem disappears?

[gijzelaerr]

hm in that case, the best solution is to report it @ debian packaging here:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=python-qwt5-qt4;dist=unstable

if they accept it, I can backport the package to our PPA.

2015-12-10 16:20 GMT+02:00 Tony Willis notifications@github.com:

If you build PyQwt from source, the PyQwt configure.py script can be fed the options --disable-numarray and --disable-numeric which should disable detection and use of numeric and numarray.

— Reply to this email directly or view it on GitHub https://github.com/ska-sa/meqtrees/issues/860#issuecomment-163633002.

Gijs Molenaar http://pythonic.nl

[twillis449]

OK - so I installed numpy 10.1 and compiled PyQwt 5 with the flags I mentioned earlier set. PyQwt 5 then works. NOTE: Some of the examples wiil fail because they call a script, anynumpy.py which immediately has a test 'from numpy.oldnumeric.compat import *', which of course crashes. Replacing this file with the code

from numpy import *
Float = float

will get usually get things working.

[o-smirnov]

@gijzelaerr: could you please complain to Debian appropriately? According to https://github.com/ska-sa/meqtrees/issues/860#issuecomment-163633002, all that needs to be done is a rebuild of PyQwt5 with --disable-numarray and --disable-numeric.

As things stand, since numpy has dropped support for numarray/numeric as of 1.9, the standard PyQwt5 package is no longer compatible with it.

[gijzelaerr]

OK I have tried the disable flags, and the packages compile. I have reported it upstream at Debian and Ubuntu:

https://bugs.launchpad.net/ubuntu/+source/pyqwt5/+bug/1568872

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820696

Lets see what happens. If this is not going to work in the end, I will make a new package. Good news, although pyqwt5 is abandoned, it is still included in Ubuntu 16.06, so we can continue this route for an other couple of years.

[gijzelaerr]

i've made an experimental package including the disable flag. Please test if that works.

https://launchpad.net/~radio-astro/+archive/ubuntu/numpyfix/

If it works, I'll try to get it into Debian / Ubuntu.

[twillis449]

I believe that ubuntu 16.04 LTS is due to be released next week (April 21). I have some other stuff that I need to check works with ubuntu 16.04 and I will test the updated PyQwt package along with other MeqTrees stuff at that time.

[gijzelaerr]

We will switch to 16.04 in the long run, but that is unrelated to this issue. Op 14 apr. 2016 4:38 p.m. schreef "Tony Willis" notifications@github.com:

I believe that ubuntu 16.04 LTS is due to be released next week (April 21). I have some other stuff that I need to check works with ubuntu 16.04 and I will test the updated PyQwt package along with other MeqTrees stuff at that time.

— You are receiving this because you were assigned. Reply to this email directly or view it on GitHub https://github.com/ska-sa/meqtrees/issues/860#issuecomment-209975140

[twillis449]

I found another machine to test this on. The sequence of commands to install etc works. However one bug has crept in: There's a file 'anynumpy.py' that ends up installed in /usr/lib/python2.7/dist-packages/PyQt4/Qwt5/anynumpy.py which currently has line 9 as 'from numpy.oldnumeric.compat import ' This actually introduces the bug in numpy v 1.9 and later which we are trying to get rid of! Replace the current line 9 with 'from numpy import ' and most examples in the PyQwt5 source code distribution will work with numpy v 1.8 or v 1.10.

I see that I already raised the anynumpy.py issue back in a comment on Dec 10, 2015 https://github.com/ska-sa/meqtrees/issues/860#issuecomment-163665197 but I think that my solution proposed here is better, and causes fewer examples to break.

[gijzelaerr]

So the disable flags are not required? Op 15 apr. 2016 7:17 p.m. schreef "Tony Willis" notifications@github.com:

I found another machine to test this on. The sequence of commands to install etc works. However one bug has crept in: There's a file 'anynumpy.py' that ends up installed in /usr/lib/python2.7/dist-packages/PyQt4/Qwt5/anynumpy.py which currently has line 9 as 'from numpy.oldnumeric.compat import ' This actually introduces the bug in numpy v 1.9 and later which we are trying to get rid of! Replace the current line 9 with 'from numpy import ' and most examples in the PyQwt5 source code distribution will work with numpy v 1.8 or v 1.10.

— You are receiving this because you were assigned. Reply to this email directly or view it on GitHub https://github.com/ska-sa/meqtrees/issues/860#issuecomment-210552227

[twillis449]

If one builds PyQwt from source and numarray and numeric are NOT installed, then the build figures that out and will ignore them. If you happen to have numarray or numeric installed and you want to have the build ignore them, then you need to set those disable flags.

[gijzelaerr]

OK, so for the packaging actually we don't have to set the disable flags,but we only need to replace the anynumpy.py file?

[twillis449]

That would appear to be the case. On my test system (XUbuntu 14.04) with numpy 1.10 I got PyQwt compiled and working from source without having to specifically set the disable flags during the configure step.

[twillis449]

FYI - I also got PyQwt 5.2 compiled and running under Ubuntu 16.04

[gijzelaerr]

i'm working together with a debian maintainer to resolve this issue. We have created experimental packages that should solve these issues. We made them for trusty and xenial. @twillis449 can you try one of these and see if it solves the issue? you can find the packages here:

https://launchpad.net/~radio-astro/+archive/ubuntu/numpyfix

[twillis449]

I'm at the SKA SDP meeting in Malta this week. I probably can't have a look at this stuff until next week sometime.

[gijzelaerr]

ok sure. Let me know as soon you have tested, then we try to get this fixed upstream at debian once and for all.

2016-07-07 16:43 GMT+02:00 Tony Willis notifications@github.com:

I'm at the SKA SDP meeting in Malta this week. I probably can't have a look at this stuff until next week sometime.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meqtrees/issues/860#issuecomment-231099055, or mute the thread https://github.com/notifications/unsubscribe/AAT6pCnfWgLrd5sqPoqfnn2PicYMsjk9ks5qTRCTgaJpZM4EgFwo .

Gijs Molenaar http://pythonic.nl

[twillis449]

@gijzelaerr Apologies for the delayed response about PyQwt. I had a lot of distractions in July. I downloaded and installed on a Ubuntu 16.04 machine as per your instructions above and things seem to install OK. Most of the PyQwt examples supplied with the original PyQwt 5.0 source code package seem to work. A few don't but it looks like those that fail have programming bugs.

[gijzelaerr]

ok cool, thanks for testing. what about meqbrowser?

[twillis449]

Next thing to test :)

My tests with the PyQwt examples show that a few of the lower level numpy functions or parameters seem to have been changed / deleted with the default numpy supplied with ubuntu 16.04, so I wouldn't be surprised if there are some other possibilities for failure in MeqTrees land not only directly related to the browser but also some of the other python code that it pulls in. @o-smirnov you should be aware of that.

[twillis449]

It would appear that basic stuff (start up, then run calico-viewMS with inspector visualization ) seems to work OK). I'll try running something which generates images - will have to copy over some FPA stuff first.

[twillis449]

NOTE: Since i would appear to have a somewhat masochistic nature I build MeqTrees from source using the 'newbies' approach given at https://github.com/ska-sa/meqtrees/wiki/BuildFromSource

[gijzelaerr]

ok well let me know when you would declare this a working fix for Meqtrees, then we close this issue and we make sure this ends up into Debian as a fix.

[sjperkins]

Using the supplied numpyfix package I am able to run meqbrowser. However, popup menu widgets don't seem to work properly -- the cursor must be offset above menu items in order to select them.

Edit: I performed this test on Xenial 16.04.

[gijzelaerr]

Do you think this is related to numpy? Otherwise we should merge the patch, close this issue and move forward.

2016-08-15 11:45 GMT+02:00 Simon Perkins notifications@github.com:

Using the supplied numpyfix package I am able to run meqbrowser. However, popup menu widgets don't seem to work properly -- the cursor must be offset above menu items in order to select them.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meqtrees/issues/860#issuecomment-239765750, or mute the thread https://github.com/notifications/unsubscribe-auth/AAT6pI41eIKxWwnYm55-cfiJQzcitLnEks5qgDUtgaJpZM4EgFwo .

Gijs Molenaar http://pythonic.nl

[twillis449]

I haven't noticed this problem. I will have a look. However I have noticed that there are a couple of numpy changes which affect some 2-D image plots. I haven't had time to figure out what the solution is.

[gijzelaerr]

OK, well, it looks like the numerical issues are resolved, I think we should proceed with trying to upload the fixed numpy package to Debian.

[olebole]

Actually, what is fixed is not numpy, but pyqwt5 (right?). What I could do is to do an "NMU" (non-maintainer upload) for this package with the fix. This will, however take some time since I want to give the maintainer then some time (usually 5-10 days) to fix it himself. And I am currently on holidays, so my time to work on this is limited, weather dependent and affected by spontaneous changes :-) So, be patient.

[gijzelaerr]

ok sure, thanks! Enjoy your holiday

[twillis449]

I just did some tests with the meqbrowser on Ubuntu 16.04 and I don't have any problems with cursor/mouse offsets relative to popup menu items.

[gijzelaerr]

Hello,

Bug #820696 reported by you has been fixed in the Git repository. You can
see the changelog below, and you can check the diff of the fix at:

    http://git.debian.org/?p=python-modules/packages/pyqwt5.git;a=commitdiff;h=27b95fb

---
commit 27b95fb85eae8aaf1805829713ea49bd06efea55
Author: Gudjon I. Gudjonsson <gudjon@gudjon.org>
Date:   Thu Aug 18 11:26:14 2016 +0200

    * Disable numarray and numeric (Closes: #820696)
    * Bump standards version to 3.9.8, no change needed
    * Change dh_clean -k to dh_prep
    * Change recommends libqwt-dev to
      suggests libqwt5-qt4-dev (Closes: #763567)
    * Remove dpatch reference from README.source

diff --git a/debian/changelog b/debian/changelog
index b03655c..c5a74fa 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,8 +1,19 @@
 pyqwt5 (5.2.1~cvs20091107+dfsg-8) UNRELEASED; urgency=medium

+  [ Ondřej Nový ]
   * Fixed VCS URL (https)

- -- Ondřej Nový <novy@ondrej.org>  Tue, 29 Mar 2016 21:52:59 +0200
+  [ Gudjon I. Gudjonsson ]
+  * Disable numarray and numeric (Closes: #820696)
+  * Bump standards version to 3.9.8, no change needed
+  * Change dh_clean -k to dh_prep
+  * Change recommends libqwt-dev to
+    suggests libqwt5-qt4-dev (Closes: #763567)
+  * Add patch 03_disable_numeric_numarray
+  * Update patch 02_examples, change Float to float
+  * Remove dpatch reference from README.source
+
+ -- Gudjon I. Gudjonsson <gudjon@gudjon.org>  Thu, 18 Aug 2016 11:13:12 +0200

 pyqwt5 (5.2.1~cvs20091107+dfsg-7) unstable; urgency=low

[twillis449]

@gijzelaerr I just did a fresh install of Ubuntu 16.04 from scratch to get rid of a lot of dead wood. It's good that you made that repository version of PyQwt. It would appear that by default Qt4 source code is not supplied (or anyway It doesn't seem to be there in the synaptic package manager) for ubuntu 16.04 and I would have had to go through a laborious install of the Qt4 source code from scratch. But indeed the repository version PyQwt version seems to pull in everything OK.

[gijzelaerr]

Do you mean you could not find the binary package? To me it looks like it is part of 16.04:

http://packages.ubuntu.com/search?keywords=python-qwt&searchon=names&suite=xenial&section=all

[twillis449]

Sorry - my bad. I did a search in synaptic for 'PyQwt' and nothing appeared. However I just did a search for 'qwt' and there is is!

[gijzelaerr]

ok so for now use the repository, until the updated qwt package lands in Debian and then Ubuntu. Not sure if it end up in 16.04, probably not. Probably wise to upload the modifed qwt package into our PPA. @olebole what do you think? I don't really like overriding packages from debian/ubuntu.

[twillis449]

Yes - it appears that the 'standard' ubuntu 16.04 synaptic database refers to a version of PyQwt from 2009/11/07 whereas when I add the astron repository to the database I'm getting the latest version. Sorry for my confusion - I have too many versions of MeqTrees running on too many computers!

[olebole]

In Debian, pyqwt5 just migrated to testing, which fixes the incompatibility. For Ubuntu-16.04, a separate upload of the fix to Ubuntu would be needed, however.

[twillis449]

It looks like the author of the python-guiqwt package has also re-written the entire lower level PyQwt package in pure python as PythonQwt (see https://github.com/PierreRaybaut/PythonQwt). The docs suggest that some of the lower level API calls may have been changed a bit. Anyway - something to keep an eye on for future reference. I shall try using PythonQwt and see how much work is involved in switching to it.