python3 support

[swnesbitt]

SHARPpy is now officially the only reason I have python2 on my system, and I suspect that this will increasingly be the case for other users.

Other than syntax issues, what are the major roadblocks of going to python3?

[wblumberg]

One of the major packages SHARPpy uses to support the GUI functionality is PySide. PySide has had some instability in its development and I believe it is not fully supported by Python 3. Because of this and the syntax issues upgrades to Python 3 would require a significant rewrite, which isn't something we can easily do right now.

However, we have been tossing around some plans for how to make our next step up with the GUI. We have some sample code that I believe uses the graphics card more heavily to make working with the Skew-T and hodograph much faster (important when viewing ensembles) that we'd like to implement. In addition, Qt 4 (what we're currently using right now) does not support the higher-resolution displays that are becoming common place. I believe support for Qt 5 is required to do this, which I hear that a new PySide package will include this, but I do not yet know the timeline: https://wiki.qt.io/PySide2

Many of us are Python 2.7 users, and haven't quite made the jump to 3 yet. Can you explain some of the benefits and downsides of making that jump? This would allow us to discuss this topic next time we meet.

[buffaloseven]

As an aside, I modified the OS X binary .plist to include

<key>NSHighResolutionCapable</key>
<true/>

And it now correctly displays the UI at retina resolutions (15" rMBP 2013, running at 2880x1800). No more blurry text!

[niallrobinson]

👍 for python 3 support

[wblumberg]

Was alerted to this thread again, which gave me a chance to revisit some of the issues associated with PySide. It looks like Qt is supporting Pyside2, which should help accelerate the development of this package. I'm thinking we should start exploring the use of this package in SHARPpy, but if we do move to it, installation of it needs to be as easy as installing pyside from the Anaconda package manager. Perhaps after SHARPpy v1.4 is released, we can take a closer look at how difficult it could be to convert to Pyside2 and therefore offer Python 3 support.

I've found hints that a software called Maya is moving towards Pyside2, which might be a sign that development of Pyside2 may have created a package that is stable enough for use since starting development in May 2015. Pyside2 needs to work with SHARPpy on the major operating systems (WIN/OSX/Linux) to minimize headaches with support. https://fredrikaverpil.github.io/2016/07/25/dealing-with-maya-2017-and-pyside2/

I've found a package that will search for the most relevant Qt bindings package within a user's directory. it looks like something like this could be used in SHARPpy to make the transition from Pyside to Pyside2 smoother: https://github.com/mottosso/Qt.py

[niallrobinson]

Maya's huge - so if they're using it, its a good sign its stable. Also, I haven't done it, but I believe its much easier to add in bespoke recipies for conda now via conda forge https://conda-forge.github.io/

(conda install sharppy anyone?)

[wblumberg]

I did some tests the other day with PySide2, and it's not quite ready for primetime with SHARPpy. I'm not quite sure why just yet. The sounding picker for the realtime observed and model data loads, with some odd graphical issues, but the SPCWindow that shows the skew-t and hodograph will not load. There's not any useful output from the command line to help me debug what the problem is just yet.

[wblumberg]

@tsupinie is working on a Python 3 implementation of SHARPpy. He's got the GUI working, so our PySide issues have been overcome. When we update the master branch from our development branch (Andover), the official branch of SHARPpy will support Python 3. As of about 4-5 months ago, there's a version of PySide 1 available through Anaconda conda-forge that works with Python 3.

[kenpryor67]

I downloaded and installed SHARPpy on my Linux computer: Linux rhw9150.star1.nesdis.noaa.gov 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I originally installed pyside and SHARPpy with Python 3.6.2. Should I re-install with Python 2.7? Thanks!

[kenpryor67]

I successfully downloaded and installed SHARPpy: Installed /data/home002/kpryor/miniconda3/envs/python2/lib/python2.7/site-packages/SHARPpy-1.3.0-py2.7.egg Processing dependencies for SHARPpy==1.3.0 Finished processing dependencies for SHARPpy==1.3.0

However, when I attempt to start the GUI, the following errors appear: (python2) ken.pryor@rhw9150: python full_gui.py Traceback (most recent call last): File "full_gui.py", line 25, in from sharppy.viz.SPCWindow import SPCWindow File "/data/home002/kpryor/miniconda3/envs/python2/lib/python2.7/site-packages/SHARPpy-1.3.0-py2.7.egg/sharppy/viz/init.py", line 1, in from skew import plotSkewT File "/data/home002/kpryor/miniconda3/envs/python2/lib/python2.7/site-packages/SHARPpy-1.3.0-py2.7.egg/sharppy/viz/skew.py", line 5, in from sharppy.viz.barbs import drawBarb File "/data/home002/kpryor/miniconda3/envs/python2/lib/python2.7/site-packages/SHARPpy-1.3.0-py2.7.egg/sharppy/viz/barbs.py", line 2, in from PySide import QtGui, QtCore ImportError: libicui18n.so.56: cannot open shared object file: No such file or directory

What other actions might be necessary to start the GUI?

[wblumberg]

@tsupinie has a working copy of SHARPpy that works with Python 3.6 (GUI works too). This should be merged into the SHARPpy Andover branch at some point.

https://github.com/tsupinie/SHARPpy/tree/py36

[wblumberg]

An additional update to this thread: PySide2 can apparently be downloaded through Anaconda's package manager, which can also be used with Python 3:

https://fredrikaverpil.github.io/2017/08/28/pyside2-easy-install/

This may serve to provide an update to SHARPpy that includes both Python 3 and PySide 2 capabilities, which would really be a nice upgrade for users.

I was able to download PySide2 flawlessly using the above link on macOS High Sierra and test SHARPpy with it using code I upgraded to support PySide2 a while ago: https://github.com/wblumberg/SHARPpy/tree/pyside2

So, SHARPpy loads the map using PySide2, but the map doesn't render similarly to PySide1. The US counties lines are drawn too thick, and the sounding points look a little like Target symbols when you zoom in close. The rest of the GUI is pretty snappy and comes up sharp on my screen. This might be a function of my screen resolution. One key problem exists: I cannot load real-time model or observed data into SPCWindow to test how that works. This was a problem in the last iteration of PySide2 that I tested, but I didn't document what happened.

When I attempt to load any data source (OBS, HRRR, etc.), the program throws the data source cannot be found error.

But if I load in archived soundings (after some adjustments), SPCWindow runs okay except for some font size issues:

I can't seem to find any major bugs with any of the program features either, but the program is definitely snappier with PySide2. Possible benefits of PySide2 is that it requires Qt5, which Matplotlib 2 works with.

[aarande]

One thing to keep in mind with all of this is to ensure support for the multitude of users that are out there. I agree using Qt.py to ensure compatibility with either PySide or PySide2 and maybe also use six to ensure python 2-3 compatibility.

I would of course love to move everything to the latest and greatest and I do at home....unfortunately for work I am usually stuck in older environments (RHEL, Python 2.7, etc) as will many other government entities and possibly academic. I am actually using SharpPy in an AWIPS2 local application and the base python is 2.7 and will be for the foreseeable future.

[CAM-Gerlach]

Not sure what the status of this work is, but I really do hope SHARPPy gets properly ported over to Qt5 (and full Python 3 compatibility) soon. Python 2 is finally EoL in a little over a year, and Qt4 has been EoL for much longer; based on what I've picked up from the Anaconda devs, it is only a matter of time before the Qt4 builds disappear from Anaconda entirely, which has rapidly become the dominant platform in the scientific Python ecosystem, particularly for packages relying on heavyweight dependencies like Qt that are tricky at best to build on non-Linux platforms. I am no expert, my own experience is limited to almost exclusively to writing Python 3 and "universal" Py2 + Py3 code and my time is limited, but if there are things the community can help chip in with, let us know.

For my part, I'll put in a vote for QtPy over Qt.Py if there's still room to choose at this point. Both have relatively similar userbases and features, support all 4 major bindings, are MIT licensed and are great tools, but QtPy may offer a few advantages for this application. Specifically, its more mature (having been around before Qt.Py), has seen more actively development recently particularly in the area of PySide2 support, has more unit tests and is tested against more platforms (Mac, Windows and Linux) than Qt.Py, is supported by the full Spyder organization with financial backing and a formal structure rather than just one guy.

Perhaps most importantly, considering it is designed specifically for the development of scientific Python applications and visualization tools (after all, it was originally written for Spyder, the Scientific Python Development Environment), it is much better supported by Anaconda (and the other major scientific Python distributions), as QtPy is shipped with every Anaconda install, is in the main defaults channel,l and is fully tested to function with each stable release, whereas Qt.Py can only be installed from the third-party conda-forge channel and is not tested in such a manner.

In the interests of full disclosure, I do note I am one of the core developers of Spyder, under the same organization that develops QtPy, although I am not one of the latter's primary contributors, so take it with a grain of salt. Thanks!

[wblumberg]

Dang. You make a very compelling argument for us to go with QtPy, @CAM-Gerlach. I may do some tests to see how well that works with SHARPpy. We have upgraded SHARPpy to work with Python 3, however, our development branches have gotten a little wonky over the last year or two (partially caused by a dissertation needing to be written) and we haven't merged the Python 3 support into the main branch just yet. I think we were waiting to do a big upgrade to Python 3, PySide2, and our developments to the Andover branch. How's your memory these days, @tsupinie ?

For @CAM-Gerlach : Do you have any insight into where the PySide2 project is in their development? I check every now and then, but my ability to access their long-term plans is often lacking.

[CAM-Gerlach]

Dang. You make a very compelling argument for us to go with QtPy

Well, I am the one who writes the grants and funding proposals for the Spyder project and does all the outreach/communications/PR-type stuff, so take that for what you will :)

@wblumberg What I know directly is pretty limited, since we mostly target PyQt5 at Spyder. As you've probably already heard, PySide has been adopted by the Qt company, ported to the latest Qt 5.11, the project officially renamed "Qt for Python" (although the package itself is still PySide2) and officially released as a "technology preview" (whether you consider that an alpha or a beta is up to the user's imagination). I've heard of some projects that do use it (our friends over at WinPython have some involvement with it, if I recall correctly), and it seems the pace of development has picked up significantly lately, at least judging by their regular status updates on their new wiki site.

However, as for practical use, I'm not sure. I'm far from an expert on that to begin with—it wasn't but two years ago I was at the NWC/NSSL as a Holling Scholar working on my first real research project involving serious programming (and ironically back then, I was a user of and big advocate for R over Python for data analysis...who would have thought that a couple years later I'd be a core developer of the dominant scientific Python IDE). If you'd like, I can ask around to put you in touch with a few people who'd know more intimately and might have some contacts with the PySide2 dev team.

These days we primarily target PyQt5 with Spyder (though still through QtPy); we officially dropped PySide a few years ago and finally PyQt4 a few months ago on our stable branch. Assuming PySide2 matures and offers comparable features, we'll probably move to officially support it again. However, the whole point of QtPy is to be able to support multiple bindings, so you can write your code and (theoretically) use it under the user's choice of binding—PySide2, PyQt5, or even obsolete Qt4-based versions.

According to the legal information I've consulted (I manage all the legal stuff for the project and have always been intrigued by the finer points of copyright law, but most certainly IANAL), this is my understanding of the legal situation at play. Since QtPy supports multiple bindings that the user selects and install themselves, with various licenses, and your application would be dynamically linked to QtPy at runtime, and that in turn to your Qt binding of choice, you're effectively insulated from any GPL considerations. Therefore, simply supporting (implicitly or explicitly) PyQt5 through QtPy can and does have no bearing on the license and derivative status of SHARPPy, so you're free to maintain your permissive license (QtPy itself is MIT, as is Spyder).

Of course, if you actually distribute a standalone bundled package that includes PyQt5, rather than the user installing it themselves (or PySide2, as they see fit), then you could (theoretically, at least) run into difficulties. We used to distribute standalone versions of Spyder for Windows and Mac, but ceased doing so a while ago for unrelated reasons—the maintainers at the time found the maintenance burden was way too high, relative to directing users to Anaconda or other open source distributions instead that already included Spyder ready to go out of the box, and the latter was much better integrated into the existing scientific Python ecosystem.

[AFstaffmet]

The Air Force recently approved installation of Anaconda Python 3.6.x (Windows 10 64 bit, no Linux). So we are extremely interested in obtaining the Python 3 version of SHARPpy. A pre-compiled binary would not be an option or desire as it would require accreditation but we hope to be able to use the standard installation. The primary use would be the analysis of our own soundings so the GUI is not necessarily required. Is there a target date in mind for an initial release?

[CAM-Gerlach]

Glad the Air Force is allow you to use modern releases of open-source software! I've got an old buddy went into the USAF to be a met.

Python 3 version of SHARPpy

I'm not one of the developers so take this with a grain of salt, but to my understanding SHARPpy has mostly worked on Python 3 for some time (IIRC I used it all the way back with Python 3.4 or something), but for full support particularly for 3.6 they recommend the using the [Andover branch]https://github.com/sharppy/SHARPpy/tree/andover) which is in active flow to be merged into mainline and released soon according to the pinned issue #155 , so you can track it there. One of the major goals of that release appears to be full pip and conda builds which you can install in the "standard" way through either of those package managers rather than just as a standalone binary.

no Linux

:'(

[wblumberg]

Hi @AFstaffmet , I think the current target date would be the end of this year and at most, the end of January. We do have a pre-release (alpha stage) of Andover (binaries and source code) sitting in our releases page (https://github.com/sharppy/SHARPpy/releases) if you want to download from that. However, I did a test on the Windows binary recently and it failed to load the Picker. However, you can clone the GitHub repo, checkout the Andover branch, and install from there if you'd like a more recent version. As an aside: are you planning on using this for research or operations at the USAF?

@CAM-Gerlach summarized very nicely as to what we're planning and have been up to.

[wblumberg]

Since Qt for Python (PySide2) was recently released, I revisited this thread and updated Andover to use PySide2 from PIP via the QtPy package described on this thread. This will allow SHARPpy to use both PySide2 and PySide1 (although we should move to support PySide2). I'm keeping the GUI changes in a separate branch though for the time being.

Initial Observations:

1. The upgrade to PySide2 makes the GUI considerably faster and smoother. The readout cursors are no longer laggy and the Picker map seems to refresh quicker.
2. Font scaling still appears to be a problem with the widgets in the SPCWindow. Much of the text appears grainy.
3. Conflicts appear to occur between Qt5 installed via Conda and the Qt5 that comes packaged with PySide when pip is used to install. May need to wait till a conda version of PySide2 becomes available.
4. Map scaling issues (see post on Oct 31, 2017) are still occurring (line width for map).
5. The calendar widget on the Picker seems to be slow to update when you switch month or year. Usually changing the focus of the Picker window refreshes it.

I'm guessing that because the scaling issues appear in both the Picker map and the SPCWindow that there's some kind of switch that needs to be flipped on in PySide2/Qt5 that can allow SHARPpy to be rendered better on variable resolution displays. I've tried some variations of this command at the top (and in various locations) of full_gui.py with very little success on my copy of macOS Mojave:

https://www.reddit.com/r/learnpython/comments/69vm4t/pyqt5_and_high_resolution_monitors/ http://doc.qt.io/qt-5/highdpi.html

@CAM-Gerlach any suggestions?

I've created a branch called pyside2-py3 with these changes. The GUI tests that run on the TravisCI Linux OSes seg fault, so maybe we need to wait for a conda version of PySide2.

There's some talk about putting PySide2 on Conda-forge (the current version is old) on Gitter: https://gitter.im/PySide/pyside2. Perhaps this could be a place to try to resolve the issues described above.

[CAM-Gerlach]

Conflicts appear to occur between Qt5 installed via Conda and the Qt5 that comes packaged with PySide when pip is used to install.

Can you just install everything into a clean environment without Qt from Anaconda? AD is on 5.9 and CF is still on 5.6, neither of which are compatible with the released stable of Qt for Python (it needs >= 5.11). You'll presumably need to install and manage the Qt library externally if I understand correctly since pip doesn't work well with non-Python binaries natively and that needs to be handled separately (thus the rationale for conda as an alternative).

May need to wait till a conda version of PySide2 becomes available.

Basically, the blocker on this is support for Qt 5.12 LTS (technically >= 5.11), which stable released version of "Qt for Python" (PySide2) requires. Right now the Anaconda distribution (AD, i.e. the defaults conda channel) is on Qt 5.9 LTS, and as I understand it conda-forge is still stuck on 5.6. Now that 5.12 has been officially released upstream along with a stable PySide2 5.12, Ray said he's working on Qt 5.12 for AD, and once that's released, getting modern PySide2 5.12 available as a conda package will be next in line.

The place to keep tabs on this stuff is the feedstock repos in the conda-forge GitHub org, particularly the Qt feedstock and PySide2 feedstock repos, Ray (mingwandroid) is the main Anaconda dev who works on that (he's a nice guy, unless you mix AD and conda-forge binary packages and expect full ABI compatibility, heh). It will presumably be released on defaults as well as conda-forge since he works mainly on the AD side.

I'm guessing that because the scaling issues appear in both the Picker map and the SPCWindow that there's some kind of switch that needs to be flipped on in PySide2/Qt5 that can allow SHARPpy to be rendered better on variable resolution displays.

I assume this is due to hiDPI issues, since we've seen the same thing, particularly on macOS (we currently don't have any devs on the team that use it as their primary OS nor with easy access to hi-DPI monitors other than me borrowing other people's to test on). I'm very much not the expert on hiDPI or anything like that, but looking at our codebase, we do essentially the following at the start of our mainwindow module:

qtpy.QtCore.QCoreApplication.setAttribute(Qt.AA_EnableHighDpiScaling, True)

We also use setAttribute(Qt.AA_UseHighDpiPixmaps) in our QApplication for hiDPI icons, and you can set the Qt scale factor manually with the QT_SCALE_FACTOR env variable. Is any of that helpful?

[wblumberg]

Thank you very much for the information @CAM-Gerlach . That gives me something to look out for. I've starred and started following both feedstock repositories. Hopefully something will come out on conda soon. We could just install everything into separate environments, but some of the examples and tests being distributed with the package require Matplotlib. I've spent a lot of time and commits over the last month getting everything (testing the code, building the binaries, deploying to Github Releases, pip, conda, documentation, etc.) working on CI, so I'm not too enthusiastic about having to mess around with the CI scripts again. I hit so many walls while trying to make TravisCI use anaconda environments (I was hoping to clean up the process) and I couldn't figure out easily how to fix some of the issues that came up.

So I tried those changes you recommended. I also tried flipping on and off the RenderHints of the QPainter (maybe it was an aliasing issue), but I there weren't any noticeable changes in the text. Here are some pictures to compare:

This isn't an exact comparison, as SHARPpy using PySide with QtPy didn't like some of the Signals, Slots, and emits SHARPpy uses. I had trouble getting QtPy to play nicely when linked to PySide with SHARPpy, so I created two separate branches to show the side by side comparisons. So this is Andover with PySide imports and pyside2-py3 with QtPy imports linking to PySide2 (these branches are practically the same).

The interesting result is that PySide2 effectively doubles the GUI window size (2360x1600), even though the SPCWindow size in SHARPpy is hard coded to be 1180x800. It appears that the thickness of the lines and fonts drawn to the QPixmaps do not properly double in size. I did an experiment after that where I made QT_SCALE_FACTOR=0.5. It made the Picker and SPCWindow much smaller than I would like, but when I resized the SPCWindow manually, the inset text was significantly sharper. So that's an improvement. The SkewT and Hodograph fonts did not resize appropriately though.

[CAM-Gerlach]

I hit so many walls while trying to make TravisCI use anaconda environments (I was hoping to clean up the process) and I couldn't figure out easily how to fix some of the issues that came up.

Sorry you're experiencing problems with that. Carlos is much more intimately familiar with this department, but I'm not aware of significant issues regarding that in particular for us across our various environments (Python 2.7, 3.5, 3.6 and 3.7; Windows, macOS and Linux, and both conda and pip installs) in our build matrix across 4 CIs (Travis, Circle, Appveyor and now Azure). If you'd like, feel free to check out our travis.yml and probably more meaningfully our install.sh which are relatively straightforward.

there weren't any noticeable changes in the text

Sorry that didn't help things for you; I wish I had more specific advise but my own HiDPI working knowledge with PyQt is very rudimentary at best. We've observed at least in Qt 5.6 and 5.9 under PyQt that some widget classes scale with HiDPI resolutions, while some don't without workarounds; since HiDPI is such a tiny proportion of our global userbase and none of use have our own HiDPI monitor on which to test, HiDPI support is still limited in Spyder itself and some elements indeed are still pixelated or (as in your screenshots) simply not sized correctly when HiDPI scaling is applied. Have you considered testing on a non-HiDPI (e.g. 1080p or 1200p) display in order to isolate any issues that aren't related to HiDPI scaling on the latest high-end monitors?

Here are some pictures to compare:

For the record, what versions of Qt and PySide2 were you using for those? As you're probably aware, there have been significant improvements in HiDPI handling in recent Qt versions, from its initial introduction in Qt 5.6 LTS, numerous changes in 5.9 LTS (which I've observed in my own testing) and additional changes in 5.12 LTS. Also, as you probably also have experianced, Qt and PySide/QtfP versions earlier than 5.12 don't fully support Mojave and I believe there may be some remaining bugs even in 5.12.

This isn't an exact comparison, as SHARPpy using PySide with QtPy didn't like some of the Signals, Slots, and emits SHARPpy uses.

If you run into issues with QtPy that can be traced to a specific bug/incompatibility with the abstraction layer itself not supporting specific parts of the modern PySide 2 API, you can open an issue on our Github tracker and we can do our best to resolve it. If you have more general problems or questions while using it or aren't sure something is a bug, you can ask at our Spyder Gitter (make sure to @ Carlos, ccordoba12 who knows far more than I about it).

The interesting result is that PySide2 effectively doubles the GUI window size (2360x1600), even though the SPCWindow size in SHARPpy is hard coded to be 1180x800.

This is either due to OS-level scaling on your Mac that gets applied to apps that don't explicitly declare themselves to be HiDPI-aware, or due to Qt's auto scaling as previously discussed, where the virtual size of the GUI window is indeed 1180 x 800 even if the physical rendered size is not. As a side-note, if you're familiar with CSS3 and things like media queries and viewport sizing, you see much the same thing on the web with a HiDPI device's CSS-visible resolution being 2x, 3x or even 4x smaller than its true physical resolution.

I'm not that familiar with how it works on recent Macs as opposed to Windows, so I'm not 100% sure which. Again, testing on a machine with no HiDPI monitors attached (even one can sometimes cause scaling issues on the other in my own testing) could help isolate these issues.

It made the Picker and SPCWindow much smaller than I would like, but when I resized the SPCWindow manually, the inset text was significantly sharper.

Essentially, it seems some widgets are getting rendered at full resolution and then downsampled to 0.5x scale, and then when manually resized, are being essentially rendered at 2x resolution thus fully exploiting your HiDPI display. Of course, that shouldn't be a problem to begin with but with the number of known issues with HiDPI on Mac between Qt, PySide and the OS-integration, its not entirely surprising for now unfortunately. While supporting HiDPI with resolution-independent auto-scaling factors is great in theory, it does add issues of significant complexity at several levels which are still in the process of being ironed out.

[AFstaffmet]

Hi @AFstaffmet , I think the current target date would be the end of this year and at most, the end of January. We do have a pre-release (alpha stage) of Andover (binaries and source code) sitting in our releases page (https://github.com/sharppy/SHARPpy/releases) if you want to download from that. However, I did a test on the Windows binary recently and it failed to load the Picker. However, you can clone the GitHub repo, checkout the Andover branch, and install from there if you'd like a more recent version. As an aside: are you planning on using this for research or operations at the USAF?

@CAM-Gerlach summarized very nicely as to what we're planning and have been up to.

Sorry for the overdue reply to your aside question. The answer is hopefully both. I am working to QC 30 years of sounding data (~10,000 files). Once completed, I want to generate an analysis of each file for a study of thunderstorm initiation, specifically why we don't get anything on days that logic says we should. The other intent is to provide the graphical output for our soundings operationally. However, we do not perform routine soundings and they are often only in the morning so there is a desire to analyse HRRR point data too. Sure wish SHARPpy could read NCEP BUFR files directly instead of relying on BUFKIT files. That BUFKIT format is horrible with the surface data separate from the above ground data. I have NCEP BUFR decoder software which is in C and Fortran but that brings me back to the same software authorization issue that I would have with the SHARPpy Windows binary. If I wasn't at the "Hello World" level of Python experience and if I had the time I'd love to convert the NCEP code to Python.

[CAM-Gerlach]

@wblumberg Just a quick FYI, but apparently Carlos heard from Ray and the Anaconda crew that the release target for Qt and PyQt 5.12 at least for AD is in the next 1-2 months, such that we're preparing to support it in our own 3.x stable (and, necessarily, QtPy). PySide 5.12 is supposed to follow directly after, so that's good news (though, again, QtPy should at least theoretically allow you to support either one with the same codebase on your end).

[CAM-Gerlach]

@wblumberg As of a month ago, qt 5.12 is successfully built and packaged for conda-forge as is PySide (Qt for Python) 5.12, and 5.13 just came out a few days ago (if you'd rather stick to the latest rather than the LTS, though I recommend the former at least or now). Meanwhile, Carlos was mostly done finishing PyQt first half of last month, but he went silent on it and must have gotten busy with something else, but since your project seems to prefer PySide anyway and has QtPy as an abstraction layer, that hopefully shouldn't be a blocker. On the other hand, Conda-Forge has terminated Python 3.5 builds some time ago, so you'll need to make sure future releases are fully 3.6 and 3.7 compatible (which should be a breeze if it isn't already)

I'm no longer officially part of the Spyder core dev team and am really focused on my research/development at the moment so I'm not longer quite as in tune with this as I once was, but when I get the chance I might be of some help and based on some work done by others here as part of VORTEX-SE and MESO (where SHARPPy is, as you're probably aware, an official part of the workflow) I've offered to help push some new sounding I/O capabilities (for IMET and/or Windsonde) upstream if you're interested.

[wblumberg]

Hi @CAM-Gerlach ,

Thank you very much for this update. I don't know how much activity you've seen online, but I've been back at it trying to get Andover released with Python 3. Issues with PySide forced me to upgrade everything to PySide2, in which I saw the conda-forge update. It wasn't too painful as I had done a lot of the groundwork in this thread. I'm not using anything less than Python 3.6 now, so that's good. And you're right about us preferring PySide 2 - I don't know the details, but @keltonhalbert found it was compatible license-wise with what we're using.

There are a few issues arising with the binaries in PySide 2 that are holding up release right now. I really don't want to use pip's version of PySide 2, but that appears to be right now the only way I can get the Windows binaries built successfully on Azure Pipelines. You may be able to put the larger story together from Issue #155 and https://github.com/pyinstaller/pyinstaller/issues/2857.

I actually didn't know it was being used in another field project other than VORTEX-SE. What is MESO? I think I'd be interested in seeing what you have for the sounding I/O capabilities. We're trying hard to stay away from a file format that forces the user to have SHARPpy working to read it.

Will you be at AMS or AGU? I think I'd like to at least meet you in person at some point!

Greg

[wblumberg]

Noting that we've moved to Python 3 now. There is a new conda version for Python 3.6, 3.7 for the Andover release (beta).

[danieldjewell]

I'm sure you're aware, but PySide (original, not PySide2) is fully deprecated and hasn't had a release (at least on github) since 2015. Trying to compile it manually it complains that the highest Python version that's compatible is 3.4. (Also, Py2.7 is officially EOL now.)

[AFstaffmet]

Getting back to square 1 with learning Python and trying SharpPy again after 1+ year (wow!), I wanted to pass on some Windows 10 (64 bit) installation errors. First my Anaconda Python is up-to-date at 3.7 (conda update -all) with all dependencies installed including PySide2. Second, trying to install with conda install -c sharppy sharppy fails (see attachment to view all errors). Third, pip install sharppy works with one unsatisfactory side effect - it downgraded numpy from 1.81.1 to 1.15.4 which breaks the current astropy 4 in Anaconda 3.7 minimum of 1.16 although it supposedly installed ok. However, it crashed on startup. I uninstalled via pip uninstall sharppy and brought numpy back to 1.18.1. Based on these 2 failures, I did not bother trying to just download and install with python setup.py install. I'm not interested in the Windows binary version.

I hope this Python noobie experience is useful in debugging Andover.

Phil

(Edited for spelling.)

sharppy error.txt

[CAM-Gerlach]

@wblumberg Is there a reason you are restricting Python compat to >3.8? Is there something you really need that's only in 3.8? Its not even available on mainstream Anaconda yet and stuff is still being updated to support it, so as an application that can be used in a library context, I'd strongly advise maintaining compat with at least 3.7-3.8; the only really big things I can think of relevant to an application like this that might be required for compat is multiprocessing.shared_memory, assignment expressions and positional-only params (at least the latter two are mostly for developer convinience and clarity and can be worked around).

@AFstaffmet If you want to use Sharppy by itself instead of in a larger workflow, you're probably better off just installing it into its own environment, especially vs. contaminating base with packages from unofficial channels and especially pip. I suggest the following from an activated Anaconda prompt:

conda create -n sharppy-env -c sharppy -c conda-forge sharppy
conda  activate sharppy-env
sharppy

The relevant Sharppy-related error is:

```python-traceback (base) C:\WINDOWS\system32>sharppy Traceback (most recent call last): File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\ProgramData\Anaconda3\Scripts\sharppy.exe\__main__.py", line 4, in File "c:\programdata\anaconda3\lib\site-packages\runsharp\full_gui.py", line 18, in from datasources import data_source File "c:\programdata\anaconda3\lib\site-packages\datasources\data_source.py", line 31, in available = imp.load_source('available', avail_loc) File "c:\programdata\anaconda3\lib\imp.py", line 171, in load_source module = _load(spec) File "", line 696, in _load File "", line 677, in _load_unlocked File "", line 724, in exec_module File "", line 859, in get_code File "c:\programdata\anaconda3\lib\imp.py", line 157, in get_data return super().get_data(path) File "", line 916, in get_data FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Philip\\.sharppy\\datasources\\available.py' ```

Seems it expects certain datafiles to be present that aren't put there by default, which could have been missed by a developer on their machine with everything set up. Interesting that its looking for a source code file in a data directory...perhaps less than ideal design, but I'm not familiar enough with it to be the judge of its requirements. In any case that's an issue for @wblumberg .

[AFstaffmet]

This is probably partly due to my python inexperience. I installed Anaconda for all users vs just me. While this is fine for now at home, down the road at work it won't be, I was in base because updating python outside of it failed. I've also discovered MetPy and had to install other dependencies. I'll try your suggestion (may be a couple days). BTW, I did go ahead and download the zip file and tried python setup.py install. All seemed well but once I executed python full_gui.py, I got qt.qpa.plugin: Could not load the Qt platform plugin "windows" in "" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Available platform plugins are: direct2d, minimal, offscreen, windows. Perhaps for now I should uninstall Anaconda and reinstall "just for me."

[CAM-Gerlach]

I installed Anaconda for all users vs just me.

This shouldn't have anything to do with the problem. You should generally install Anaconda/Miniconda as an individual user unless you have a specific reason to do otherwise, and it could lead to problems updating conda itself in base, but I don't see how it could cause this.

I was in base because updating python outside of it failed.

What do you mean, outside of it? Outside of it where? In another environment, or having not activated any environment? With another Python interpreter? Which? Failed with what full, complete and exact error message? In general, its usually a good idea to create separate environments for separate applications to avoid dependency conflicts and other issues, which is exactly what happened to you above as you can see from the message. You also really don't want to fix packages from other channels, especially conda-forge, as Sharppy requires and was built with, with those from defaults (AD) or you might run into binary incompatibilities as may what you describe above.

I did go ahead and download the zip file and tried python setup.py install.

First off, you should always use pip install . instead of the legacy python setup.py install; in the modern packaging ecosystem the latter is a strict superset of the former and should always be preferred. If anyone tells you to install via the latter, replace it with the former and let them know to update their advice. This is not entirely their fault, given how messed up the Python packaging situation has been for a long time (though its certainly a lot better than it used to be). Hopefully in the future things will be better with PEP 517 and PEP 518, and tools like Flit and Poetry

Second, in what environment or Python interpreter did you run this? Its likely not going to work well on Windows systems since you need to install binary packages (e.g. Qt) that pip can't install by itself. That is why generally with applications (like we, the devs of the Spyder IDE, do) especially on Windows it is usually best for newer users to install them via conda/Anaconda, since it takes care of all the binary dependencies and package management. I would strongly recommend the approach of installing Sharppy into an isolated environment as stated above as the best way to avoid problems long term, unless the author provides a working standalone installer.

Perhaps for now I should uninstall Anaconda and reinstall "just for me."

Again, I don't see how that in any way solves your problems:

• First, you ran into the issue of an incompatible version spec by Sharppy. This can be worked around by either installing into a fresh, clean environment with the right channel setup, or by Sharppy being more liberal in its dependency specification.
• Second, you ran into a possible bug or install issue with Sharppy itself. This has nothing to do with Anaconda, as far as I can tell, so reinstalling Anaconda won't fix it. Again, its either a sharppy bug or a something went wrong with how you installed it.
• Third, I'm not sure what you're referring to when you say "updating Python outside of it" but if you don't have any environment activated when you try to run a command, then Anaconda is not going to work regardless of how it is installed.
• Finally, your problem installing the standalone is an install issue with that, not Anaconda, if you tried to install that in Anaconda itself you are in for trouble

That said, reinstalling may not be a bad idea because you have likely contaminated your base environment with the above efforts (mixing channels, pip installs, manual installs, etc). This is, again, why you should leave your base environment alone except for updating conda itself since unlike any other environment, which you can easily just delete and recreate if something goes wrong, if you contaminate base you have to reinstall all of Anaconda and loose all your environments, etc which is >10x more time and work.

Best of luck...

[AFstaffmet]

So I apparently misunderstood your previous comment regarding pip - sounded like it was not a good idea. When I referred to outside base vs inside base, I was referring to your use of the term base and now I realize your were referring to the Anaconda installation without any other packages as in right after a fresh installation. When I wrote 'I was in base because updating python outside of it failed,' the issue was that my Anaconda was installed last spring so I was updating it with 'conda update -all' after opening an 'Anaconda prompt' in Windows. The update failed. A little searching and it turned out I needed to right click and choose run as administrator so that the prompt had admin rights. The update then worked. I now realize that '(base)' appears at the command prompt when it is run as admin. Apparently I should have installed sharppy from a prompt that was opened without being administrator which does not include '(base)' in the command prompt.

I believe I will perform a fresh Anaconda installation and try the sharppy installation again without being in the base command prompt. Your replies are very much appreciated! FYI, while I'm not a professional programmer, I'm also not a programming noob (started with Fortran IV (!), do some C and a lot of Excel VBA and VisualBasic), just a python noob unfortunately.

@wblumberg Looking forward to updates regarding the seeming compatibility restrictions Andover may be imposing - just a typo? Many thanks to the team for the hard work.

I'll keep on plugging away as time permits. Thanks again!!!!

[wblumberg]

Hi all. I made some time to read through the thread this evening, and I think I understand some of what might be going on.

1.) I never setup the sharppy package on conda to be dependent on Python 3.8, so I'm not sure what's causing that message. I set up continuous integration to only build v1.4.0b1 for Python 3.6 and 3.7 for OS X, Linux, and Windows 64 bit, and that's what's up there right now. I ran "conda search sharppy --info --channel sharppy" on my command line, but I can't find any evidence of such dependency ( sharppy_deps.txt ). However, the same issue arose recently on a Twitter thread that I wasn't able to figure out.

@CAM-Gerlach, might you be able to take a look at the conda recipe I'm using and the build script to see if you can spot any issues? Please see the below links:

https://github.com/sharppy/SHARPpy/blob/andover/ci/deploy_conda.sh https://github.com/sharppy/SHARPpy/blob/andover/conda-recipe/meta.yaml

conda-forge is an added channel because the package requires PySide2.

2.) The pip version you downloaded has bugs and that's one of them. In that version the user's .sharppy directory does not update correctly on install. I do not recommend pip be used for downloading the SHARPpy package - in fact I stopped uploading to it some time ago. The version you downloaded was v1.4.0a5. You can probably confirm that by doing a "sharppy --version". The most recent one on conda is v1.4.0b1.

[CAM-Gerlach]

I don't see anything obvious in either the search results or the deploy scripts that would create the 3.8 limitation (although I assume the 3.5 and 2.7 specs are due to earlier versions its checking against). I suspect it has something to do with current incompatible packages that the solver found that limited compat to old versions of Sharppy only supporting 3.5 and 2.7, but I'm not sure why the 3.8 is there in the spec except that for some reason, some version was built against it at some point or its just some quirk of conda's solver. To note, the most restrictive requirement by a long shot is only building against a specific minor version of Numpy; I assume you will relax this in a future conda-forge release running on their build infra. Also, I see your Linux build is apparently commented out in those build scripts although this obviously has nothing to do with this problem.

I could not repro it because it very likely had to due with some downstream dependency requirement incompatible with something in @AFstaffmet 's existing environment, which is why I would suggest you strongly urge users to always install Sharppy in a clean, isolated environment to avoid these issues. I did so myself and had no issue installing and running Sharppy with just the simple commands I outlined above, except for for having an old .sharppy directory from many years ago that the install did not clean out and recreate (causing an error trying to read the Python 2-only scripts inside), though deleting it and relaunching Sharppy did the tick and I can launch it and view soundings just fine.

As you are probably aware, its a rather bad practice to have executable scripts deployed into a data/config directory as opposed to declarative configuration (TOML, JSON, etc); I assume its for extensibility but its terribly unmanagable and can create all sorts of problems like this, although it seems you've reduced that to the just the available.py script in the current version. Also, the choice of XML over JSON or TOML for configuration in this day and age is curious, though I assume its either for legacy or external reasons.

At least per standard practices in modern Python packaging, updating external data/config directories should generally be done at runtime (as Sharppy apparently does at least when I delete it), not at install-time. Support for doing so is deprecated from setuptools though I'm sure conda allows it to some degree due to its wider scope. This is probably a check that would be better as part of your application startup routine. Again, another reason to avoid executable code and other such files in external data directories unless user modification is an explicit and motivated requirement as it can quickly become hard to manage with standard tools.

Hopefully this insight is helpful...really I'm no expert at this, just learning as I go along like us all.

[wblumberg]

@CAM-Gerlach thank you for reviewing those deploy scripts and the conda recipe. I can't imagine what compatibility issue may have shown up that forced it to use only 3.5 and 2.7. The reasoning behind the strict 1.15 dependency is because NumPy has made subtle changes in the past that break SHARPpy in many utterly fantastic ways, and so we've started strongly controlling the NumPy version that gets used. I just bumped it up to 1.17, and I think I'll make a new conda release to see if that helps. 1.17 is only built for >=3.6 so maybe that might help solve the problem.

Per your request about strongly encouraging users to download into a clean isolated environment, I agree. I've updated the documentation, but something's going on with Travis CI and Github right now where it just failed to upload.

With respect to your comment about XML, JSON, TOML, and the available.py script. XML was chosen about 5 years ago, so that's a legacy choice. Similarly, the use of executable scripts in the data/config directory was also a legacy choice and was done for extensibility purposes. We'll think about this problem. For now, the installation guide (https://sharppy.github.io/SHARPpy/install.html) just highlights the ~/.sharppy directory as being a potential source of incompatibility and recommends the users delete theirs. Maybe when @tsupinie gets done with his PhD this spring he'll revisit this (he was the original creator of the data sources system).

[wblumberg]

Okay. There's a Python 3 version up on conda-forge right now of v1.4.0b and v1.4.0. I'll just close this extremely long thread since we're officially doing the Python 3 thing now. Andover was merged into master over the past two days.