search

Reading-eScience-Centre / pycovjson

Create CovJSON files from common scientific data formats
BSD 3-Clause "New" or "Revised" License
12 stars 7 forks source link

Python 2.7 support #8

Closed guygriffiths closed 7 years ago

guygriffiths commented 8 years ago

Most of the academic community is still using python 2.7, and this code is not backwards-compatible with it. For example, on my system trying to run pycovjson-convert gives a syntax error:

Traceback (most recent call last):
  File "/usr/local/bin/pycovjson-convert", line 9, in <module>
    load_entry_point('pycovjson==0.1.0', 'console_scripts', 'pycovjson-convert')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "build/bdist.linux-x86_64/egg/pycovjson/cli/convert.py", line 8, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pycovjson-0.1.0-py2.7.egg/pycovjson/write.py", line 11
    def __init__(self, output_name: object, dataset_path: object, vars_to_write: object, tiled=False, tile_shape=[] ) -> object:
                                  ^
SyntaxError: invalid syntax
guygriffiths commented 8 years ago

Trying again with python3, it seems that the pip install method only works for python 2.7. python setup.py install also only works for python 2.7.

So currently I can only install this with python 2.7, but the syntax means it can only run on python 3.

jonblower commented 8 years ago

Yes, we didn't have time to make it compatible with both Pythons. Since this is probably a utility for admins / data managers rather than end users we thought it was OK to focus on Py3. Would be nice if it could work with both though!

lewismc commented 8 years ago

Can someone using Python 2.7 try this again? I try to use 3.X as much as possible and don't have the 2.7 install handy.

lewismc commented 8 years ago

Hi Folks, I got a new laptop recently and have the following issues

lmcgibbn@LMC-056430 /usr/local/pycovjson(master) $ python --version
Python 2.7.12 :: Continuum Analytics, Inc.

cd pycovjson
python setup.py install
...
lmcgibbn@LMC-056430 /usr/local/pycovjson(master) $ python setup.py install
running install
running bdist_egg
running egg_info
creating pycovjson.egg-info
writing requirements to pycovjson.egg-info/requires.txt
writing pycovjson.egg-info/PKG-INFO
writing namespace_packages to pycovjson.egg-info/namespace_packages.txt
writing top-level names to pycovjson.egg-info/top_level.txt
writing dependency_links to pycovjson.egg-info/dependency_links.txt
writing entry points to pycovjson.egg-info/entry_points.txt
writing manifest file 'pycovjson.egg-info/SOURCES.txt'
reading manifest file 'pycovjson.egg-info/SOURCES.txt'
writing manifest file 'pycovjson.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.6-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/pycovjson
copying pycovjson/__init__.py -> build/lib/pycovjson
copying pycovjson/convert.py -> build/lib/pycovjson
copying pycovjson/model.py -> build/lib/pycovjson
copying pycovjson/read_netcdf.py -> build/lib/pycovjson
copying pycovjson/write.py -> build/lib/pycovjson
copying pycovjson/write_netcdf.py -> build/lib/pycovjson
copying pycovjson/writeNetCDF.py -> build/lib/pycovjson
creating build/lib/pycovjson/cli
copying pycovjson/cli/__init__.py -> build/lib/pycovjson/cli
copying pycovjson/cli/convert.py -> build/lib/pycovjson/cli
copying pycovjson/cli/viewer.py -> build/lib/pycovjson/cli
creating build/lib/pycovjson/test
copying pycovjson/test/__init__.py -> build/lib/pycovjson/test
copying pycovjson/test/test.py -> build/lib/pycovjson/test
creating build/bdist.macosx-10.6-x86_64
creating build/bdist.macosx-10.6-x86_64/egg
creating build/bdist.macosx-10.6-x86_64/egg/pycovjson
copying build/lib/pycovjson/__init__.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson
creating build/bdist.macosx-10.6-x86_64/egg/pycovjson/cli
copying build/lib/pycovjson/cli/__init__.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson/cli
copying build/lib/pycovjson/cli/convert.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson/cli
copying build/lib/pycovjson/cli/viewer.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson/cli
copying build/lib/pycovjson/convert.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson
copying build/lib/pycovjson/model.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson
copying build/lib/pycovjson/read_netcdf.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson
creating build/bdist.macosx-10.6-x86_64/egg/pycovjson/test
copying build/lib/pycovjson/test/__init__.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson/test
copying build/lib/pycovjson/test/test.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson/test
copying build/lib/pycovjson/write.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson
copying build/lib/pycovjson/write_netcdf.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson
copying build/lib/pycovjson/writeNetCDF.py -> build/bdist.macosx-10.6-x86_64/egg/pycovjson
byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/cli/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/cli/convert.py to convert.pyc
byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/cli/viewer.py to viewer.pyc
byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/convert.py to convert.pyc
byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/model.py to model.pyc
  File "build/bdist.macosx-10.6-x86_64/egg/pycovjson/model.py", line 255
    def get_tiles(self, tile_shape: object,  array) -> object:
                                  ^
SyntaxError: invalid syntax

byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/read_netcdf.py to read_netcdf.pyc
  File "build/bdist.macosx-10.6-x86_64/egg/pycovjson/read_netcdf.py", line 29
    def print(self):
            ^
SyntaxError: invalid syntax

byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/test/__init__.py to __init__.pyc
byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/test/test.py to test.pyc
byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/write.py to write.pyc
  File "build/bdist.macosx-10.6-x86_64/egg/pycovjson/write.py", line 12
    def __init__(self, output_name: object, dataset_path: object, vars_to_write: object, tiled=False, tile_shape=[]) -> object:
                                  ^
SyntaxError: invalid syntax

byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/write_netcdf.py to write_netcdf.pyc
byte-compiling build/bdist.macosx-10.6-x86_64/egg/pycovjson/writeNetCDF.py to writeNetCDF.pyc
creating build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying pycovjson.egg-info/PKG-INFO -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying pycovjson.egg-info/SOURCES.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying pycovjson.egg-info/dependency_links.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying pycovjson.egg-info/entry_points.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying pycovjson.egg-info/namespace_packages.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying pycovjson.egg-info/requires.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying pycovjson.egg-info/top_level.txt -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
copying pycovjson.egg-info/zip-safe -> build/bdist.macosx-10.6-x86_64/egg/EGG-INFO
creating dist
creating 'dist/pycovjson-0.3.8-py2.7.egg' and adding 'build/bdist.macosx-10.6-x86_64/egg' to it
removing 'build/bdist.macosx-10.6-x86_64/egg' (and everything under it)
Processing pycovjson-0.3.8-py2.7.egg
Copying pycovjson-0.3.8-py2.7.egg to /Users/lmcgibbn/miniconda2/lib/python2.7/site-packages
Adding pycovjson 0.3.8 to easy-install.pth file
Installing pycovjson-viewer script to /Users/lmcgibbn/miniconda2/bin
Installing pycovjson-convert script to /Users/lmcgibbn/miniconda2/bin

Installed /Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/pycovjson-0.3.8-py2.7.egg
Processing dependencies for pycovjson==0.3.8
Searching for pandas
Reading https://pypi.python.org/simple/pandas/
Best match: pandas 0.19.0
Downloading https://pypi.python.org/packages/58/2c/62ba69f6cf16693fc341df084e8262ebb89f678ddc534164099fc7ddeb0b/pandas-0.19.0.zip#md5=6243f3abeaed035720bd77237bdade53
Processing pandas-0.19.0.zip
Writing /var/folders/mw/f_p4r21n5ld16hnyd7j8r8w4fv8362/T/easy_install-_FCCat/pandas-0.19.0/setup.cfg
Running pandas-0.19.0/setup.py -q bdist_egg --dist-dir /var/folders/mw/f_p4r21n5ld16hnyd7j8r8w4fv8362/T/easy_install-_FCCat/pandas-0.19.0/egg-dist-tmp-8fL03S
package init file 'pandas/io/tests/sas/__init__.py' not found (or not a regular file)
warning: no files found matching 'README.rst'
no previously-included directories found matching 'doc/build'
warning: no directories found matching 'examples'
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '#*' found anywhere in distribution
warning: no previously-included files matching '.git*' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
warning: no previously-included files matching '*.png' found anywhere in distribution
package init file 'pandas/io/tests/sas/__init__.py' not found (or not a regular file)
package init file 'pandas/io/tests/sas/__init__.py' not found (or not a regular file)
UPDATING build/lib.macosx-10.6-x86_64-2.7/pandas/_version.py
set build/lib.macosx-10.6-x86_64-2.7/pandas/_version.py to '0.19.0'
Traceback (most recent call last):
  File "setup.py", line 62, in <module>
    entry_points=_entry_points,
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/install.py", line 67, in run
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/install.py", line 117, in do_egg_install
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/easy_install.py", line 409, in run
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/easy_install.py", line 645, in easy_install
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/easy_install.py", line 696, in install_item
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/easy_install.py", line 741, in process_distribution
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/pkg_resources/__init__.py", line 826, in resolve
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/pkg_resources/__init__.py", line 1092, in best_match
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/pkg_resources/__init__.py", line 1104, in obtain
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/easy_install.py", line 664, in easy_install
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/easy_install.py", line 694, in install_item
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/easy_install.py", line 875, in install_eggs
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/easy_install.py", line 1114, in build_and_install
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/easy_install.py", line 1100, in run_setup
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 246, in run_setup
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 166, in save_modules
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 141, in resume
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 154, in save_modules
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 243, in run_setup
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 273, in run
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 242, in runner
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/sandbox.py", line 46, in _execfile
  File "/var/folders/mw/f_p4r21n5ld16hnyd7j8r8w4fv8362/T/easy_install-_FCCat/pandas-0.19.0/setup.py", line 680, in <module>

  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/bdist_egg.py", line 161, in run
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/bdist_egg.py", line 147, in call_command
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg/setuptools/command/install_lib.py", line 10, in run
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/command/install_lib.py", line 111, in build
    self.run_command('build_ext')
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/lmcgibbn/miniconda2/lib/python2.7/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/var/folders/mw/f_p4r21n5ld16hnyd7j8r8w4fv8362/T/easy_install-_FCCat/pandas-0.19.0/setup.py", line 371, in build_extensions

  File "/var/folders/mw/f_p4r21n5ld16hnyd7j8r8w4fv8362/T/easy_install-_FCCat/pandas-0.19.0/setup.py", line 129, in build_extensions

ImportError: Building pandas requires cython
RileyWilliams commented 8 years ago

@lewismc At the moment pycovjson requires python3, installing the latest version of conda for python 3.5 will allow you to use it.

lewismc commented 8 years ago

OK, so this issue can therefore be closed off as null and void? I am happy to upgrade to Python3 locally.

lewismc commented 8 years ago

OK so I just upgraded to new Miniconda Python 3.X. I am getting a error saying that pandas requires cython. We don't have cython defined as a dependency https://github.com/Reading-eScience-Centre/pycovjson/blob/master/setup.py#L23

lewismc commented 8 years ago

The same happened for netCDF4 support. I think we need to add both cython and netCDF4 to thr requirements.

RileyWilliams commented 8 years ago

I removed netcdf4 as a dependency today as I was testing something with RTD, will add cython as a dependency tomorrow when I'm on my laptop aswell as adding netcdf.

letmaik commented 8 years ago

Cython should not be a dependency as this would mean that the user needs a compiler like gcc which is never the case on Windows for example. The issue is that the dependencies of pycovjson are not meant to be installed directly via pip install pycovjson but rather with conda, which was the whole point of using conda in the first place. So either of the two should be supported:

lewismc commented 8 years ago

@letmaik

Cython should not be a dependency as this would mean that the user needs a compiler like gcc which is never the case on Windows for example.

I agree

The issue is that the dependencies of pycovjson are not meant to be installed directly via pip install pycovjson but rather with conda, which was the whole point of using conda in the first place.

I get that and the conda install is fine, however that is not good for testing locally right? The issue I am having is when I attempt to install the pycovjson source package (master branch for example) locally for iterative testing on our data products. This is when the dependency errors begin.

letmaik commented 8 years ago

Why shouldn't it be good for local testing?

lewismc commented 8 years ago

If I use master branch and attempt an install as follows python setup.py install it doesn't install and I need to retrieve the dependencies manually. If this is the intended behavior (I would be surprised if it is), then that is fine. If it is not (which I suspect is the case) then I suggest we take action to resolve it in this ticket.

letmaik commented 8 years ago

That is intended. I don't see why there is a difference between a local development install and a "normal" install. Both use the same installation routines in the back, so for both you need to install dependencies somehow, and in this case the easiest is via conda.

On 05.10.2016 19:54, Lewis John McGibbney wrote:

If I use master branch and attempt an install as follows |python setup.py install| it doesn't install and I need to retrieve the dependencies manually. If this is the intended behavior (I would be surprised if it is), then that is fine. If it is not (which I suspect is the case) then I suggest we take action to resolve it in this ticket.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Reading-eScience-Centre/pycovjson/issues/8#issuecomment-251765751, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgaLNuMWg04B8DTIX045ZsU0tFrPPseks5qw_JqgaJpZM4J33Ut.

lewismc commented 8 years ago

OK sounds good. This has been a worthwhile thread and is good for reference. Someone can close it off now.

RileyWilliams commented 8 years ago

Just to let you know, the packaged version of pycovjson hosted on conda is not as up to date as the version on pip(which is the latest tagged release)

lewismc commented 8 years ago

ack I saw that, thanks

lewismc commented 7 years ago

@RileyWilliams @letmaik does someone want to close this issue off?