search

locationtech / rasterframes

Geospatial Raster support for Spark DataFrames
http://rasterframes.io
Apache License 2.0
246 stars 45 forks source link

pweave output sensitive to python environment #142

Closed metasim closed 5 years ago

metasim commented 5 years ago

Running pweave in different Python environments results in different results. In one case the correct output (markdown with code evaluated) is generated. In the other only the input code is show with no evaluated results.

cc: @vpipkt

Environments

Working Environment

Running pweave

sbt:RasterFrames> pyrasterframes/pySetup pweave
[info] Copying '/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/src/main/python' to '/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python'
[info] Copying '/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/src/test/python' to '/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/tests'
[info] Running 'python setup.py pweave' in /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python
running pweave
/anaconda3/lib/python3.6/site-packages/setuptools/dist.py:481: UserWarning: The version specified ('0.8.0-SNAPSHOT') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % self.metadata.version
--------------------------------------------------
Running InitializingEnvironment
--------------------------------------------------
status
status
Processing chunk 1 named None from line 12
Processing chunk 2 named None from line 24
2019-06-07 20:02:19 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Processing chunk 3 named None from line 38
Processing chunk 4 named None from line 45
Processing chunk 5 named None from line 45
Weaved /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/docs/InitializingEnvironment.py to /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/docs/InitializingEnvironment.md
--------------------------------------------------
Running example
--------------------------------------------------
Processing chunk 1 named None from line 42
Processing chunk 2 named None from line 94
Processing chunk 3 named None from line 112
Processing chunk 4 named None from line 121
Weaved /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/docs/example.py to /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/docs/example.md
[success] Total time: 14 s, completed Jun 7, 2019 8:02:27 PM

Conda Pweave Info

$ python --version
Python 3.6.8 :: Anaconda custom (64-bit)

$ conda info
     active environment : None
...

$ pip show Pweave
Name: Pweave
Version: 0.30.3
Summary: Scientific reports with embedded python computations with reST, LaTeX or markdown
Home-page: http://mpastell.com/pweave
Author: Matti Pastell
Author-email: matti.pastell@helsinki.fi
License: LICENSE.txt
Location: /anaconda3/lib/python3.6/site-packages
Requires: nbformat, pygments, ipython, jupyter-client, ipykernel, markdown, nbconvert
Required-by:

Broken Environment

Running pweave

sbt:RasterFrames> pyrasterframes/pySetup pweave
[info] Copying '/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/src/main/python' to '/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python'
[info] Copying '/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/src/test/python' to '/Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/tests'
[info] Running 'python setup.py pweave' in /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python
running pweave
--------------------------------------------------
Running InitializingEnvironment
--------------------------------------------------
/anaconda3/envs/py3.7/lib/python3.7/site-packages/setuptools/dist.py:481: UserWarning: The version specified ('0.8.0-SNAPSHOT') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % self.metadata.version
status
Processing chunk 1 named None from line 12
Processing chunk 2 named None from line 24
Processing chunk 3 named None from line 38
Processing chunk 4 named None from line 45
Processing chunk 5 named None from line 45
Weaved /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/docs/InitializingEnvironment.py to /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/docs/InitializingEnvironment.md
--------------------------------------------------
Running example
--------------------------------------------------
status
status
Processing chunk 1 named None from line 42
Processing chunk 2 named None from line 94
Processing chunk 3 named None from line 112
Processing chunk 4 named None from line 121
Weaved /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/docs/example.py to /Users/sfitch/Coding/OSS/locationtech-rasterframes/pyrasterframes/target/python/docs/example.md
ERROR:root:Invalid alias: The name clear can't be aliased because it is another magic command.
ERROR:root:Invalid alias: The name more can't be aliased because it is another magic command.
ERROR:root:Invalid alias: The name less can't be aliased because it is another magic command.
ERROR:root:Invalid alias: The name man can't be aliased because it is another magic command.
[success] Total time: 4 s, completed Jun 7, 2019 8:09:33 PM

Conda Pweave Info

$ python --version
Python 3.7.3

$ conda info
     active environment : py3.7
     active env location : /anaconda3/envs/py3.7

...

$ pip show Pweave
Name: Pweave
Version: 0.30.3
Summary: Scientific reports with embedded python computations with reST, LaTeX or markdown
Home-page: http://mpastell.com/pweave
Author: Matti Pastell
Author-email: matti.pastell@helsinki.fi
License: LICENSE.txt
Location: /anaconda3/envs/py3.7/lib/python3.7/site-packages
Requires: jupyter-client, pygments, markdown, ipykernel, nbconvert, ipython, nbformat
Required-by:
metasim commented 5 years ago

Pertinent to https://github.com/locationtech/rasterframes/pull/143

metasim commented 5 years ago

Immediate Dependencies

Working Environment

$ for p in nbformat jupyter-client markdown nbconvert ipython ipykernel pygments; do pip show $p | egrep 'Name|Version'; done
Name: nbformat
Version: 4.4.0
Name: jupyter-client
Version: 5.2.2
Name: Markdown
Version: 3.1.1
Name: nbconvert
Version: 5.3.1
Name: ipython
Version: 6.2.1
Name: ipykernel
Version: 4.8.0
Name: Pygments
Version: 2.2.0

Broken Environment

$ for p in nbformat jupyter-client markdown nbconvert ipython ipykernel pygments; do pip show $p | egrep 'Name|Version'; done
Name: nbformat
Version: 4.4.0
Name: jupyter-client
Version: 5.2.4
Name: Markdown
Version: 3.1.1
Name: nbconvert
Version: 5.5.0
Name: ipython
Version: 7.5.0
Name: ipykernel
Version: 5.1.0
Name: Pygments
Version: 2.4.0
metasim commented 5 years ago

Helpful links:

metasim commented 5 years ago

Fixed in https://github.com/s22s/rasterframes/commit/029c220c3db035cfe0804f5135ce0e95417b021b, but does make me concerned about using pweave now, especially since there's a PR to fix it that's quite old and unmerged.

vpipkt commented 5 years ago

https://github.com/mpastell/Pweave/issues/140 🤔

On Fri, Jun 7, 2019, 20:56 Simeon H.K. Fitch notifications@github.com wrote:

Fixed in s22s@029c220 https://github.com/s22s/rasterframes/commit/029c220c3db035cfe0804f5135ce0e95417b021b, but does make me concerned about using pweave now, especially since there's a PR to fix it that's quite old and unmerged.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/locationtech/rasterframes/issues/142?email_source=notifications&email_token=AB3P4L3T7T4B37YNODLPIYDPZL7TXA5CNFSM4HV7WGHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXHJWOI#issuecomment-500079417, or mute the thread https://github.com/notifications/unsubscribe-auth/AB3P4LYSJELH2IOXQQSWYLDPZL7TXANCNFSM4HV7WGHA .