Storm 1.0.3

[dmitry-saritasa]

hi Barry,

Petrel complains that Apache storm version is not equal to 1.0.2 when I submit into storm cluster v 1.0.3 (due to supplied Jar file?)

../../jvmpetrel/target/storm-petrel-*-SNAPSHOT.jar

Where can I get jar file for 1.0.3?

Thanks, Dmitry

[barrywhart]

Are you using the 1.0.1.0.3 version from here? That one should work, unless I built it incorrectly.

If necessary, you can build from source by installing your desired Storm version, then running python setup.py bdist_egg from the Petrel source directory.

Let me know if the pypi version is wrong, I can build and upload a corrected one. The build process is still slightly manual, unfortunately, for example, the jvmpetrel/pom.xml file has to be updated for each new Storm version. There may be a better way to do it; I am not a Maven expert.

[barrywhart]

Instructions on building from source: https://github.com/AirSage/Petrel#installing-petrel-from-source

[dmitry-saritasa]

https://pypi.python.org/simple/petrel/

petrel-0.9.3.0.3-py2.7.egg
petrel-0.9.3.0.1-py2.7.egg
petrel-0.9.0.0.1-py2.7.egg
petrel-1.0.1.0.3-py2.7.egg
petrel-0.10.0.0.3-py2.7.egg
petrel-0.9.3.0.2-py2.7.egg
petrel-1.0.1.0.3-py2-none-any.whl
petrel-1.0.2.0.3-py2.7.egg
petrel-1.0.2.0.3-py3.5.egg
petrel-0.8.2.0.1-py2.7.egg
petrel-0.9.4.0.3-py2.7.egg

Only petrel-1.0.2.0.3-py3.5.egg is available for python 3.5

[dmitry-saritasa]

Just in case the following is the message I get:

(storm) > $ petrel submit  --destjar ../app.v0.1.3.jar --config topology.yaml app-persistence-v013
[Errno 2] No such file or directory: '/home/devops/.pyenv/versions/3.5.2/envs/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/generated/storm-petrel-1.0.3-SNAPSHOT-jar-with-dependencies.jar'
Traceback (most recent call last):
  File "/home/devops/.pyenv/versions/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/cmdline.py", line 126, in main
    func(**args.__dict__)
  File "/home/devops/.pyenv/versions/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/cmdline.py", line 54, in submit
    logdir=logdir)
  File "/home/devops/.pyenv/versions/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/package.py", line 100, in build_jar
    shutil.copy(source_jar_path, dest_jar_path)
  File "/home/devops/.pyenv/versions/3.5.2/lib/python3.5/shutil.py", line 235, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/devops/.pyenv/versions/3.5.2/lib/python3.5/shutil.py", line 114, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/home/devops/.pyenv/versions/3.5.2/envs/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/generated/storm-petrel-1.0.3-SNAPSHOT-jar-with-dependencies.jar'

[barrywhart]

I made some mistakes uploading. Apparently even though the Python code is compatible with both 2.7 and 3.5, I need to explicitly upload a wheel file for Python 3.5. I'll try and do this in the next day or two.

[barrywhart]

I uploaded a new build, petrel-1.0.3.0.3-py3-none-any.whl. Uploading the corresponding Python 2.7 wheel next. Have not yet done a test download/install.

[barrywhart]

Download from pypi in Python 3.5 looks good:

(Petrel-test-3.5.3) barry@Dell660s:~/dev/Petrel$ pip install petrel==1.0.3.0.3
Collecting petrel==1.0.3.0.3
  Downloading petrel-1.0.3.0.3-py3-none-any.whl (313kB)
    100% |████████████████████████████████| 317kB 745kB/s 
Requirement already satisfied: six==1.10.0 in /home/barry/.pyenv/versions/3.5.3/envs/Petrel-test-3.5.3/lib/python3.5/site-packages (from petrel==1.0.3.0.3)
Collecting PyYAML==3.10 (from petrel==1.0.3.0.3)
  Using cached PyYAML-3.10.tar.gz
Collecting thrift==0.9.3 (from petrel==1.0.3.0.3)
  Using cached thrift-0.9.3.tar.gz
Building wheels for collected packages: PyYAML, thrift
  Running setup.py bdist_wheel for PyYAML ... done
  Stored in directory: /home/barry/.cache/pip/wheels/cc/2a/d6/5a7108e2281e4c783740d79c40eac3ebc2d4157b1c7e4f17ef
  Running setup.py bdist_wheel for thrift ... done
  Stored in directory: /home/barry/.cache/pip/wheels/e5/20/32/cbe6d90e33b19825ea6d251ff0b1c778df8941750f5d5c3d3f
Successfully built PyYAML thrift
Installing collected packages: PyYAML, thrift, petrel
Successfully installed PyYAML-3.10 petrel-1.0.3.0.3 thrift-0.9.3

[barrywhart]

Getting an error when the wordcount topology starts up:

5131 [SLOT_1024] ERROR o.a.s.d.worker - Error on initialization of server mk-worker
java.io.FileNotFoundException: File '/tmp/6a9ce996-17a2-4065-adae-f82db42d6aee/supervisor/stormdist/test%20topology-1-1492557826/stormconf.ser' does not exist

[barrywhart]

The problem occurs with both Python 2.7 and 3.5 on Storm 1.0.3. I do not see the same issue with Storm 1.0.1. I will try Storm 1.0.2.

[barrywhart]

I uploaded 2.7 and 3.5 wheel files to pypi and verified that Storm 1.0.2 works correctly with Python 3.5.3. The error that is occurring in 1.0.3 sounds like a Storm issue that other people have reported at various times. (Try doing a web search for "stormconf.ser does not exist".)

It seems likely to be a Storm bug or a problem with my environment. Can you try Storm 1.0.3 with Petrel 1.0.3.0.2 and see if it works for you?

[dmitry-saritasa]

hi Barry - thank you! I will test tomorrow morning and let you know results

On Tue, Apr 18, 2017 at 5:12 PM, Barry Hart notifications@github.com wrote:

I uploaded 2.7 and 3.5 wheel files to pypi and verified that Storm 1.0.2 works correctly with Python 3.5.3. The error that is occurring in 1.0.3 sounds like a Storm issue that other people have reported at various times. (Try doing a web search for "stormconf.ser does not exist".)

It seems likely to be a Storm bug or a problem with my environment. Can you try Storm 1.0.3 with Petrel 1.0.3.0.2 and see if it works for you?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/AirSage/Petrel/issues/90#issuecomment-295022238, or mute the thread https://github.com/notifications/unsubscribe-auth/ABqqxckeOoXj9RnhT1WUw8FrYVqfwXXiks5rxVFjgaJpZM4M-GKv .

-- Simplicity - Beauty of the Person. Clickbrand.com - making tomorrow today

[dmitry-saritasa]

this is the error I got once topology is submitted

Already using interpreter /usr/local/bin/python3.5                                                                                                                                                                                        [0/0]
Using base prefix '/usr/local'
New python executable in /tmp/petrel-proj-persistence-v019-debug-20-1493249743/venv/bin/python3.5
Also creating executable in /tmp/petrel-proj-persistence-v019-debug-20-1493249743/venv/bin/python
Installing setuptools, pip, wheel...done.
Installing thrift==0.9.3
The directory '/home/storm/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/storm/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting thrift==0.9.3
  Downloading thrift-0.9.3.tar.gz
Installing collected packages: thrift
  Running setup.py install for thrift: started
    Running setup.py install for thrift: finished with status 'done'
Successfully installed thrift-0.9.3
Installing PyYAML==3.10
The directory '/home/storm/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/storm/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting PyYAML==3.10
  Downloading PyYAML-3.10.tar.gz (241kB)
Installing collected packages: PyYAML
  Running setup.py install for PyYAML: started
    Running setup.py install for PyYAML: finished with status 'done'
Successfully installed PyYAML-3.10
Installing petrel==1.0.3.0.3
Searching for petrel==1.0.3.0.3
Reading https://pypi.python.org/simple/petrel/
No local packages or working download links found for petrel==1.0.3.0.3
error: Could not find suitable distribution for Requirement.parse('petrel==1.0.3.0.3')

And then I'm getting tons of *.log files representing bolts and spouts tasks around 300 bytes long with similar content

[devops@storm-wk1-prod]~/storm/supervisor/stormdist/trivver-persistence-v019-debug-20-1493249743/resources% cat petrel23982_tbolts.log
Beginning task setup
/usr/bin/flock
Using existing venv: /tmp/petrel-proj-persistence-v019-debug-20-1493249743/venv
Task setup took 1 seconds
Launching: python -m petrel.run tbolts /var/log/storm/supervisor/stormdist/proj-persistence-v019-debug-20-1493249743/resources/petrel23982_tbolts.log

setup.sh

cat setup.sh
pip install -r requirements/production.txt
export PYTHONPATH=`pwd`

requirements/production.txt:

# Apache Storm Py lib
# you will need to install this lib manually
# so it's not called inside storm's supervisor
# petrel==1.0.3.0.3

# Important
# pip install --upgrade setuptools
# pip install --upgrade pip
# or you get nasty error
# see this thread
# https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2990
setuptools==34.3.2
thrift==0.9.3

# Database abstraction layer
psycopg2==2.7.1

# Timezones and datetime utils
pytz==2016.10

# redis SDK
redis==2.10.5

# nice string conversion procedures
inflection==0.3.1

# opbeat
opbeat==3.5.2

# boltons = super cool batteries included
# https://boltons.readthedocs.io/en/latest/
boltons==17.1.0

# msgpack for faster messaging to avoid JSONs
# http://msgpack.org/
msgpack-python==0.4.8
u-msgpack-python==2.3.0

# csv for fake data
tablib==0.11.4

# fake data generator
faker==0.7.10

# google cloud SDK (for BigTable)
# https://github.com/GoogleCloudPlatform/google-cloud-python
google-cloud==0.23.0

# signal processing
blinker==1.4

# cached property on classes
cached-property==1.3.0

# click for CLI
# http://click.pocoo.org/5/
click==6.7

# slacker
# https://github.com/os/slacker
# slack channel notifications
slacker==0.9.42

# kafka client
kafka-python==1.3.3

# this is needed for petrel
pyyaml==3.10

switched back petrel-1.0.2.0.3 installed with easy_install and now topology works successfully, snapshot of my virtualenv log after install https://gist.github.com/dmitry-saritasa/381c66cb5738f77c09149d238720dee1

[barrywhart]

I found an issue which I think explains this error:

Installing petrel==1.0.3.0.3
Searching for petrel==1.0.3.0.3
Reading https://pypi.python.org/simple/petrel/
No local packages or working download links found for petrel==1.0.3.0.3
error: Could not find suitable distribution for Requirement.parse('petrel==1.0.3.0.3')

Petrel was still trying to use easy_install to install itself from PyPI. I think now that it's being distributed as .whl files, easy_install won't work -- pip must be used instead.

I made that change and pushed a new version to PyPI: 1.0.3.0.3.post1. Because of the stormconf.ser issue I mentioned before, I can't actually test this new release, but I did verify it installs successfully with pip:

(Petrel-test-3.5.3) barry@Dell660s:~/dev/Petrel/samples/wordcount$ pip install petrel==1.0.3.0.3.post1
Collecting petrel==1.0.3.0.3.post1
  Downloading petrel-1.0.3.0.3.post1-py3-none-any.whl (556kB)
    100% |████████████████████████████████| 563kB 703kB/s 
Collecting thrift==0.9.3 (from petrel==1.0.3.0.3.post1)
Collecting PyYAML==3.10 (from petrel==1.0.3.0.3.post1)
Requirement already satisfied: six==1.10.0 in /home/barry/.pyenv/versions/3.5.3/envs/Petrel-test-3.5.3/lib/python3.5/site-packages (from petrel==1.0.3.0.3.post1)
Installing collected packages: thrift, PyYAML, petrel
Successfully installed PyYAML-3.10 petrel-1.0.3.0.3.post1 thrift-0.9.3

I have created a PR so you can see the changes related to adding 1.0.3 support. Please take a look; in case of issues, this may help you figure out any remaining problems more quickly.

https://github.com/AirSage/Petrel/pull/93

[LW-YUNKAI]

hi Barry, thank you for your work! I got the same issue when I submit the topology, I wondered if I should start up petrel directly or start up zookeeper and storm first, and I want to run it on a cluster, should I change storm.yaml in storm?

6109 [SLOT_1027] ERROR o.a.s.d.worker - Error on initialization of server mk-worker
java.io.FileNotFoundException: File '/tmp/f9ad8a9c-52ad-49b2-9c18-2302fd5b6449/supervisor/stormdist/test%20topology-1-1508916726/stormconf.ser' does not exist

[barrywhart]

@LW-YUNKAI: I am no longer an AirSage employee, and AirSage asked me to turn over control of the AirSage GitHub organization to them. As a result, I am no longer supporting Petrel. I regret I cannot help you.

[dmitry-saritasa]

oh that's really a bad news. That means that petrel is officially DEAD? Is it possible to take over support and maintenance of Petrel? Or intellectual property is owned by AirSage completely and nothing we can do?

On Wed, Oct 25, 2017 at 11:58 AM, Barry Hart notifications@github.com wrote:

@LW-YUNKAI https://github.com/lw-yunkai: I am no longer an AirSage employee, and AirSage asked me to turn over control of the AirSage GitHub organization to them. As a result, I am no longer supporting Petrel. I regret I cannot help you.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/AirSage/Petrel/issues/90#issuecomment-339435529, or mute the thread https://github.com/notifications/unsubscribe-auth/ABqqxdOIY9DV1sYI7QAZtGlXOoz3U6enks5sv4TxgaJpZM4M-GKv .

-- Simplicity - Beauty of the Person. Clickbrand.com - making tomorrow today

[LW-YUNKAI]

@barrywhart It is a pity! I hope that STORM could have better support for python in the future! Thank you anyway!