Open dadokkio opened 6 years ago
Hi, yeah that seems fine.
The issue is actually how the wheel file gets unpacked on the YARN worker nodes. If you unzip the wheel file you'll see that there isn't an init.py file underneath the top level "ts" directory -- so the worker nodes that try to import "ts.anythingelse" will fail.
msanders in ~/Downloads
> mv ts_flint-0.6.0-py3-none-any.whl flint.zip
msanders in ~/Downloads
> unzip flint.zip
Archive: flint.zip
inflating: ts/flint/__init__.py
inflating: ts/flint/_version.py
inflating: ts/flint/clocks.py
inflating: ts/flint/context.py
inflating: ts/flint/dataframe.py
inflating: ts/flint/error.py
inflating: ts/flint/functions.py
inflating: ts/flint/group.py
inflating: ts/flint/java.py
inflating: ts/flint/readwriter.py
inflating: ts/flint/serializer.py
inflating: ts/flint/summarizers.py
inflating: ts/flint/udf.py
inflating: ts/flint/utils.py
inflating: ts/flint/windows.py
inflating: ts_flint-0.6.0.dist-info/top_level.txt
inflating: ts_flint-0.6.0.dist-info/WHEEL
inflating: ts_flint-0.6.0.dist-info/METADATA
inflating: ts_flint-0.6.0.dist-info/RECORD
The fix is to tell pip that flint also provides a top level "ts" directory: https://github.com/twosigma/flint/pull/65
This results in happier situation for the workers:
msanders in ~/code/flint/python/dist on master
> unzip ts_flint-0+untagged.302.g5aa2ada.dirty-py3-none-any.whl
Archive: ts_flint-0+untagged.302.g5aa2ada.dirty-py3-none-any.whl
inflating: ts/__init__.py
inflating: ts/flint/__init__.py
inflating: ts/flint/_version.py
inflating: ts/flint/clocks.py
inflating: ts/flint/context.py
inflating: ts/flint/dataframe.py
inflating: ts/flint/error.py
inflating: ts/flint/functions.py
inflating: ts/flint/group.py
inflating: ts/flint/java.py
inflating: ts/flint/readwriter.py
inflating: ts/flint/serializer.py
inflating: ts/flint/summarizers.py
inflating: ts/flint/udf.py
inflating: ts/flint/utils.py
inflating: ts/flint/windows.py
inflating: ts_flint-0+untagged.302.g5aa2ada.dirty.dist-info/METADATA
inflating: ts_flint-0+untagged.302.g5aa2ada.dirty.dist-info/WHEEL
inflating: ts_flint-0+untagged.302.g5aa2ada.dirty.dist-info/top_level.txt
inflating: ts_flint-0+untagged.302.g5aa2ada.dirty.dist-info/RECORD
Hi, I'm trying to use flint submitting a pyspark job on yarn.
Using same approach on master local works properly while on yarn seems to refer to invalid path and the import fails.
I was able to use the library following this similar topic extracting python code from the jar and copying it in my working directory.
Is that ok or there is a better way to proceed?