arobrien
commented
1 year ago A few projects have optional dependencies on accelerator libraries. For example GeoPandas will use pygeos if available, otherwise fallback to alternative implementations (with configuration overrides). I think this serves as a good model for how this speedup can be made available to users who need it, without fracturing the ecosystem.
# _compat.py
INSTALL_PYGEOS_ERROR = "To use PyGEOS within GeoPandas, you need to install PyGEOS: \
'conda install pygeos' or 'pip install pygeos'"
try:
import pygeos # noqa
# only automatically use pygeos if version is high enough
if Version(pygeos.__version__) >= Version("0.8"):
HAS_PYGEOS = True
PYGEOS_GE_09 = Version(pygeos.__version__) >= Version("0.9")
PYGEOS_GE_010 = Version(pygeos.__version__) >= Version("0.10")
else:
warnings.warn(
"The installed version of PyGEOS is too old ({0} installed, 0.8 required),"
" and thus GeoPandas will not use PyGEOS.".format(pygeos.__version__),
UserWarning,
)
HAS_PYGEOS = False
except ImportError:
HAS_PYGEOS = False
def set_use_pygeos(val=None):
"""
Set the global configuration on whether to use PyGEOS or not.
The default is use PyGEOS if it is installed. This can be overridden
with an environment variable USE_PYGEOS (this is only checked at
first import, cannot be changed during interactive session).
Alternatively, pass a value here to force a True/False value.
"""
global USE_PYGEOS
global USE_SHAPELY_20
global PYGEOS_SHAPELY_COMPAT
env_use_pygeos = os.getenv("USE_PYGEOS", None)
if val is not None:
USE_PYGEOS = bool(val)
else:
if USE_PYGEOS is None:
USE_PYGEOS = HAS_PYGEOS
if env_use_pygeos is not None:
USE_PYGEOS = bool(int(env_use_pygeos))
# validate the pygeos version
if USE_PYGEOS:
try:
import pygeos # noqa
# validate the pygeos version
if not Version(pygeos.__version__) >= Version("0.8"):
if SHAPELY_GE_20:
USE_PYGEOS = False
warnings.warn(
"The PyGEOS version is too old, and Shapely >= 2 is installed, "
"thus using Shapely by default and not PyGEOS."
)
else:
raise ImportError(
"PyGEOS >= 0.8 is required, version {0} is installed".format(
pygeos.__version__
)
)
# Check whether Shapely and PyGEOS use the same GEOS version.
# Based on PyGEOS from_shapely implementation.
from shapely.geos import geos_version_string as shapely_geos_version
from pygeos import geos_capi_version_string
# shapely has something like: "3.6.2-CAPI-1.10.2 4d2925d6"
# pygeos has something like: "3.6.2-CAPI-1.10.2"
if not shapely_geos_version.startswith(geos_capi_version_string):
warnings.warn(
"The Shapely GEOS version ({}) is incompatible with the GEOS "
"version PyGEOS was compiled with ({}). Conversions between both "
"will be slow.".format(
shapely_geos_version, geos_capi_version_string
)
)
PYGEOS_SHAPELY_COMPAT = False
else:
PYGEOS_SHAPELY_COMPAT = True
except ImportError:
raise ImportError(INSTALL_PYGEOS_ERROR)
if USE_PYGEOS and env_use_pygeos is None and SHAPELY_GE_20:
warnings.warn(
"Shapely 2.0 is installed, but because PyGEOS is also installed, "
"GeoPandas will still use PyGEOS by default for now. To force to use and "
"test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. "
"You can do this before starting the Python process, or in your code "
"before importing geopandas:"
"\n\nimport os\nos.environ['USE_PYGEOS'] = '0'\nimport geopandas\n\n"
"In a future release, GeoPandas will switch to using Shapely by default. "
"If you are using PyGEOS directly (calling PyGEOS functions on geometries "
"from GeoPandas), this will then stop working and you are encouraged to "
"migrate from PyGEOS to Shapely 2.0 "
"(https://shapely.readthedocs.io/en/latest/migration_pygeos.html).",
stacklevel=6,
)
USE_SHAPELY_20 = (not USE_PYGEOS) and SHAPELY_GE_20
set_use_pygeos()I think that the (having not done any profiling of my own....) core slowdowns are likely happening in the file read/write, and in the strip_output() method (along with the equivalent zeppelin method. strip_output() isn't really too complicated, so maintaining two parallel implementations wouldn't be too hard, given a robust enough test suite. I want to restructure the code to be a little bit more modular in the area, including creating a StripOptions class or similar, anyway, which would facilitate a clearer and more robust interface between these two projects.
mlucool
kynan
Coming from https://github.com/kynan/nbstripout/issues/33#issuecomment-1345175852.
@kynan
Rust is not ideal for everyone because 1) we can't have wheels for every OS and 2) it's harder to tinker with. I was thinking something along the lines of if this project added nbstripout-fast as an optional dependency that can gracefully fail then IFF nbstripout-fast installed and the options you want to use overlap (most do), it calls nbstripout-fast, if not it continues to call the python code (of course we can add a flag to force one or the other).
Mostly I just wanted to put the thought out there and see if there was any traction. I totally understand if this does not seem like something you want to do for this project as it's not a super clean design. Open to other ideas as well.