search

MolSSI / QCFractal

A distributed compute and database platform for quantum chemistry.
https://molssi.github.io/QCFractal/
BSD 3-Clause "New" or "Revised" License
144 stars 47 forks source link

Lazy load zstandard #827

Closed mattwthompson closed 5 months ago

mattwthompson commented 5 months ago

Description

This PR defers import zstandard until it's needed. Some weird packaging issues cause a conflict that prevents any QCPortal import. I'm not sure when these compression/caching modules are used, i.e. by common user operations or only by admins. If they're ever not needed, this avoids an unnecessary crash.

Changelog description

Lazy load zstandard

Status

bennybp commented 5 months ago

Playing around with the zstd stuff, I see they have an environment variable for how the backend gets loaded. It seems to help in my case. https://python-zstandard.readthedocs.io/en/latest/api_usage.html

$ PYTHON_ZSTANDARD_IMPORT_POLICY='cffi_fallback' python3 -c 'from openeye import oechem; import zstandard;print("ok")'
ok

So openeye bundles a zstd dynamic library with it? I see a ./libs/python3-linux-x64-g++10.x/libzstd.so.1 in the openeye package. I think using cffi would then make zstandard imports use the installed library that's part of the zstd package, but I'm not positive and no idea if that would cause problems.

mattwthompson commented 5 months ago

I wish I found that a while back - that's exactly what I need!