vaexio / vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
https://vaex.io
MIT License
8.22k stars 590 forks source link

Remove pkg_resources from direct imports #2293

Closed franz101 closed 1 year ago

franz101 commented 1 year ago

This is a hot fix for the IO read issues caused by using the legacy pkg_resources.iter_entry_points

legacy pkg_resources.iter_entry_points is replaced with importlib.metadata.entry_points

try:
  if sys.version_info[:2] >= (3, 10):
      # pylint: disable=no-name-in-module
      from importlib.metadata import entry_points
  else:
      from importlib_metadata import entry_points
except ImportError:
  import pkg_resources
  entry_points = pkg_resources.iter_entry_points
franz101 commented 1 year ago

The discussion of the issue can be found here: https://github.com/vaexio/vaex/issues/2283

franz101 commented 1 year ago

@JovanVeljanoski @maartenbreddels

This fixes the reading file error: https://colab.research.google.com/drive/1EG9898VtmO19FwfZKd_LJzqkz_YVWwlE?usp=sharing#scrollTo=S-T0Wjd3B_YU

Screenshot 2022-12-01 at 17 10 29

Here is an example of the fix: https://colab.research.google.com/drive/1HNtACZhFteXlqrpFQR3wGtnPuropRx1j?usp=sharing

The reason is due to pkg_resources unable to find file openers without kernel restart: https://git.smhi.se/climix/climix/-/merge_requests/165/diffs

maartenbreddels commented 1 year ago

Ok, just to make it clear I understand.

maartenbreddels commented 1 year ago

Man, I can't believe this is all so flakey, importlib_metadata==4.0 seems to work, 3.0 doesn't seem to have the groups argument.

franz101 commented 1 year ago

Actually it is not about the speed here. In Google colab after a fresh installation without a restart, pkg_resources entrypoints are an empty list. Therefore it won't even try any openers and fail.

I agree can't believe how bad python handles lazy loads and has API changes across releases.

maartenbreddels commented 1 year ago

Many thanks for helping us sort out this mess! :)

franz101 commented 1 year ago

its my first large open source project contribution. happy to contribute where I can. ❤️

maartenbreddels commented 1 year ago

Many thanks :)