search

lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

Dependency improvements #1324

Closed tomwhite closed 4 months ago

tomwhite commented 5 months ago

Lithops has a lot of dependencies in the base package, many of which are not needed when running Lithops.

There are a few things we could do here:

  1. Move dependencies for backends into 'extra' packages
    • boto3 - move to lithops[aws] (see #1323)
    • ibm* - move to lithops[ibm]
    • kubernetes?
  2. Make dependencies for optional features optional
    • redis
    • pandas/matplotlib/seaborn - for plotting
  3. Remove unused dependencies
    • lxml
JosepSampe commented 5 months ago

If I'm not wrong, the docker and the python-dateutil packages can also be removed. I remember they were used in the past, but now I can't find any import of these libraries.

JosepSampe commented 5 months ago

@gilv Do you agree on moving IBM dependencies from the default installation to extra packages? so that when using an IBM backend, users will have to install lithops like this: pip install lithops[ibm]?

JosepSampe commented 4 months ago

Closing as completed