Optimizing the Package Size

[kkiani]

Description:

To reduce the package size for users, we need to optimize dependency management. Currently, all dependencies are installed regardless of use case, which can unnecessarily bloat the package. For example, if a user is developing a Flask application with Damavand, they don’t need to install PySpark, which is used for Sparkle applications.

Proposed Solution:

Implement optional dependencies in the setup using pip’s extras feature (e.g., pip install damavand[Sparkle]). Allow users to install only the necessary dependencies based on their specific use case, such as Sparkle for PySpark applications or Flask for web applications.

Tasks:

• [ ] Define optional dependencies for different use cases.
• [ ] Ensure that core functionality remains intact while users can selectively install additional integrations (e.g., Sparkle, Flask).
• [ ] Test installation paths for different use cases to ensure smooth user experience.

Acceptance Criteria:

• [ ] Package size is reduced by making unnecessary dependencies optional.
• [ ] Users can install Damavand with specific dependencies based on their application needs.
• [ ] Documentation is updated to reflect the usage of optional dependencies.

[kkiani]

PDM optional dependencies should address this needs. For example to make sparkle an optional depenedency:

# first remove sparkle and pyspark from main dependencies
pdm remove pyspark
pdm remove sparkle

# add sparkle as optional dependency
pdm add -G spark "git+https://github.com/DataChefHQ/sparkle.git"

However, the CI/CD needs to be updated to install all optional dependencies.