codingwithsurya
commented
1 year ago no python-package-conda.yml action no requirements.txt no conda/environment.yml (conda deprecation) update installation script in package.json at the root update dockerfiles (train dockerfile + Dockerfile) update readme to describe how to get setup on poetry (and add resource on pyenv to get the right python version) add pre-commit hooks in poetry to make sure poetry.lock and pyproject.toml are correctly in sync pre-commit hooks of interest to add are: poetry-check, poetry-lock. Learn more How to build pre-commit hooks into github: Here
Precommit YAML file to add to: Here
make sure to add logger to the list of packages as we just added it
karkir0003
dwu359
farisdurrani
Upgraded to Poetry to Improve Dependency Management + Code Cleanup
What user problem are we solving? Currently, our Python backend strongly depends on Anaconda for environment and package management, resulting in a lack of separation between development and production environments. This creates challenges in maintaining and replicating the exact dependencies required for each environment.
What solution does this PR provide?
This pull request introduces the use of Poetry as a Python package manager tool to better define our development and production dependencies. By adopting Poetry, we can achieve better separation between the two environments and simplify the management of dependencies. In addition, this sped up build checks from 30mins to 5 mins. This migration also allows for venv caching as well.
The key changes in this PR include:
Created a pyproject.toml file in the poetry branch, which serves as the configuration file for Poetry.
Populated the pyproject.toml file with the necessary dependencies, both for development and production modes. The dependencies were inspired by the existing conda/environment.yml or requirements.txt files.
Updated the Python build check in the GitHub Actions workflow to use Poetry for environment setup. This ensures that the correct dependencies are installed during the CI/CD process.
Updated the Dockerfiles to use Poetry for environment setup. This ensures consistency between local development and deployment environments.
Ensured that installing Python dependencies updates the pyproject.toml file automatically. This eliminates the need to manually add dependencies to the conda/environment.yml file or requirements.txt.
Code Cleanup
Also some minor code cleanup occurred with the wiki.tsx file. The path names were wrong and directly needed the entire path when accessing the png file. So, import statements were deleted and path names were directly implemented in the code. Also the learn.tsx file was deleted as it was blank and not in use.
Testing Methodology How did you test your changes and verify that existing functionality is not broken
To test these changes and verify that existing functionality is not broken, the following steps were taken:
Checked out the poetry-setup branch locally and ensured that Poetry was installed.
Built and ran the backend application locally using Poetry for environment setup. Confirmed that the application started successfully and all existing functionality was working as expected.
Triggered the CI/CD pipeline using the GitHub Actions workflow and verified that the build process completed without errors.
updated Docker files with the new Poetry-based environment setup and tested using docker build.
Requested feedback and conducted code reviews with team members to validate the changes and address any potential issues.
Any other considerations
Update documentation and ReadME files + we need to gitignore poetry.lock
add pre-commit hooks in poetry to make sure poetry.lock and pyproject.toml are correctly in sync