Poetry setup

[codingwithsurya]

Upgraded to Poetry to Improve Dependency Management

What user problem are we solving? Currently, our Python backend strongly depends on Anaconda for environment and package management, resulting in a lack of separation between development and production environments. This creates challenges in maintaining and replicating the exact dependencies required for each environment.

What solution does this PR provide?

This pull request introduces the use of Poetry as a Python package manager tool to better define our development and production dependencies. By adopting Poetry, we can achieve better separation between the two environments and simplify the management of dependencies. In addition, this sped up build checks from 30mins to 5 mins. This migration also allows for venv caching as well.

The key changes in this PR include:

Created a pyproject.toml file in the poetry-setup branch, which serves as the configuration file for Poetry.

Populated the pyproject.toml file with the necessary dependencies, both for development and production modes. The dependencies were inspired by the existing conda/environment.yml or requirements.txt files.

Updated the Python build check in the GitHub Actions workflow to use Poetry for environment setup. This ensures that the correct dependencies are installed during the CI/CD process.

Updated the Dockerfiles to use Poetry for environment setup. This ensures consistency between local development and deployment environments.

Ensured that installing Python dependencies updates the pyproject.toml file automatically. This eliminates the need to manually add dependencies to the conda/environment.yml file or requirements.txt.

Testing Methodology How did you test your changes and verify that existing functionality is not broken

To test these changes and verify that existing functionality is not broken, the following steps were taken:

Checked out the poetry-setup branch locally and ensured that Poetry was installed.

Built and ran the backend application locally using Poetry for environment setup. Confirmed that the application started successfully and all existing functionality was working as expected.

Triggered the CI/CD pipeline using the GitHub Actions workflow and verified that the build process completed without errors.

Deployed the updated Docker containers with the new Poetry-based environment setup to a testing/staging environment. Conducted thorough testing of the backend services, including various APIs and functionality, to ensure no regressions occurred.

Requested feedback and conducted code reviews with team members to validate the changes and address any potential issues.

Any other considerations

Update documentation and ReadME files + we need to gitignore poetry.lock

add pre-commit hooks in poetry to make sure poetry.lock and pyproject.toml are correctly in sync

[karkir0003]

You also sped up the build checks for our app as well. Flex that in the description as well! @yeaitsurya

[karkir0003]

@codingwithsurya. make sure you tackle the following items as well in this PR

1. no python-package-conda.yml action
2. no requirements.txt
3. no conda/environment.yml (conda deprecation)
4. update installation script in package.json at the root
5. update dockerfiles (train dockerfile + Dockerfile)
6. update readme to describe how to get setup on poetry (and add resource on pyenv to get the right python version)
7. add pre-commit hooks in poetry to make sure poetry.lock and pyproject.toml are correctly in sync pre-commit hooks of interest to add are: poetry-check, poetry-lock. Learn more

How to build pre-commit hooks into github: Here

Precommit YAML file to add to: Here

[farisdurrani]

Very good @codingwithsurya ! This is an important PR for the backend to evolve. Just make sure to add logger to the list of packages as we just added it