ported ETL-PIPELINE from andy

[aybanda]

Ported ETL Pipeline from andy

Changes Made

1. Project Structure:

   • Created a new directory for the ETL pipeline project
   • Copied main.py, requirements.txt, and .gitignore from the original project

2. Environment Variables:

   • Implemented use of python-dotenv for loading environment variables
   • Added API_URL as an environment variable in main.py

3. Testing:

   • Created test_main.py with unit tests for main functions
   • Added pytest.ini for test configuration and markers

4. Documentation:

   • Updated README.md with setup instructions, usage guide, and testing information

5. Version Control:

   • Initialized new git repository
   • Changed remote origin to point to the new repository

Files Changed

• main.py: Updated to use environment variables
• test_main.py: New file with unit tests
• requirements.txt: Updated with new dependencies
• .env: New file for environment variables (not committed)
• .gitignore: Updated to include .env and other project-specific files

How to Test

1. Clone the repository
2. Set up a virtual environment and install dependencies:

   python -m venv venv
   source venv/bin/activate  # On Windows use `venv\Scripts\activate`
   pip install -r requirements.txt

3. Create a .env file with the necessary environment variables
4. Run the tests:

   pytest

5. Run the main script:

   python main.py

[bellanov]

Do you know how to write issues? Or use Jira, GitHub projects, etc.?

Those are a lot more formal in tracking work nowadays. There needs to be a ticket 🎟 with your name on it. I can assign this one to you, as it's currently tied to me -.-

Will help others with tracking / viewing open source contributions. Isolates each intention to push code and shows organization.

[aybanda]

yes I'm comfortable working with them. Good work you have done so far on your project.

[bellanov]

Finally getting back to my tasks. Let me move this forward a step.

Got ideas of not only making this installable (via pypi) but creating a fully-fledged CLI.

[bellanov]

Thanks again.

[aybanda]

Sure, that sounds like a plan

Finally getting back to my tasks. Let me move this forward a step.

Got ideas of not only making this installable (via pypi) but creating a fully-fledged CLI.