App rewrite in django

[codebanesr]

Rebuilding the Backend with Django

We rewrote the backend using Django. This lets us make the architecture more modular and easier to expand later.

To start the new server, run:

python manage.py runserver
celery -A dj_backend_server worker --loglevel=info    

or with docker docker-compose up -d

Ready for Testing

What We Changed

• Added REST APIs and templates for the frontend views
• Switched to an event-driven architecture with Django signals to make adding new features easier
• Integrated the core logic into the Django app so we don't need a separate backend server

Improvements

• Accelerated web scraper through lxml (C++)
• Enhanced PDF parsing with PyPDFium2Loader (C++)
• Introduction of Python Celery for efficient message broker tasks
• Enabled grpc with Qdrant vector store
• Qdrant dashboard at 0.0.0.0:6333/dashboard

Next Steps

• Add support for the llama-index model
• Write tests for the new APIs and components
• Set up CI/CD pipelines for automated testing and deployment

This is ready now ready to be merged!!