Rewrite the app in Python Django :rocket:

[raphodn]

I've been thinking lately...

Why ?

Having a couple of years of experience in Python Django, but only a few months now with Python FastAPI, I'm struggling to develop the backend as fast and solid as I wish..

Advantages of moving from FastAPI to Django

• a shell !
• ORM much better than sqlalchemy (joins, prefetch)
• API : use DRF : better serialization, better filtering (instead of https://github.com/arthurio/fastapi-filter)
• an admin interface :heart_eyes:
• faster development, more solid backend, better manage the app's growth (my feeling is that FastAPI is good for prototyping/small projects, but harder to manage for bigger projects, with many dependencies (alembic, sqlalchemy, filter...) that are far from on-par with Django)

Disadvantages

• time to rewrite the app
• no typing
• API slower ? how slower ?
• cron managed by the server ? or django-q !

Extra info

• the initial POC - https://github.com/raphodn/open-prices-poc - was written in Django

[raphodn]

Todo

• [x] django setup : https://github.com/openfoodfacts/open-prices/pull/362
• [x] models : https://github.com/openfoodfacts/open-prices/pull/364
• [x] validation & tests : https://github.com/openfoodfacts/open-prices/pull/374 & https://github.com/openfoodfacts/open-prices/pull/392
• [x] api & serializers
  • Users : https://github.com/openfoodfacts/open-prices/pull/367
  • Location, Product & Price : https://github.com/openfoodfacts/open-prices/pull/368
  • Proof : https://github.com/openfoodfacts/open-prices/pull/369
  • Location & Price POST : https://github.com/openfoodfacts/open-prices/pull/370
  • Price PATCH & DELETE : https://github.com/openfoodfacts/open-prices/pull/372
  • set Price.source & Proof.source : https://github.com/openfoodfacts/open-prices/pull/366/commits/3576ae093cd2485d4d97a0a8831ac2c1fed16d7c
  • update price_count (on create or delete) : https://github.com/openfoodfacts/open-prices/pull/377
  • tests : https://github.com/openfoodfacts/open-prices/pull/371
  • config, remove final slash, refactoring, fixes... : #379 & #381 & #384 & #385 & #389 & #401 & #402 & #426
• [x] authentication : https://github.com/openfoodfacts/open-prices/pull/365
• [x] tasks, external APIs : https://github.com/openfoodfacts/open-prices/pull/373
  • on Price create, create/link Product & Location
  • on Proof create, create/link Location
  • on Product create, fetch from OFF
  • on Location create, fetch from OSM
• [x] cron, async
  • daily product db sync : https://github.com/openfoodfacts/open-prices/pull/376
  • daily data dump : https://github.com/openfoodfacts/open-prices/pull/380
  • make "fetch Product from OFF" & "fetch Location from OSM" async : https://github.com/openfoodfacts/open-prices/pull/382
• [x] devops (Docker, CI, deployment, Makefile) : https://github.com/openfoodfacts/open-prices/pull/402 & https://github.com/openfoodfacts/open-prices/pull/406 & https://github.com/openfoodfacts/open-prices/pull/411 & #415
• [ ] error mgmt :
  • logging
  • Sentry : https://github.com/openfoodfacts/open-prices/pull/400
• [x] create init migrations files for each apps : https://github.com/openfoodfacts/open-prices/pull/413
• [x] performance (N+1) : https://github.com/openfoodfacts/open-prices/pull/403
• [x] remove the fastapi code : https://github.com/openfoodfacts/open-prices/pull/383

Questions

• Django Admin : generate a superuser ? (as we don't really manage users, nor store passwords)
• migrate lat & lon DecimalField to PointField ? PostGIS ?

[raphodn]

Which folder architecture ?

Existing

Repo	Config	Apps	api	web	deps	README	Choice
django tutorial	mysite/mysite	mysite/<apps>
MTES-MCT/envergo	envergo/config	envergo/envergo/<apps>	envergo/requirements.txt	envergo/README.md
MTES-MCT/aides-territoires	aides-territoires/src/core	aides-territoires/src/<apps>	inside each app	inside each app	aides-territoires/Pipfile	aides-territoires/README.md
quiz-anthropocene/admin-backend	admin-backend/core	admin-backend/<apps>	admin-backend/api	admin-backend/pyproject.toml	admin-backend/README.md
betagouv/Aidants_Connect	Aidants_Connect/aidants_connect	Aidants_Connect/<apps>	Aidants_Connect/api	Aidants_Connect/www	Aidants_Connect/pyproject.toml	Aidants_Connect/README.md
gip-inclusion/le-marche	le-marche/config	le-marche/lemarche/<apps>	le-marche/lemarche/api	le-marche/lemarche/www	le-marche/pyproject.toml	le-marche/README.md	looks good!
gip-inclusion/les-emplois	les-emplois/config	les-emplois/itou/<apps>	les-emplois/itou/api	les-emplois/itou/www	les-emplois/pyproject.toml	les-emplois/README.md	looks good!
betagouv/ma-cantine	ma-cantine/macantine	ma-cantine/data (models, ...)	ma-cantine/api	ma-cantine/web	ma-cantine/requirements.txt	ma-cantine/README.md
raphodn/open-prices-poc	open-prices-poc/config	open-prices-poc/<apps>	open-prices-poc/api	open-prices-poc/www	open-prices-poc/pyproject.toml	open-prices-poc/README.md

Todo

(closest to le-marche & les-emplois)

open-prices/config
open-prices/open_prices/<apps>
open-prices/open_prices/api
open-prices/open_prices/www
open-prices/pyproject.toml
open-prices/README.md

[raphodn]

Which tool to manage crons ?

Need

• cron : fetch & update all products from OFF every day
• async tasks : fetch new location details from OSM, new product details from OFF

Tools comparison

Name	Pros	Cons	Stars	Choice
django-crontab	- simple	- only manages crons - no admin integration	~800
django-celery	- Celery is famous	- cannot use DB as broker (since v4.2 ?) --> need to install Redis dependency	~1500
django-q	- manage tasks, cron... - can use DB as broker - admin integration	- not maintainted anymore !!	~1800
django-q2	- replaces django-q	~300	yes !

[raphodn]

@monsieurtanuki deployed in prod !

[raphodn]

created a discussion here with a quick todo of next/upcoming issues : https://github.com/openfoodfacts/open-prices/discussions/421