Datenschule / jedeschule-scraper

MIT License
22 stars 15 forks source link

Create only one session in db_pipeline #102

Closed k-nut closed 3 years ago

k-nut commented 3 years ago

This is required because SQLAlchemy will ensure that an object coming from the database does not change sessions. In the previous implementation, if an item existed already, we would find it in the database from update_or_create from which it would bring its session. We would then try to add it to the session crated in process_item which would fail. Since both methods exist in the same module though, they can simply share a module-global session. Changing it to this allows us to update entries that already exist in the database.