If a dataset is over 10k koop will process an export in 10k chunks from the DB. To do this is puts the dataset into a "processing" state and uses an in memory queue to process each page. this should change to use a persistent job queue so that if the server stops, pages are processed when the server starts back up.
This will solve the problem of datasets getting put into the "processing state" and never removed from that state.
If a dataset is over 10k koop will process an export in 10k chunks from the DB. To do this is puts the dataset into a "processing" state and uses an in memory queue to process each page. this should change to use a persistent job queue so that if the server stops, pages are processed when the server starts back up.
This will solve the problem of datasets getting put into the "processing state" and never removed from that state.