Engine - stopping in the middle, processing batches of notices

petabyte-research / redflags

Automagically checks and filters risky public procurements

http://www.redflags.eu/

Apache License 2.0

4 stars 4 forks source link

Engine - stopping in the middle, processing batches of notices #44

Closed KrzysztofMadejski closed 8 years ago

KrzysztofMadejski commented 8 years ago

Yesterday I've left engine to process Polish notices. Today it was still running. Not a big surprise as there are a lot of them. I've checked the website and to my surprise the new notices are not included in the results. I've stopped the enging and run post-processing script hoping some aggregation may need to be run - still result is the same, only 60 pages on http://pl.redflags.transparencee.org/

I would expect that processed notices are inserted into the database from time to time? Isn't this the case? Do I as a an admin need to process it in small chunks?

Can mySQLExporter be configured to do it in batches?

juzraai commented 8 years ago

MySQLExporter spits out all data into text files then loads them into the database. This loading operation is performed when the engine finishes processing all notice in the given scope.

So do not stop the engine, if you need those data. :)
If you need smaller batches, you need to define a smaller scope.
Or you can implement a new exporter gear which stores data into database on-the-fly - but it will be much more slower. (EDIT: INSERT method would be much more slower; LOAD DATA (which is used now) would be okay with smaller chunks I guess.)

KrzysztofMadejski commented 8 years ago

Ok, thank you for the info!