Batch database writes in scraper to improve performance

[lordastley]

Ran cProfile on a test corpus of 18 images with no preexisting database or thumbnails. python -m cProfile -s time scraper.py > out

A snippet of the log follows:

         126909 function calls (121243 primitive calls) in 52.063 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       58   41.905    0.722   41.905    0.722 {method 'commit' of 'sqlite3.Connection' obj
ects}
        1    3.845    3.845    3.845    3.845 {method 'executescript' of 'sqlite3.Connection' objects}
       19    3.099    0.163    3.099    0.163 {method 'stretch' of 'ImagingCore' objects}
      950    2.398    0.003    2.398    0.003 {method 'decode' of 'ImagingDecoder' objects}
       19    0.192    0.010    0.192    0.010 {method 'histogram' of 'ImagingCore' objects}
       18    0.153    0.009    0.153    0.009 {method 'convert' of 'ImagingCore' objects}
       19    0.085    0.004    0.085    0.004 {method 'crop' of 'ImagingCore' objects}
       38    0.052    0.001    0.052    0.001 {PIL._imaging.new}

Roughly 78% of the execution time is consumed by writes to the SQLite db.

[lordastley]

Sorting with -s tottime provides a better view of overall execution time:

         126909 function calls (121243 primitive calls) in 41.196 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       58   32.413    0.559   32.413    0.559 {method 'commit' of 'sqlite3.Connection' obj
ects}
       19    3.085    0.162    3.085    0.162 {method 'stretch' of 'ImagingCore' objects}
        1    2.448    2.448    2.448    2.448 {method 'executescript' of 'sqlite3.Connection' objects}
      950    2.413    0.003    2.413    0.003 {method 'decode' of 'ImagingDecoder' objects}
       19    0.196    0.010    0.196    0.010 {method 'histogram' of 'ImagingCore' objects}
       18    0.153    0.009    0.153    0.009 {method 'convert' of 'ImagingCore' objects}
       19    0.086    0.005    0.086    0.005 {method 'crop' of 'ImagingCore' objects}
       38    0.057    0.001    0.057    0.001 {PIL._imaging.new}