chris48s
commented
7 years ago Also more sophisticated approach to #720 - running these in series is also a bottleneck
In descending order of solution desirability:
- Find a way to let them all share the same files without treading on each others' toes so they can all run in parallel
- Download the files for each importer to a different location to they can all run in parallel and just accept you need to store the files N times
- The scripts don't actually need to run in complete isolation. You can 'interleave' Scotland scripts with England/Wales scripts, just not other Scotland scripts
Full import was so fast.. then I fixed #721 and now is crazy slow again - needs additional optimization or I wil go insane :(
idea 1:
In
create_address_records_for_council(), create a list of all the local authorities which touch the target local auth then limit the search space for the spatial join where we work out the local auth to a subquery of just those areas. For example, if the target auth is Vale of Glamorgan our search space only needs to be Vale of Glamorgan, Bridgend, Cardiff and RCT, not the entire country!idea 2:
~Try and do the lookup using ONSAD first instead of using the spatial join and then fall back to a spatial lookup only if the UPRN is not in ONSAD~ (see #746)