Build in the multiblock loop

[joehand]

The main function from the topology library currently runs a single block. To run multiple blocks, would it work to:

1) Process the shapefile 2) Separate out each block 3) Create a celery task for each individual block 4) Return results as they finish on a block-by-block basis.

This is basically how Christa is already running the multiple block shapefiles.

_The question:_ will returning blocks separately be a problem on the front-end? If so, then can we create a main task (whole shapefile) w/ sub-tasks for each block?

[danrademacher]

I am assuming a person using this site would expect to submit a single file and get a single file back. that said, for purely display purposes (vs downloading) I could imagine it being useful to get each parcel as a separate GeoJSON object that we can show on the map as they are completed.

In that case, this ties into #9. If we can show iterations of the process as we go and then return one unified file at the end, that might be best.

@mojodna, do you have any opinion on this?

[mojodna]

Not really. I like the idea of processing each block separately, but I'm not sure if Celery has an appropriate abstraction for combining the results of multiple tasks together (I suspect that it does...)

I agree that showing iterations and returning a single file is the ideal user experience.

[mojodna]

Chords are Celery's abstraction for groups of tasks with individual callbacks.