Add mechanism to submit bulk edits to Colouring London via automated methods

[edwardchalstrey1]

We want to be able to make bulk edits to the Colouring London database.

These "Computational" edits (as opposed to manual/crowdsourced) should be somehow made via website API OR at least make sure these are recorded as edits, rather than just changing the values in the database.

Python/SQL scripts for these edits should be added to the /etl dir and their use described in the README.

We should also add different icons (doesn't need to be these exact ones) to the edit history and/or next to each filled in field that differentiate the source of edits:

• 👤 Person icon for crowdsourced <- silhouette icon
• 🖥️ Computer icon for computationally inferred (e.g. some rule that has been come up with to link geometries part of a single site)
• 🧑‍🤝‍🧑 Another icon for publicly available data (e.g. listed buildings EPC data), maybe split into two:
  • 🏛️ Non-private data released by government
  • 🔓 Non-private data released by organisations other than government
• 📶 Differentiate live streamed updates (e.g. planning applications)

We want the option to have a manual edit and a computational edit on the same field (multiple emojis/icons).

[edwardchalstrey1]

AlanTuring_ComGenTest has some computational edits already

[edwardchalstrey1]

I've started the above on branch auto-methods

[edwardchalstrey1]

Automated methods examples

• automatically geolocate and colour groups of domestic garages (simplest form) which have two or more rectangular ground plans of a very similar shape and size joined parallel to each other (see below)
• colouring-cities/colouring-london#112

[tomalrussell]

There are scripts which run against the API which have been used for bulk uploads - e.g. https://github.com/colouring-cities/colouring-london/blob/54a40d3d8c07af5b90d0c3f979a0c561e0f931b6/etl/join_building_data/load_csv.py

• loop over rows of data from e.g. a CSV
• GET to find the building to edit (by location, UPRN, TOID...), possibly to check what data is already on the site
• POST to add/edit data

Thinking aloud: this feature looks like it requires something like additional "edit metadata" to record the edit source type. Should this be a field in the edit history?

I would assume the frontend presentation is always the latest edit by default, and there is no change needed to the buildings table or the core process of making updates and entries in the edit history, except adding this additional field.

Should the source type (computational/crowdsourced/public/streamed...) be self-certified? i.e. something that is POSTed alongside the building data when making an edit? Should it be associated with the user - so a single user always makes a certain kind of edit? (then we could derive it server-side).

[edwardchalstrey1]

^ brilliant, great to see there is a precedent for bulk uploads against the API.

thanks for the suggestions on source type too - will need to think carefully about best approach

[polly64]

@matkoniecz can you add link to this from bulk upload issue and close?