The following is a suggestion / proposal based on the work done recently on classifications.
From doing some classification, I’ve noticed that some of the codes / names are the same in the imported 911 incidents CSV sheet (that I use for testing) and the common incident types table (generated from this file).
It's not true for all rows, but here are some values present in both for example:
Trespass
DUI
Welfare Check
I’m guessing there will be some values like this that will be present in all imported files because they use very simple and general terminology. This led me to wonder if we could implement a simple auto-classification system that just does string matching (we could define it as requiring manual review if we want). That would allow instant classification of some data set rows with only the more complex rows left to classify. Because it simply compares strings, it's only a small effort to implement. Note that this can (and probably should) happen in a background job to minimize the impact on the web app.
This could also be the very humble beginning of a more advanced auto-classifying algorithm. The next logical step would be to use past classifications to automatically "guess" how to classify new data (with some still very simple string matching using approved classifications as reference).
The following steps involve more advanced things like machine learning but I don't think we need to worry about that for now.
The following is a suggestion / proposal based on the work done recently on classifications.
From doing some classification, I’ve noticed that some of the codes / names are the same in the imported 911 incidents CSV sheet (that I use for testing) and the common incident types table (generated from this file).
It's not true for all rows, but here are some values present in both for example:
TrespassDUIWelfare CheckI’m guessing there will be some values like this that will be present in all imported files because they use very simple and general terminology. This led me to wonder if we could implement a simple auto-classification system that just does string matching (we could define it as requiring manual review if we want). That would allow instant classification of some data set rows with only the more complex rows left to classify. Because it simply compares strings, it's only a small effort to implement. Note that this can (and probably should) happen in a background job to minimize the impact on the web app.
This could also be the very humble beginning of a more advanced auto-classifying algorithm. The next logical step would be to use past classifications to automatically "guess" how to classify new data (with some still very simple string matching using approved classifications as reference).
The following steps involve more advanced things like machine learning but I don't think we need to worry about that for now.