davanstrien
commented
3 years ago @MikeTrizna, we thought it would be great to submit something to this. It could also serve as a useful deadline to have something more ready + help us focus on what we're going to do.
As a way of getting stuck into the question of labels that are useful to libraries, it might be better to start with labels + images we can already get access to.
One option suggested by @glenrobson was to use the LCTGM headings described in #9 as our initial data. We can get access to images with these labels from Wikidata + LOC (and maybe BL).
The rough steps we thought of was to:
- generate an initial training set from Wikidata images + labels since this will probably be the most straightforward to acquire.
- train a model on this data to get an early sense of how well/crappy a model is doing on it
- move to the larger LOC data and train/fine-tune our model on that dataset.
- evaluate which labels it does well/badly on
- depending on how well we do training a model, move to inference on the Europeana data
- these predictions would then be used as a starting point for an annotation task:
- is the predicted label correct (we may choose to focus on a subset here)
- create additional bounding box data using these predictions. i.e. if the label is person, car etc., then we show that to someone, and they draw a bounding box for that object if it is there.
This would allow us to explore a few nice topics:
- how well can a CNN model learn from library metadata
- are there labels a model does particularly badly on?
- what (if any) features do the badly performing labels share
- how well does transfer learning work between different collections (would a big GLAM focused ImageNet style dataset be useful?)
- there will also be some fun deployment-related things we can over-engineer if we feel like it.
- we should learn more about what works well/less well when using IIIF as a way of gathering data and/or making predictions
@glenrobson, let me know if I mangled this or missed anything. @MikeTrizna would be great to hear if this sounds semi-reasonable. I think this will touch on most of what we discussed last time, but the order has been slightly flipped, which I think makes a lot of sense.
glenrobson
Deadline beginning of April