Step 3: identify Commons works linked to non-artwork items that don't have Wikidata artwork items and create the artwork items

baskaufs commented 3 years ago

[x] 1. Research and resolve inception date issues noted in issue_log.txt file. See #16. In most (all?) cases, make changes to the ACT data dump CSV file act_data_fix.csv if the ACT data is in error, or do nothing if the error is in the Commons data.

[x] 2. Decide on artwork type for all works lacking "instance of" values noted in the log file. The types currently handled are:

    {'string': 'drawing', 'qid': 'Q93184'},
    {'string': 'painting', 'qid': 'Q3305213'},
    {'string': 'manuscript', 'qid': 'Q87167'},
    {'string': 'sculpture', 'qid': 'Q860861'},
    {'string': 'carving', 'qid': 'Q97570030'},
    {'string': 'textile', 'qid': 'Q28823'},
    {'string': 'garment', 'qid': 'Q11460'},
    {'string': 'photograph', 'qid': 'Q125191'},
    {'string': 'architecture', 'qid': 'Q811979'},
    {'string': 'mosaic', 'qid': 'Q133067'},
    {'string': 'fresco', 'qid': 'Q22669139'},
    {'string': 'print', 'qid': 'Q11060274'},
    {'string': 'mural', 'qid': 'Q219423'},
    {'string': 'watercolor', 'qid': 'Q3305213'}

The string values should be entered in the ACT data dump CSV file in the ObjectFunction column and the script will assign the Q IDs. Let Steve know if there should be new types added to this list. Charlotte will do

[x] 3. Make any desired name changes to the ACT data dump CSV file based on comparisons made on the log file. Steve will do
[x] 4. ~Steve fix processing script so that the ambiguous creator matches get put into a file instead of just put into the screen printout as in capture of output.~ Run the script again after the name fixes above. Manually add artists to the output CSV after deciding about the ambiguous matches. Issue #13
[x] 5. Look at the works with missing author items and the artist data wrangled from the two source files about them and decide manually whether to create new items for them or just leave them as names because of lack of information. Take notes on this so that we have some criteria for the workflow when we start doing a larger number of works. After manually creating any artists, add their Q IDs manually to the output spreadsheet. Steve note: I rarely had to do this, but often ended up recharacterizing the type of the item, which made the "artist" string irrelevant since it was often actually the photographer.
[x] 6. Steve decide whether VanderBot can be fixed to handle "some value" values (anonymous nodes) or if we need to just use the Q ID for "anonymous" and let a bot fix it. Issue #21 Decide whether works lacking artist information should be flagged as anonymous or just left without artist info.
[x] 7. Steve check his Commons-scraping script to figure out why it missed some of the existing artworks. Issue #18 Also perform a manual scan of works from major museums whose works are probably already in Wikidata. Also consider alternate ways to discover existing items mentioned in the issue. If necessary, run the candidate new items through the scraping script again to screen them. Steve will do
[x] 8. Sort out the 3D categories from the 2D (sculpture, carving, textile, garment, architecture). Assess how many there are and how bad it would be to just create the items given what we already know. In particular, check the creators to look for conflation between the photographer and the person who actually created the 3D work (architect of building, sculptor, etc.) There needs to be a longer-term solution for this problem (see issue #19) but for now we just need to get these 100 or so works uploaded, so hopefully just manually fix problems and get on with it. Anne will do
[x] 9. Perform proofreading on the output CSV and fix stuff like trailing open parentheses, dumb artist names, etc. #20
[x] 10. Check for duplicate rows. Not sure how they would get there, but I found one.
[x] 11. Use VanderBot to create the new items.
[x] 12. Go to the non-artwork items and manually remove the ACT ID statements (which have now been replaced by links to the artwork items).

baskaufs commented 3 years ago

Complications:

Many of the cases will have minimal data from Commons. For example, this. Issues: who is actually the artist? What recognition is necessary for the Commons user Judgefloro
Here is a case where the artwork is in a gallery (BYU), but the gallery hasn't made the metadata freely available: commons page. See this from the gallery and this from some company (associated with the gallery?). In this case, how much metadata from ACT can be used to make up for the lack of information from the gallery?

baskaufs commented 3 years ago

This query should just give us the works that we need to create Wikidata artwork items for (then delete the ACT ID from the non-artwork hits):

# Since all cleanup has been done where a work has a Wikidata item that's an artwork
# this query should just be the works that need to have artwork items created.
select distinct ?work1 ?actId1 ?work1Label ?class1Label where {
  ?work1 wdt:P9092 ?actId1.
  ?work1 wdt:P18 ?commonsImage.
  optional {?work1 wdt:P31 ?class1.}

  minus {?work1 wdt:P31 wd:Q3305213.} # painting
  minus {?work1 wdt:P31 wd:Q15711026.} # altarpiece  
  minus {?work1 wdt:P31 wd:Q93184.} # drawing
  minus {?work1 wdt:P31 wd:Q22669139.} # fresco
  minus {?work1 wdt:P31 wd:Q15123870.} # lithograph
  minus {?work1 wdt:P31 wd:Q8362.} # miniature
  minus {?work1 wdt:P31 wd:Q133067.} # mosaic
  minus {?work1 wdt:P31 wd:Q219423.} # mural
  minus {?work1 wdt:P31 wd:Q125191.} # photograph
  minus {?work1 wdt:P31 wd:Q11060274.} # print
  minus {?work1 wdt:P31 wd:Q1064538.} # quilt
  minus {?work1 wdt:P31 wd:Q245117.} # relief sculpture
  minus {?work1 wdt:P31 wd:Q860861.} # sculpture
  minus {?work1 wdt:P31 wd:Q2282251.} # seven-branched candlestick
  minus {?work1 wdt:P31 wd:Q1473346.} # stained glass
  minus {?work1 wdt:P31 wd:Q179700.} # statue
  minus {?work1 wdt:P31 wd:Q18761202.} # watercolor painting
  minus {?work1 wdt:P31 wd:Q48498.} # illuminated manuscript
  minus {?work1 wdt:P31 wd:Q184296.} # tapestry

  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". } # Helps get the label in your language, if not, then en language
 }
order by ?work1Label

baskaufs commented 3 years ago

Used the "Script to pull data for works needing processing" section of the compare_metadata_sources.ipynb script to generate files act_data_fix.csv and commons_data_fix.csv that we can use to create the items for this phase.